{"id":4450,"date":"2024-07-18T13:13:13","date_gmt":"2024-07-18T13:13:13","guid":{"rendered":"https:\/\/weasweb.com\/?p=4450"},"modified":"2024-08-14T10:20:59","modified_gmt":"2024-08-14T10:20:59","slug":"automatic-text-data-gathering-and-processing-using-ai","status":"publish","type":"post","link":"https:\/\/weasweb.com\/automatic-text-data-gathering-and-processing-using-ai\/","title":{"rendered":"Automatic Text Data Gathering and Processing using AI"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

The company needed to complete a complex migration on a tight deadline to avoid millions of dollars in post-contract fess and fines.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Automatic Text Data Gathering and Processing using AI\n<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"Data\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In 2021, approximately 80 zettabytes of data were generated worldwide, and this is projected to grow to over 150 zettabytes by 2025. With such exponential growth, the challenge lies in efficiently searching and finding relevant information. To address this, we proposed a technical solution utilizing automatic web searching and scraping, as well as AI models and NLP for data processing.<\/strong><\/p>\n

Our focus was on news articles related to specific topics or entities, but the methods can be applied to various other use cases such as market research, sentiment analysis, business intelligence, and machine learning dataset production.\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The Challenges<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Some of the challenges we faced during the project included addressing web server restrictions to avoid IP bans<\/strong> due to high request volume, ensuring the accuracy of data extraction<\/strong> from diverse HTML documents, devising effective methods to handle unstructured data from news articles and blogs, scaling the automatic processes for web searching and scraping to handle the increasing data volume, and ensuring compliance<\/strong> with web policies and regulations to avoid potential legal and ethical issues.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Solutions Provided<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The automatic processing of news articles involves stages like searching, scraping, content extraction, and processing.<\/strong><\/p>\n

\u00a0<\/p>\n

1. Web Searching<\/strong><\/span><\/h5>\n

For the automatic web searching we\u2019ve identified the following options:<\/p>\n

We could either develop an in-house solution<\/strong> using Python, Scrapy, and Selenium, or integrate an external API<\/strong> like SerpWow to quickly retrieve results from search engine results pages in JSON format.<\/p>\n

\u00a0<\/p>\n

2. Web Scraping<\/strong><\/span><\/h5>\n
\n
\n
\n
When we were scraping web pages, it was important to handle issues related to time gaps between requests and web server restrictions.<\/b> We used Python with Scrapy and a proxy service to ensure efficient scraping. Our tests showed that for thousands of URLs, the process took only a couple of minutes.<\/div>\n
\u00a0<\/div>\n<\/article>\n
\n
3. Content extraction
<\/strong><\/span><\/h5>\n
\n
\n
\n
After scraping HTML documents, the main content for news or blog articles needed to be extracted along with important metadata such as article title, author, source, and date. We utilized dragnet, an AI\/ML solution, and integrated Python libraries like newspaper, goose3, and others to extract content and metadata. The extraction of article dates posed challenges due to varied formats. By combining methods, libraries, custom logic, and algorithms, we achieved over 93% accuracy<\/b> in extracting content and metadata from the majority of the tested scraped pages.<\/div>\n
\n
\n
\u00a0<\/div>\n<\/div>\n<\/div>\n<\/article>\n<\/div>\n<\/div>\n