This post was originally published on this site
Web scraping. It sounds like hard work, but it is more clever than arduous.
The technique exploits a simple truth: The front end of the web site, which you see, must talk to the back end to extract data, and display it. A web crawler or bot can gather this information. Further work can organize the data for analysis.
Digital marketers are forever seeking data to get a better sense of consumer preference and market trends. Web scraping is yet one more tool towards that end.
First crawl, then scrape
“In general, all web scraping programs accomplish the same two tasks: 1) loading data and 2) parsing data. Depending on the site, the first or second part can be more difficult or complex.” explained Ed Mclaughlin, partner at Marquee Data, a web scraping services firm.
Web scraping bears some resemblance to an earlier technique: web crawling. Back in the 1990s, when the internet occupied less cyber space, web crawling bots compiled lists of web sites. The technique is still used by Google to scrape for key words to power its search engine, noted Himanshu Dhameliya, sales director at process automation and web scraping company Rentech Digital.
For Rentech, web scraping is just obtaining “structured data from a mix of different sources,” Dhameliya said. “We scrape news web sites, financial data, and location reports.”
“Web scraping data is collected on a smaller scale,” said George Tskaroveli, project manager at web scrapers Datamam, “still amounting to millions of data points, but
Read more here: https://martech.org/how-web-scraping-can-be-a-valuable-data-source/