Web scraping is the automated collection of data from websites using a scraper program and proxy servers to handle the request-response cycle.
Scraped data can be used to analyze past trends like SEO rankings and competitor pricing or to predict future trends like market demand.
Public datasets like Common Crawl have limitations, so custom web scraping provides fresher, more relevant, and multimodal data for training superior AI models.
Retrieval augmented generation (RAG) uses live web scraping to integrate the most current external knowledge directly into an LLM's response generation process.
Web scrapers face major challenges from anti-bot measures like fingerprinting and CAPTCHAs, as well as from inconsistent and messy HTML structures.
AI classification models trained on labeled HTML data can automatically validate responses to detect blocks and adaptively parse messy content without hardcoded selectors.
An AI-powered tool can take a natural language prompt and a list of URLs to automatically generate parsing instructions and extract structured data.
Web scraping provides the fresh, high-quality data that AI models need to function, while AI makes the scraping process itself smarter and more resilient.
19:20Jan Curn
21:17Lars Kölker
35:16Ankit Patel
28:29Taqi Jaffri, Kapil Gupta & Farooq Sheikh and Tomislav Tipurić
28:09Jan Oberhauser, Mirko Novakovic, Alex Laubscher & Keno Dreßel
26:08Alexander Lichter
28:20Dieter Flick
26:04José Kadlec
Jobs that call for the skills explored in this talk.
Gemma Analytics