PyCon Pittsburgh. April 15-23, 2020.

Talk: Distributed Web Scraping in Python

Presented by:

J. Weissbock

Description

Web scraping is easy in Python when it is just a single web page. It becomes much more difficult when you regularly have to scrape 10,000s webpages efficiently, quickly and ensuring minimal failures from traditional network issues. In this talk we iteratively look at how a distributed web scraper is built, how to build one and why.