Web scraping is easy in Python when it is just a single web page. It becomes much more difficult when you regularly have to scrape 10,000s webpages efficiently, quickly and ensuring minimal failures from traditional network issues. In this talk we iteratively look at how a distributed web scraper is built, how to build one and why.