Getting blocked for too many requests will be all too familr to those who scrape large amounts of data an often. We should be fair to websites and not spam their servers, however in the times where we don’t have an option, or are blocked from the website altogether sending our request through a proxy is the best option.
In this video I go through how to implement rotating proxies using requests with Python. We look at scraping some free proxies and writing a script to see if they work. Although free proxies aren’t any use for actual web scraping projects the principles are the same.
Code here: https://github.com/jhnwr/rotatingproxies