I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using Google in Tor.
By caching, I just mean storing of data locally so you don't have to request it again under a certain timeframe. I use my own caching scripts written in Python, if you use a 3rd party library then data deletion does not matter too much either if you configure it properly and backup the data - html/json data compresses really well using lzma2 in 7-zip.
There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debugging.
A project like Scrapy should have caching on by default, but it seems to be an afterthought. Repeatable and reproducible parsing of cached websites is necessary, e.g. if you find additional data fields that you want to parse without downloading the entire site over again.
On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.
That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable and there is no browser overhead with page rendering.
It's extremely rare to be ip-blocked by any website just for using the Google's user agent from a non-specific range. IP's get re-used and you can switch to a new one easily, so it's really not common or good practice for this to happen.
Over 7 years experience with Python software development, cloud services, data mining, web crawling, databases. Want to extract or crawl data from a website such as business listings, sports data, government data, site directories, either one-off, periodically, or in real-time? Contact me via email: dmn001(at.[gmail
Right, if you're just matching a substring in a list of strings then you don't need regex for that.
Also, lxml is over 20x faster than BeautifulSoup, although it doesn't really matter with such a low frequency of requests, it's something to note if you had another project that required parsing e.g. 1,000's of html pages.
Over 7 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, either one-off, periodically, or in real-time? Contact me via email in profile.
I don't think I have the other videos, IIRC he removed or made private the previous videos shortly after the later ones were uploaded and that was the impetus for downloading them in the first place.
It would just be much easier to use the Facebook graph api, there is an official Python module and is well documented, and would be less likely to hit rate limits or other blocks - ironically that was one of the reasons that the author used scraping instead of the api.
Over 5 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website such as business listings, sports data, directories, either one-off, periodically, or in real-time? Contact me via email in profile.
Btw, I have no association with this company, other than I like to keep track of their blog and events regularly as it ties in quite closely with my line of work. Thought I'd post as it's an article that fits in the startup culture of being able to adapt quickly and dealing with situations you can't prepare for, whilst turning them around into a positive.