I actually came up with a very effective method for identifying scraping and blocking it in near real-time. The challenge that I've had was that I was being scraped via many many proxies/IPs in short spurts using a variety of user agents - so as to avoid, or make difficult detection. The solution was simply to identify bot behavior and block it:
1. Scan the raw access logs via 1 minute cron for the last 10,000 lines - depending on how trafficked your site is
2. parse the data by IP, and then by request time
3. search for IP's that have not requested a universal and necessary elements like anything in the images or scripts folder, and that made repetitive requests in a short period of time - like 1 second.
4. Shell command 'csf -d IP_ADDY scraping' to add to the firewall block list.
This process is so effective of identifying bots/spiders that I've had to create a whitelist for search engines and other monitoring services that I want to continue to have access to the site.
Most scrapers don't go to the extent of scraping via headless browsers - so, for the most part, I've pretty much thwarted the scraping that was prevalent on my site.
1. Scan the raw access logs via 1 minute cron for the last 10,000 lines - depending on how trafficked your site is
2. parse the data by IP, and then by request time
3. search for IP's that have not requested a universal and necessary elements like anything in the images or scripts folder, and that made repetitive requests in a short period of time - like 1 second.
4. Shell command 'csf -d IP_ADDY scraping' to add to the firewall block list.
This process is so effective of identifying bots/spiders that I've had to create a whitelist for search engines and other monitoring services that I want to continue to have access to the site.
Most scrapers don't go to the extent of scraping via headless browsers - so, for the most part, I've pretty much thwarted the scraping that was prevalent on my site.