Ask HN: Who is running the abusive scrapers?
Personally I once had to triple app capacity and double database capacity just for just one scraper. I remember Cloudflare once figured out that Perplexity was one of the bad guys ( https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ ) but has anyone else figured out who else is running large scale and poorly behaved scrapers?
4 comments
Who is running the abusive scrapers?
Don't know, I just block them. These [1] are some of the methods I use with exception to method 7. I need to make time to try out the SSL fingerprint detection. They are crude methods but most of the bots / scrapers are kicked to the curb. The feds shutting down NetNut helped a lot too. I am looking forward to them shutting down more residential proxies.
Mirror [2]
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
[2] - https://archive.is/kVLJZ
Don't know, I just block them. These [1] are some of the methods I use with exception to method 7. I need to make time to try out the SSL fingerprint detection. They are crude methods but most of the bots / scrapers are kicked to the curb. The feds shutting down NetNut helped a lot too. I am looking forward to them shutting down more residential proxies.
Mirror [2]
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
[2] - https://archive.is/kVLJZ
never got a name, best we managed was network attribution. ours came mostly from tencent cloud plus a second hetzner asn, all spoofed chrome on windows user agents, and it rotated from digitalocean to tencent within a day once we started blocking asns. need to tell that it executed the AWS WAF js challenge and replayed the token, but never solved an interactive captcha, so flipping the action from challenge to captcha is what finally stopped it