Aggressive AI scrapers are making it kinda suck to run wikis(weirdgloop.org)
weirdgloop.org
Aggressive AI scrapers are making it kinda suck to run wikis
https://weirdgloop.org/blog/clankers
4 comments
Requiring readers to log in to view content doesn't really work. Some will register so that they can use the account to crawl. Some won't even care if they are requesting hundreds of URLs a minute. Sure, that sets off alarms, but it's a game of whack-a-mole again.
DSBL of various sorts work (mostly) for emails.
Site owners could absolutely run a distributed realitime IP/metadata reputation system.
Site owners could absolutely run a distributed realitime IP/metadata reputation system.
Malicious actors also buy 'aged' accounts, so you'd need to continually monitor this reputation? Seems a hard challenge to solve for individual site owners.
It's easy and effective, aged accounts buying is rare and cumbersome, when the rest are filtered out, the "buyers" stick out like sore thumbs... then wackamole is fun.