The bot I had was using unique IPs for each request. Some were from cloud providers but most were just random residential ISPs. I couldn't see any obvious connections so rate limiting would've had to be a global rate limit.
Each IP only makes ~1 request though so easy to detect after the fact.
I guess they will run out of IPs at some point so maybe if I had logged each one forever and shown a challenge only to them, it would have fixed it eventually. Just depends how big their pool of IPs is.
I think people get annoyed when it's suggested they spend time optimising or even re-writing their websites to handle high traffic loads just to cater to AI bots ripping their content.
It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good output isn't that easy, but the wiki doesn't have many users so that would be fine if it wasn't for the AI bots.
The AI bots ignore robots.txt and were initially hitting the site with ~1k rps crawling every combination. Even that would be manageable as there's currently ~150,000 combinations, except they kept re-crawling the whole lot each day. The server could manage it but it was a massive waste of resources.
They were using residential IPs and only sending 1 request from each IP making it impossible to block. In the end I gave up and put a Cloudflare challenge in front of it. I don't want to use Cloudflare but the alternative is forcing users to login to view diffs or remove them entirely.
This feels like it's becoming less and less true, good quality items are becoming so expensive now or very hard to find.
I do think it is still very true for tools though. It's nearly always worth getting decent ones, they nearly give better results or are easier to use and last so much longer.
It likely is AI scrapers essentially doing a DDoS. They use separate IPs (and vary the UA) to prevent blocking.
I have a site which is currently being hit (over 10k requests today) and it looks like scrapers as every URL is different. If it was a DDoS, they would target costly pages like my search not every single URL.
SQLite had the same thing: https://sqlite.org/forum/forumpost/7d3eb059f81ff694
As have a few other open source repositories. It looks like badly written crawlers trying to crawl sites as fast as possible.
Unless Trump wants to start WW3 the US won't be able to do anything, and even if he did start WW3, Europe would be able to destroy those bases.
The whole point of them was to give the US influence while improving US security. Given Europe can't trust Trump will come to their aid, they won't give the US as much influence over Europe.
Similar to the one SQLite had: https://www2.sqlite.org/forum/forumpost/7d3eb059f81ff694?t=h
Each IP only makes ~1 request though so easy to detect after the fact.
I guess they will run out of IPs at some point so maybe if I had logged each one forever and shown a challenge only to them, it would have fixed it eventually. Just depends how big their pool of IPs is.