I maintain a long robots.txt and also 403 the user agents there when they request anything but robots.txt. I've blocked traffic from a few countries altogether and also block anything on SpamHaus' DNSBL.
> The A.I. company’s advisers are pushing its chief executive, Sam Altman, to move slowly after SpaceX’s stock has been volatile and as the start-up grapples with financial challenges.
SpaceX's stock volatile? It's a shame nobody saw that coming.
[email protected]
follow.coryd.dev/@cory
https://bsky.app/profile/coryd.dev