The post raises several points that I wholeheartedly agree with, but the framing is poor and honestly kind of elitist (or just short-sighted). Maybe to the point that I think much of it might just be bait, lol. For example:
> Ask a twenty-two-year-old to connect to a remote server via SSH. Ask them to explain what DNS is at a conceptual level. Ask them to tell you the difference between their router’s public IP and the local IP of their laptop. Ask them to open a terminal and list the contents of a directory. These are not advanced topics. Twenty years ago these were things you learned in the first week of any serious engagement with computers.
What? Computers were everywhere in all kinds of domains by 2006, but you can bet that your average accountant of the time would most likely not be able to SSH into a server (nor should they need to...) I guess it really depends on what the author qualifies as a "serious engagement with computers."
It obviously depends on local laws, but it's very commonly illegal to sell prepared food without a license/permit. You might not get caught selling food on FB Marketplace, but that doesn't make it any less allowed.
I agree with the author regarding Apple's walled-garden app distribution, but the analogy just doesn't work here.
I'm interested in how the poison data was generated and why it's "practically endless". It looks like bits of code, structured data, and prose, but with small modifications that make it subtly incorrect. Usually off-by-a-few numbers, e.g. I got the text of GPL-3.0 with a copyright date of 2738.
Who says you need to pipe the entire document with JSON-LD directly into the context window? I agree, that is very wasteful. You can just parse the relevant bits out and convert the JSON-LD data into something like your txt format before presenting it to the LLM. Bake that right into whatever tool it uses to scrape websites.
Anubis doesn't target crawlers which run JS (or those which use a headless browser, etc.) It's meant to block the low-effort crawlers that tend to make up large swaths of spam traffic. One can argue about the efficacy of this approach, but those higher-effort crawlers are out of scope for the project.