Anybody know why these web crawling/bot standards are not evolving ? I believe robots.txt was invented in 1994(thx chatgpt). People have tried with sitemaps, RSS and IndexNow, but its like huge$$ organizations are depending on HelloWorld.bas tech to control their entire platform.
I want to spin up endpoints/mcp/etc. and let intelligent bots communicate with my services. Let them ask for access, ask for content, pay for content, etc. I want to offer solutions for bots to consume my content, instead of having to choose between full or no access.
I am all for AI, but please try to do better. Right now the internet is about to be eaten up by stupid bot farms and served into chat screens. They dont want to refer back to their source and when they do its with insane error rates.
As many have stated, I would assume more than 38%. But good quality content is rare and the dynamic content made page combinations go infinite 20 years ago.
We are maintaining 25y old urls is a bit ehh. cumbersome, and I sometime wonder if its worth it. Most of the traffic seem to come from bots and they do seem to learn some of the 301's. It seems to be good for SEO, etc.
Some users also gets redirected to the content on the new urls. It feels a bit like helping an elderly person over the street to where the shop is.
Anyway, I hope that bots and humans trust our services more.
NextStep/OSX was the only desktop OS that did not feel like a downgrade from Amiga Workbench