The AI Nerf Is Real(isitnerfed.org)
isitnerfed.org
The AI Nerf Is Real
https://isitnerfed.org
2 comments
Hey, this looks cool! Are you planning to open-source the project by any chance? It would be useful to see or contribute to the code generation evaluations you use
We run a variety of tests through Claude Code and the OpenAI API (using GPT-4.1 as a reference point for comparison). We also have a Vibe Check feature that lets users vote whenever they feel the quality of LLM answers has either improved or declined. Over the past few weeks of monitoring, we’ve noticed just how volatile Claude Code’s performance can be.
Chart is here: https://i.snipboard.io/RydmH7.jpg
1) Up until August 28, things were more or less stable.
2) On August 29, the system went off track — the failure rate doubled, then returned to normal by the end of the day.
3) The next day, August 30, it spiked again to 70%. It later dropped to around 50% on average, but remained highly volatile for nearly a week.
4) Starting September 4, the system settled into a more stable state again.
It’s no surprise that many users complain about LLM quality and get frustrated when, for example, an agent writes excellent code one day but struggles with a simple feature the next. This isn’t just anecdotal — our data clearly shows that answer quality fluctuates over time.
By contrast, our GPT-4.1 tests show numbers that stay consistent from day to day.
And that’s without even accounting for possible bugs or inaccuracies in the agent CLIs themselves (for example, Claude Code), which are updated with new versions almost every day.
What’s next: we plan to add more benchmarks and more models for testing. Share your suggestions and requests — we’ll be glad to include them and answer your questions.
https://isitnerfed.org