Anonymous records feel like one of those features that are obviously useful once you work with JSON-heavy systems, but surprisingly uncommon in statically typed languages
Unfortunately, model quality is not the only criterion for users, and often not even the most important one. Adoption is also driven by marketing, UX, integrations, pricing, ecosystem, and a lot of other non-benchmark factors.
Also, model providers are not interested to have their models compared head-to-head under identical conditions. And “Model A is better than Model B” is almost meaningless by itself. Better for what task? With what prompt? What inputs? What budget? What failure tolerance?
It would be nice to have a place where users could run their own benchmarks, define evaluation criteria for their actual use cases, and make those runs verifiable by others.
Totally agree. I used to work with a team that built a project for creating ontologies of Git repositories. The goal was to help LLMs onboard faster and navigate the repo better.
In the end, it became heavy overengineering: people no longer understood not only the repo itself, but also the extra layer describing it. Meanwhile, coding assistants are already quite good at reading codebases directly.
It would be nice to see some metrics. I think the missing layer here is evaluation. If agents are going to produce applications, the platform needs not only guardrails, but public-ish evidence that those guardrails actually catch failures