At first, we kind hated AI tools. We’d ask it to build something and we’d get the most generic looking nonsense that we’d ever seen. Over time we learned with everyone else about .md files, context windows, and how to basically “onboard” an agent into a project.
The benchmark tries to measure that gap directly. How agents suck until you not only tell them about your code, but also the context around what you’re building.
Gatekeeping is probably the wrong word here. But like, letting someone create a script that uses regex to parse some annoying forms they have to deal with in my totally made up hypothetical is a good thing.
While there's gratification in leaning how it all works, I don't think that should be a requirement to use a computer to do a task. And the smaller that wall the better IMHO.
I don't remember who said it first but "software teams always ship their org chart" has always stuck with me. How a team is organized has such an outsized hand it what they build.
At first, we kind hated AI tools. We’d ask it to build something and we’d get the most generic looking nonsense that we’d ever seen. Over time we learned with everyone else about .md files, context windows, and how to basically “onboard” an agent into a project. The benchmark tries to measure that gap directly. How agents suck until you not only tell them about your code, but also the context around what you’re building.