Very interesting paper and I must say that I totally agree with it. But also that is something that is not new.
I would say that the initial expectation is a bit off. I never expected that picking up any agentic coding solution, drop it in a project and fire at it a list of tasks would just magically work and follow a project pre-defined constraints.
I do not believe that any agentic coding stack comes out of the box capable of this. Agents still need proper mechanics to understand the context, constraints and objectives reliably and that's still a work in progress as we can see by the constant updates on tools and skills and processes from Leading AI labs. They are now trying to fill that additional layer, which by the way could be much more profitable then bare model and token consumption.
I would also argue that current OS models, like the ones tested, if properly driven can already produce production code following the desired constraints.
What has been you experience? What has your production code looked like in recent months?
I have noticed particularly in recent weeks and maybe couple of months that token costs are just ridiculous.
I can understand the upcoming IPOs and instinctive pressure to show profits ... but let's be honest, showcasing burning 1.3 million USD in tokens by a single developer in a month is the most ridiculous thing I have seen in my entire life.
The general principles still apply. You expect investing X and have a return on such investment.
Unfortunately that's not so easy to promise or expect.
There's no real 1 to 1 correlation between amount of code written and returns, and even less between tokens burned and returns.
I start to believe that the current token pricing approach, followed at the moment by all leading labs (especially considering OS models capabilities), is bordeline delusional ...
Actually the ultimate combo would be to learn this with the learning mode tools provided by AI providers.
I must say that it really is a super interesting and efficient way to learn.
Any of you tried them ?
Judging by how things are moving ( pricing models, limits, harness patchy updates ), it feels like the real salvation will be a combination of more mature OS models and some open source harness setup like OpenCode or similar. I'm feeling like OS models are nearly there, and with the proper setup and harness might already be there. What are the general thoughts on this ?
What has been you experience? What has your production code looked like in recent months?