For me it was earlier this year when I started dusting off some old stalled projects and had an agent work on them. In a few days I:
* Built a clone of the Alpha Zero implementation[1] my team built at oracle
* Ported my hobby NES emulator from javascript to rust[2] (this actually took less than 30 minutes and worked on the first try)
* Implemented all of the lessons from the C++ Grandmasters Challenge (which eventually led to a complete c++ compiler[3])
The thing that flipped the switch was using it to build things that I actually put sweat-equity in to previously. I knew how hard these things were to build, so it landed in a way that other projects had not.
I've been experimenting quite a bit with long-horizion agentic coding[1] and I have also noticed that agents seem to perform worse when forced into certain architectural patterns. I have found that is a bit better when including the constraints along the way instead of adding them after the fact. There seems to be a side-effect I have been calling "calcification", where a pattern starts appearing in the codebase and the agent follows the pattern to the point where it dominates the context and becomes self-reinforcing. This could potentially be a strength or a weakness for existing code bases depending the codebase quality. I will have more insights on this soon as more from-scratch runs conclude that include architectural guidance from the beginning.
I think there is a flow in most organizations from:
llm -> prompt -> result
llm -> prompt + prompt encoded as skill -> result
llm -> prompt + deterministic code encoded as skill -> result
I do think prompting to generate code early can shortcut that path to deterministic code, but we're still essentially embedding deterministic code in a non-deterministic wrapper. There is a missing layer of determinism in many cases that actually make long-horizon tasks successful. We need deterministic code outside the non-deterministic boundary via an agentic loop or framework. This puts us in a place where the non-deterministic decision making is sandwiched in between layers of determinism:
deterministic agentic flows -> non-deterministic decision making -> deterministic tools
This has been a very powerful pattern in my experiments and it gets even stronger when the agents are building their own determinism via tools like auto-researcher.
This is actually a very interesting moment to potentially overcome network effects, because more and more code is going to be written by agents. If a crdt approach is measurably better for merging by agent swarms then there is incentive to make the switch. It also much easier to get an agent to change its workflow than a human. The only tricky part is how much git usage is in the training set so some careful thought would need to be given to create a compatibility layer in the tooling to help agents along.
* Built a clone of the Alpha Zero implementation[1] my team built at oracle
* Ported my hobby NES emulator from javascript to rust[2] (this actually took less than 30 minutes and worked on the first try)
* Implemented all of the lessons from the C++ Grandmasters Challenge (which eventually led to a complete c++ compiler[3])
The thing that flipped the switch was using it to build things that I actually put sweat-equity in to previously. I knew how hard these things were to build, so it landed in a way that other projects had not.
[1]: https://medium.com/oracledevs/lessons-from-implementing-alph...
[2]: https://github.com/vishvananda/popeye
[3]: https://medium.com/@vishvananda/i-spent-2-billion-tokens-wri...