Doesn't this whole argument fall apart if we consider iteration over time? Sure, the initial implementation might be uncoordinated, but once the subagents have implemented it, what stops the main agent from reviewing the code and sorting out any inconsistencies, ultimately arriving at a solution faster than it could if it wrote it by itself?
"Helped build itself" is a bit of a stretch here, it makes it sound as if the model was doing lasting self-improvements.
What the article describes is that the model was able to tweak to its own deployment harness (memory, skills, experimental loop etc) to improve performance on benchmarks. While impressive, it's not doing any modifications to its own weights by e.g. modifying the training code.
In my experience, one reason for unnecessarily complex solutions during vibe coding is the incremental work pattern. Most users don't spend much time designing the solution, but instead jump quickly to implementation and then iterate. When doing that, the models seem prone to applying more short-sighted patches to existing code instead of doing a larger refactor that would simplify it all.
Other than spending more time on design, I also usually ask the agent to spawn a few subagents to review an implementation from different perspectives like readability, simplicity, maintainability, modularity etc, then aggregate and analyze their proposals and prioritize. It's not a silver bullet and many times there are no objective right answers, but it works surprisingly well.
When I want to solve a new problem with an agent, I always ask it to search broadly for prior work in the given area online, and then analyze if we can build our solution using it as inspiration.
I see it as the solution being out there in “idea space”, and by having the agent search beforehand we can more efficiently explore this space before converging on the final solution.
Limit access to whatever their project requires. The difference is that human interns have some common sense and won't suddenly be hijacked by a hidden message they stumble upon while searching the web, instructing them to exfiltrate a bunch of proprietary data. It is surprisingly easy to get an agent to do that though
This works well for vibecoding on a codebase in isolation, which to be fair is what the author is addressing. I don’t think it solves the problems at the current frontier of agent use though, where you expose internal infrastructure via tools to make the agent maximally productive. How to do this safely is still unsolved
Maybe. Personally I find it hard to tell how sincere this is. The cynical take is that this is just an attempt to secure their own position, especially if AI progress slows down and competition increases. However, if it does not and we’re truly approaching superintelligence, I could imagine there being voices in the company that are genuinely concerned about how society will handle such a shift.
Usage limits are more generous and GPT 5.4 is a good model, but yes, UI/UX lags behind Claude Code. Currently I'm especially missing /rewind with code restoration and proper support for plugin marketplaces
Not to give Snowflake credit for a design that clearly wasn't a sandbox, but I think it's worth recognizing that they probably added the escape hatch because users find agents with strict sandboxes too limited and eventually just disable it. The core issue is that models still lack basic judgment. Most human devs would see a README telling them to run wget | sh from some random URL and immediately get suspicious. Models just comply.
I've found that spending most of my time on design before any code gets written makes the biggest difference.
The way I think about it: the model has a probability distribution over all possible implementations, shaped by its training data. Given a vague prompt, that distribution is wide and you're likely to get something generic. As you iterate on a design with the model (really just refining the context), the distribution narrows towards a subset of implementations. By the time the model writes code, you've constrained the space enough that most of what it produces is actually what you want.
Does this matter in practice though? By modifying some of the generated code and not taking a solution produced by an LLM end-to-end but borrowing heavily from it, can't a human claim full ownership of the IP even though in reality the LLM did most of the relevant work?
I think AGENTS.md will still have a place regardless. There are conventions, design philosophies, and project-specific constraints that can't be inferred from code alone, no matter how good the judgment
Would have loved to see a more detailed breakdown of performance by task type. The commit metadata is right there, seems straightforward to tag commits as feature vs refactor vs bug fix vs API change and report per-category numbers.
Producing the most plausible code is literally encoded into the cross entropy loss function and is fundamental to the pre-training. I suppose post training methods like RLVR are supposed to correct for this by optimizing correctness instead of plausibility, but there are probably many artifacts like these still lurking in the model's reasoning and outputs. To me it seems at least possible that the AI labs will find ways to improve the reward engineering to encourage better solutions in the coming years though.
This would be my guess too. It can probably be generated synthetically or via agentic rollouts, but high quality long context examples where outputs meaningfully depend on long-range interactions probably remain scarce
This seems like a win for open source maintainers pressed on time and resources. Whether or not LLMs find novel security risks or just pattern-match known issues, many vulnerabilities are discovered late (or never) simply because nobody has the bandwidth to audit every file.
In a way, isn't this the same old data moat that always existed in AI/ML, but supercharged? Generalist models can now reason over proprietary data as context instead of requiring you to train narrow expert models on it. What changed is you no longer need an ML team to turn that data into value.
Very interesting benchmark, excited to see what comes out of this. Considering humans are enourmously more sample efficient compared to today's models, it seems clear there's a lot of room to close that gap. The fact that they hit 5.5x in the first week with relatively straightforward changes suggests we're nowhere near the ceiling for data efficiency