Where Do AI Coding Agents Fail?(arxiv.org)
arxiv.org
Where Do AI Coding Agents Fail?
https://arxiv.org/abs/2601.15195
3 comments
Sounds like your experience matches what's being described in the paper. Even if we get correct code, it might not carry design intent.
Not surprising that AI PRs fail when they touch lots of files or try to implement features nobody asked for. Human context still matters a lot.
I always have to tell the agent to use functional iterators and itertools all the time but it still prefers to use primitive for loop and push into a mutable array. Not that it doesn't get the thing done but please when you can iter collect why can't you use them
It also lacks the ability to use high level data structure such as bit vector and matrices. I doubt they could use even harder stuff such as B+ tree, red black tree or Fibonacci heap...
In my experimentation in building vibewasm, a wasm engine using a binding that I wrote the low layer of sljit manually, and I instructed Opus 4.5 to build a wasm engine out of it, and take in designs from other wasm engine that is based on sljit but in C or C++...pwart and walrus to be specific
I have a specific case on the use of bit vector, at least for Opus, it always tend to use hash/btree set for this.
I have to explicitly explain and tell Opus that the way you are marking the register can be represented as a bit vector, because the number of registers are bounded (15) and bit vectors will have an ultimate space save. But a next refactor attempt to rewrite the SSA layer into RTL (register transfer language that targets sljit), the same mistake happened again. It turns out my prompt of following previous design didn't work, or Opus simply just couldn't justify that bit vector is the best despite user requirements.
I have to revert that change and do it myself.