We're building AI testing tools at QA.tech and this matches my experience. Great post. The hard part was never generating code. It's figuring out if what came out is actually correct. Our team runs multiple AI agents in parallel writing code and honestly we spend way more time on verification than generation at this point. The ratio keeps getting worse as the models get better at producing plausible-looking stuff.
The codebase growth numbers feel right to me. Even conservative 2x productivity gains break most review processes. We ended up having to build our own internal review bot that checks the AI output because human review just doesn't keep up. But it has to be narrow and specific, not another general model doing vibes-based review.
We are using neo4j to power our agents at QA.tech.
Essentially to make it behave more like a human so that it learns and builds up an understanding of the pages it should test we map interactions into a knowledge graph stored in Neo4j. The consists of Pages and Actions on the page as well as links to documentation sections and other relevant info, together with descriptions, metadata and embeddings for search.
To make the agents better at planning and understanding the context of the page it can search the graph for relevant information and expand through the graph for more context.
This works remarkably well. I think our agents (when they have interacted a bit w the page) are some of the best browser agents I have tested.
I would highly recommend this but you need to put some effort into a nice ontology for the graph and making the tolling right for your use case. Its really not just plug and play. :)
Loving this! Very surprising that the LLMs of today are so bad at understanding interfaces but it also makes it a very interesting case for finetuning!
The codebase growth numbers feel right to me. Even conservative 2x productivity gains break most review processes. We ended up having to build our own internal review bot that checks the AI output because human review just doesn't keep up. But it has to be narrow and specific, not another general model doing vibes-based review.