AI code review will very probably have signal-to-noise problems.
It is good to see practical solutions aimed at addressing this.
I wonder if fine-tuning models would help - this isn't addressed in the blog.
While Sonnet-3.5 excels in accuracy-token, o1 excels in self-reflection in small isolated tasks. With AlphaCodium, o1 tasks are broken to small isolated tasks, while the flow introduced in AlphaCodium actually guides the steps and overall decision making framework.
We will see more of these frameworks for different use cases
Hey, co-creator here, I agree with the sentiment that code coverage may be a proxy and even sometimes a vanity metric but at the same time, IMO unit regression tests are necessary for a maintainable production codebase. I personally don’t feel confident making changes to production code that isn’t tested.
Specifically for generating unit regression tests the Cover-Agent tool already works quite well in the wild for some projects, especially isolated projects (as opposed to complex enterprise-level code). You can see in the few (somewhat cherry-picked) examples we posted [0] that it generates working tests that increase coverage (they were cherry-picked in the sense that these are examples we like to work with often internally at CodiumAI).
I believe that it’s possible to generate additional meaningful tests including end-to-end tests by creating a more sophisticated flow that uses prompting techniques like reflection on the code and existing tests, and generates the tests iteratively, feeding errors and failures back to the LLM to let it fix them. Just as an example. This is somewhat similar to the approach we used with AlphaCodium [1] which hit 54% on the CodeContests benchmark (DeepMind’s AlphaCode 2 hit 43% [2] with the equivalent amount of LLM calls).
If like me you think tests are important but hate writing them, please consider contributing to the open source to help make it work better for more use cases.
https://github.com/Codium-ai/cover-agent
Hey, one of the creators here.
As mentioned in the post, TestGen-LLM (by Meta) focused on Kotlin, and the prompts were very Kotlin-oriented.
In Cover-Agent (by CodiumAI) we tried to reimplement Meta's work, and stay mostly similar to the original implementation, although we did a bit improved the prompts. But it isn't generic enough.
We believe we know how to improve generality, as we did with our PR-Agent, and here is a rough plan:
https://github.com/Codium-ai/cover-agent/issues/13
To put it simple:
Like copilot, it does have an auto-complete and also chat interface. Different than copilot, it focuses on generating a full code task plan, then having the auto completion work acrroding to your plan, and it check the of quality code.
Like agents, it tries to help you complete a full task, yet,does that in tandem with you, as you work inside tour favorite IDE writing the code with you. In addition, there is focus on code quality, testing, and fetching relevant context from your codebase.
We’ve come up with a bit of a different concept for what a coding agent should be. We believe it should work in tandem with a developer inside the IDE. Over time as the tech improves, it will get more and more autonomy.
We’ve been using our coding agent internally and see a 5-10x boost on some tasks.
The agent is available now to Codiumate VS Code users. We want to hear what kind of tasks it works well on and improve it over time to expand the task set. Would love to get feedback.
There are tools focused on Code Generation, and those that are focused on Code Analysis or Integrity.
Their UX/UI might be very different as well as the underlying tech.
I wonder if you have tried tools that are dedicated to Code Integrity, e.g. generating tests?
I think that the next step is getting an official "checked" mark by the SWE bench team