Devin Review: AI to Stop Slop(cognition.ai)
cognition.ai
Devin Review: AI to Stop Slop
https://cognition.ai/blog/devin-review
33 comments
Have been using Devin Review for a little bit, and I think it's the first of the many "code review" LLM-bots that have come out that doesn't actively feel like "slop". Seems like they must have some integrations with codemaps or deepwiki (the Cognition products I use most often) to power the insights.
My favorite feature has been organizing the files by "logical flow" rather than alphabetically, which feels like such a tiny change but it's such a huge QOL upgrade. A lot of the features seem inspired by Graphite, which is also really enjoyable.
My favorite feature has been organizing the files by "logical flow" rather than alphabetically, which feels like such a tiny change but it's such a huge QOL upgrade. A lot of the features seem inspired by Graphite, which is also really enjoyable.
so they graciously accept that "AI" agents generate slop (by the very title of the post)
so why would they be any better at reviewing?
so why would they be any better at reviewing?
AI reviewing AI does lead to a cycle of ouroboros slop, that’s why Devin Review is actually more of a UI for you to read code easier, not like other code review tools that try to do all the reviewing themselves
Disclaimer: i work here
Disclaimer: i work here
I’m all in for building more intuitive UIs to make the review process less cumbersome. But with the current capabilities of LLMs, under no circumstances should we allow AI to be the final judge of whether something should be merged into trunk.
Code review is the last line of defense we have against our systems being invaded by the massive amount of slop that’s getting generated left and right.
Instead of trying to automate the code review process, maybe we should spend more energy on making the scaffolding around it better: better diff tools, semantically grouped files (as Devin mentioned), and better UI for large diffs (GitHub’s UI is horrible for anything beyond a thousand lines).
Code review is the last line of defense we have against our systems being invaded by the massive amount of slop that’s getting generated left and right.
Instead of trying to automate the code review process, maybe we should spend more energy on making the scaffolding around it better: better diff tools, semantically grouped files (as Devin mentioned), and better UI for large diffs (GitHub’s UI is horrible for anything beyond a thousand lines).
(I work at Cognition, opinions my own etcetc)
True! Devin Review doesn’t make the kind of judgements you mention, it just does its best to find bugs and help you understand the code faster. I managed to review a PR on an airplane (without starlink) with it earlier this week lol
True! Devin Review doesn’t make the kind of judgements you mention, it just does its best to find bugs and help you understand the code faster. I managed to review a PR on an airplane (without starlink) with it earlier this week lol
Yeah, I wasn’t alluding to Devin reviewing and merging the changelog. This was more of a general statement, since a lot of code review tools seem to get this part wrong.
A lot of energy is being spent on making reviews faster, when reviews are intentionally meant to scale sublinearly. The goal should be: how can we make the process more convenient and less error-prone?
A lot of energy is being spent on making reviews faster, when reviews are intentionally meant to scale sublinearly. The goal should be: how can we make the process more convenient and less error-prone?
Agreed!
I can foresee a future of induced demand, where by making PRs "easier" to review, you will end up with way more PRs to review, leading PR backlogs as backed up with PRs as ever. Except now dev teams will have trust-me-bro LLM reviews convincing them that they don't actually need to do full code reviews on code they're putting into production. What could go wrong?
Very good point. So when we designed this we actually had that in mind. Devin Review is not supposed to replace your judgment and “give the answer”. It just organizes the PR in a way that makes it way easier for YOU to understand.
I was being partly facetious and I think this is probably the way things are going. I guess it's just hard to stomach that devs will end up relying on these tools more than their own intuition. But I suppose that ship has sailed already for a lot of people.
> Devin Review is free and available for PRs on regular GitHub repositories (not GitHub Enterprise). Public PRs don’t require a Devin account.
I guess the tokens are cheap enough or their pockets are deep enough, but this still seems surprising. I guess they can chalk it up to a marketing cost.
I guess the tokens are cheap enough or their pockets are deep enough, but this still seems surprising. I guess they can chalk it up to a marketing cost.
Asynchronous tokens are significantly cheaper than ones where you need them immediately at a high rate.
There a lots of free, high quality models on OpenRouter too, BTW...
There a lots of free, high quality models on OpenRouter too, BTW...
Overall I've been really impressed with Devin. IMO it's the best tool for AI generating features if you know what you're looking for, have patterns to follow, etc. I suspect the context they build about your project helps a ton.
I was literally just working on a system, using Devin to do the review no less, to add a bunch of the rules we have that are outside of linting's capability to tackle the same kind of thing. Tools like Copilot and Qodo have very high noise ratios, but do occasionally catch legit bugs. Devin Review could be a great complement, and hopefully they'll make it so we can add our own rules soon.
I was literally just working on a system, using Devin to do the review no less, to add a bunch of the rules we have that are outside of linting's capability to tackle the same kind of thing. Tools like Copilot and Qodo have very high noise ratios, but do occasionally catch legit bugs. Devin Review could be a great complement, and hopefully they'll make it so we can add our own rules soon.
I work at Cognition, lmk any feedback, will share with the team!
Devin? Now that's a name I've not heard in a long time...a lonnng time.
Seriously, in this age of Claude Code and Codex, does anyone use Devin, or even know someone who does? Do they have any users at all?
Ironically, their product has probably got massively better in the last couple of years, because the underlying LLMs got massively better at coding and long-context tasks. But that doth not a successful business model make, and unless you’re Cursor (and even then I’m not so sure) this is a very very hard space to succeed in without owning your own frontier model (i.e being Anthropic, OpenAI, or Google).
Seriously, in this age of Claude Code and Codex, does anyone use Devin, or even know someone who does? Do they have any users at all?
Ironically, their product has probably got massively better in the last couple of years, because the underlying LLMs got massively better at coding and long-context tasks. But that doth not a successful business model make, and unless you’re Cursor (and even then I’m not so sure) this is a very very hard space to succeed in without owning your own frontier model (i.e being Anthropic, OpenAI, or Google).
We wrote the actual paper on “stopping slop”
https://arxiv.org/abs/2510.15061
https://arxiv.org/abs/2510.15061
> code review—not code generation—is now the bottleneck to shipping great products.
Unsurprising, since a human still needs to understand and verify the code, be that as it's written or as it's reviewed. AI's only managed to move the brainpower required from the fun part to the tedious and boring part.
Unsurprising, since a human still needs to understand and verify the code, be that as it's written or as it's reviewed. AI's only managed to move the brainpower required from the fun part to the tedious and boring part.
Eh, code review has _always_ been the bottleneck (both for the author and any other reviewers). Pulling the agent slot machine for anything remotely challenging is just inflicting pain for no reason on yourself - if quality matters. If not, let it rip.
Otherwise, you’re gonna have to read every line (including those not in the diff) anyways. Typing it out - or getting the AI to do it at a speed you can comprehend - isn’t a meaningful slowdown at all.
Otherwise, you’re gonna have to read every line (including those not in the diff) anyways. Typing it out - or getting the AI to do it at a speed you can comprehend - isn’t a meaningful slowdown at all.
I wanted to look into their pricing for Devin+ and I have to say, ACU are entirely too opaque/confusing/complicated. The entire description of them is shrouded in mystery. And this part confuses me even more:
> Aside from the few ACUs required to keep the Devin VM running, Devin will not consume ACUs when:
> Waiting for your response
> Waiting for a test suite to run
> Setting up and cloning repositories
Ok, that kind of makes sense, but what does "the few ACUs required to keep the Devin VM running" mean? These cost $2.50/ea so "a few" means $5+ and on what time scale? Daily? Monthly?
The lowest plan comes with $20 ACUs but they don't list anywhere how far that gets you or even rough examples. I guess if you want to kick the tires $20 isn't a crazy amount to test it out yourself and maybe I'm just not the target market (I kind of feel like I am though?) but I wish their pricing made sense.
> Aside from the few ACUs required to keep the Devin VM running, Devin will not consume ACUs when:
> Waiting for your response
> Waiting for a test suite to run
> Setting up and cloning repositories
Ok, that kind of makes sense, but what does "the few ACUs required to keep the Devin VM running" mean? These cost $2.50/ea so "a few" means $5+ and on what time scale? Daily? Monthly?
The lowest plan comes with $20 ACUs but they don't list anywhere how far that gets you or even rough examples. I guess if you want to kick the tires $20 isn't a crazy amount to test it out yourself and maybe I'm just not the target market (I kind of feel like I am though?) but I wish their pricing made sense.
I'd rank this the best of the three.
Generally I actually like Gemini reviews a lot (I guess code review tasks stop it going off track drunk like Gemini tends to do when coding?) but at the moment for some reason my Gemini auth is broken and I can't work out how to fix it. Yay Google.