Reviewing in the Age of AI
12 comments
yeah the tricky part with AI generated code is knowing what to actually review. most non-technical folks don't catch the stuff that'll bite you later.
honestly just focusing on the boring parts helps. does it crash? does it eat battery? does it actually save your data when you close it? run it on an old device if you can. the fancy code stuff matters less than whether it works when someone's internet is spotty.
Our teams efforts have shifted toward reviewing plans as a team.
Before code hits a humans eyes it's went through a few independant review passes from our review agents.
If it's low complexity / blast radius it gets auto merged.
If it's high complexity / blast radius it gets flagged for human review.
And funny enough our team has agreed even our human review layer has the best results if we have an agent create supplementary descriptions of the code and potential issues that we read and apply human judgement.
Reason being is that it can be very pendantic. Both in a good and bad way. Either flagging things that are non-concerns, or catching things my lazy human eye wouldnt have caught. e.g. A docstring mis-describing the actual shape of an object.
Before code hits a humans eyes it's went through a few independant review passes from our review agents.
If it's low complexity / blast radius it gets auto merged.
If it's high complexity / blast radius it gets flagged for human review.
And funny enough our team has agreed even our human review layer has the best results if we have an agent create supplementary descriptions of the code and potential issues that we read and apply human judgement.
Reason being is that it can be very pendantic. Both in a good and bad way. Either flagging things that are non-concerns, or catching things my lazy human eye wouldnt have caught. e.g. A docstring mis-describing the actual shape of an object.
interesting! how do you review each other's plans? do you just copy-paste them into chat or something else?
In these times i would argue on probably getting the code reviewed by an ai agent which has been specifically trained on code quality, robustness and company or product specific code practices, that reduces the loan on manual reviewing by a large margin
that definitely helps but i'm wondering if that whole process (via an agent or via a human teammate) should be performed at the plan stage instead of the review stage. this should reduce the back and forth after the pr is opened (avoiding the delays and costs of running the CI several times)
you've identified a real structural probiem with how most teams are still operating.And we're working on it.
What are you working on? Curious to hear more
In the last couple of companies I've worked in, I've felt both overwhelmed by PRs to review and disappointed by my teammates that were just rubber-stamping my PRs
wondering if the problem is the teammates rubber-stamping or the tool/workflow being outdated
There is a huge asymmetry between who is generating the code and who is reviewing it, making the review process even more painful than it used to be.
Wondering whether if instead of reviewing PRs, we should instead move towards reviewing plans so that no code is generated before at least another person approves the plan.
Once the code is generated, the users who contributed to the plan can still review it but the fact that both participated in the plan should help reduce the asymmetry.
Feels like we need a way to collaborate and iterate on plans. Would love your thoughts on this.