Both show up as PRs. Both require manual triage. The kill switch treats them
identically by blocking both.
What maintainers actually need is automated risk classification at the gate.
We built a GitHub Action (codeguard-action, MIT) that does this:
- Parses the diff, identifies what zones are touched
- Classifies risk L0 through L4 based on what changed (not who submitted)
- Runs proportional AI review (1-3 models depending on risk tier)
- Posts structured findings to the PR
- Seals everything into a hash-chained evidence bundle
The "1 in 10 AI PRs is legitimate" stat from the GitHub discussion tells me
9 out of 10 could be auto-filtered before a maintainer ever sees them. That's
not a ban - it's triage.
Daniel Stenberg shouldn't have had to kill curl's bug bounty. He needed a
filter that could tell the difference between a real vulnerability report and
AI-generated noise. Risk classification solves this without shutting down the
program. When AI increases code volume by 10x but reviewer count stays flat, you
don't need better review tools. You need risk triage. Not every PR deserves
the same attention:
- Typo fix to a README? L0. Auto-approve with an evidence log.
- New utility function with tests? L1. One model scans it, posts findings.
- Changes to auth middleware or payment flow? L3. Three models have to reach
consensus before a human even looks at it.
- Production deployment config? L4. Models + mandatory human sign-off.
We've been building this (codeguard-action on GitHub, MIT licensed) - a
GitHub Action that classifies PR risk, runs multi-model review proportional
to that risk, and produces a cryptographic evidence bundle proving what was
checked. The evidence is hash-chained and independently verifiable offline
with a separate tool.
The point isn't to replace human reviewers. It's to stop burning them out
on L0-L1 changes so they have capacity for the L3-L4 ones that actually
matter.
The 786-PR-backlog problem mentioned upthread isn't a review problem. It's
a triage problem.
The question auditors actually ask isn't "did your tool catch this bug?" It's "can you prove this change was reviewed, by whom, and that the code didn't change between review and merge?"
None of the tools benchmarked here produce verifiable evidence. They produce comments. A green checkmark on a PR tells you someone clicked a button. It doesn't tell you what they saw, whether the diff changed after review, or what risk level the change carried.
We took a different approach: instead of building another AI reviewer, we built a governance layer that wraps whatever review process you already use. Every PR gets a cryptographically sealed evidence bundle -- the exact diff, risk tier (L0-L4), findings, and a SHA-256 hash chain. Verifiable offline with one command. Open source, Apache 2.0.
https://github.com/DNYoussef/codeguard-action
Not a replacement for code review tools. An audit trail for them.