FWIW Claude Code Opus 4.5 ranks ~71% accuracy on the OpenSSF CVE Benchmark that we ran against DeepSource (https://deepsource.com/benchmarks).
We have a different approach, in that we're using SAST as a fast first pass on the code (also helps ground the agent, more effective than just asking the model to "act like a security researcher"). Then, we're using pre-computer static analysis artifacts about the code (like data flow graphs, control flow graphs, dependency graphs, taint sources/sinks) as "data sources" accessible to the agent when the LLM review kicks in. As a result, we're seeing higher accuracy than others.
Haven't gotten access to this new feature yet, but when we do we'd update our benchmarks.
It's $8/100K lines of code. Since we're using a mix of models across our main agent and sub-agents, this normalizes our cost.
> I could easily see hitting 10k+ LOC on routine tickets if this is being run on each checkpoint. I have some tickets that require moving some files around, am I being charged on LOC for those files? Deleted files? Newly created test files that have 1k+ lines?
We basically look at the files changed that need to be reviewed + the additional context that is required to make a decision for the review (which is cached internally, so you'd not be double-charged).
That said, we're of course open to revising the pricing based on feedback. But if it's helpful, when we ran the benchmarks on 165 pull requests [1], the cost was as follows:
- Autofix Bot: $21.24
- Claude Code: $48.86
- Cursor Bugbot: $40/mo (with a limit of 200 PRs per month)
We have several optimization ideas in mind, and we expect pricing to become more affordable in the future.
We haven't included Gemini Code Assist or Gemini CLI's code review mode in our benchmarks[1] (we should do that), but functionally, it'll do the same thing as any other AI reviewer. Our differentiator is that since we're using static analysis for grounding, you'll see more issues with lower false positives.
We also do secrets detection out of the box, and OSS scanning is coming soon.
I have a product which is a direct competitor of one of YC11 companies. We have pretty decent traction, and we are solving the same problem significantly different from them. What are our chances? Does YC accept competing companies?
We have a different approach, in that we're using SAST as a fast first pass on the code (also helps ground the agent, more effective than just asking the model to "act like a security researcher"). Then, we're using pre-computer static analysis artifacts about the code (like data flow graphs, control flow graphs, dependency graphs, taint sources/sinks) as "data sources" accessible to the agent when the LLM review kicks in. As a result, we're seeing higher accuracy than others.
Haven't gotten access to this new feature yet, but when we do we'd update our benchmarks.