Created an open-source benchmark for code security scanners and ran a bunch of them along with LLMs on real vulnerable code. Fable 5 is on there also as of yesterday, but that's the gated public model. The one we all wants to see is Mythos 5, and it's locked to a handful of vetted orgs.
So does anyone here have access to Mythos 5? And can run it against the benchmark.
Would genuinely like to see what it scores and at what cost.
This is one of the most practical breakdowns I’ve seen for a while. The spec.md as a living architecture map is smart, and documenting auth guard pattern sites as new modules get added is exactly the kind of thing that prevents issues creeping in.
The bit I’d push on: do your reviewer agents catch logic errors… things like a double negative auth check or a race condition in a payment flow. Those usually pass a check because code looks intentional and clean. Curious whether your reviewers are prompted specifically for security logic or more for spec conformance?
“Don’t merge code you don’t understand” is the right closer. Most setups don’t force that discipline cause people dont have the knowledge :)
Author here. The finding that surprised me most while writing this wasn’t the breach numbers. It was the Stanford result: developers with AI assistance introduced more flaws than those without, and felt more confident about their code. The confidence gap is the problem, not just the code quality.
The LLM secret predictability angle is something I’m still digging into and will be a separate article. There’s a lot more to it than I could cover here.
Genuinely curious: for anyone shipping vibe-coded projects, are you actually running any kind of security check before it goes live? Prompting the AI for a review, using a scanner, doing it manually, or just crossing your fingers? And if you are using an agent workflow for it, what does that look like? Any specific agent skills or tools you’ve found useful versus just adding noise?
That's a great question. This is how I would think about it:
The number of vulnerabilities by itself doesn't mean much. It has more to do with the size of the codebase and the attack surface than with the quality of the code. There is a big difference between 10 findings in 500 lines and 10 findings in 500k lines.
What matters more:
1. How bad it is and how easy it is to use. An auth bypass is not the same as a timing attack in theory. Check to see if the vulnerabilities are in code paths that can be reached in your deployment.
2. The strongest signal is the maintainer's response. How quickly do they reply? Do they take the results seriously or ignore them? A project that fixes problems quickly and gets people involved in a good way is much better than one that has no findings and no security process. For LangFuse specifically, they agreed with two of the findings and said that two of them were acceptable risks. This is a reasonable response. It's worth following up on the V4 non-response, but maintainers are busy and things get missed.
3. The kind of bugs is important. It's normal for any codebase to have logic errors, like the ones we found. You don't want to see the same type of vulnerability happen over and over again, because that means there is a systemic gap.
The only reason a project shows up in our results is because it's popular enough for us to look at it. I'd be more worried about projects that have never been looked at for security.
After reading about the ClawHavoc campaign and seeing how fast malicious
skills were spreading on ClawHub (1,100+ at last count), I figured it
would be useful to have something where people can actually practice
telling the difference between a legit skill and a bad one.
The game gives you realistic skill snippets. Some are safe, some are
modeled on real attack patterns - fake driver installs, hidden bash
execution, credential pass-through to the LLM context window. You
classify each one under time pressure and get feedback on what you
missed and why.
5 rounds, runs in the browser, no signup.
Happy to talk about the attack patterns or how I put the scenarios
together.
Author here. We built a security scanner called Kolega that does semantic analysis instead of pattern matching. To see if it actually worked, we ran it against 45 open source projects and reported what it found through responsible disclosure.
225 vulnerabilities. 41 reviewed by maintainers so far, 37 accepted, 4 rejected. 90% acceptance rate.
The bugs weren't exotic. They were things like:
if not user_id is not None - a double negative in Phase that means the permission check never runs. Nine auth bypasses total.
torch.load() without weights_only=True in vLLM - RCE via pickle deserialization in one of the most popular inference frameworks.
RestrictedPython sandbox in Agenta where __import__ was explicitly added to safe_builtins. Four different escape routes to arbitrary code execution.
SQL injection in NocoDB's Oracle client - Semgrep scanned the same codebase and found 222 issues, 208 of which were false positives, and missed this one entirely.
The interesting part to me wasn't that we found bugs. It's that these are all syntactically correct - the code compiles, runs, looks fine in review. The problems are semantic. No pattern matcher catches not X is not None because it's valid Python. You have to understand what the developer intended.
135 findings are still waiting on maintainer response. 4 were rejected - some we thought were exploitable, maintainers disagreed. We document those too.
Happy to discuss specifics on any of the projects or argue about methodology.