From what I understand, the copy fail bug was found by researcher who noticed something weird and then using AI to scan the codebase for instances where that becomes a problem.
I bet that with a slightly looser prompt/harness, the LLM could have found these twin bugs too.
Yet at the same time, I also think that if the human researcher had manually scanned the code, he'd have noticed these bugs too.
FWIW I do think LLMs are great tools for finding vulnerabilities in general. Just that they were visibly not optimally applied in this case.
The RxRPC one is definitely a different root cause (although caused by a very similar mistake).
For the ESP one it's a bit harder to tell. I don't think the wrong thing was fixed, just that there was a very similar bug in almost the same spot. Could be wrong about that though.
This is very similar in root cause and exploitation to Copy Fail.
Which illustrates pretty well something that's lost when relying heavily on LLMs to do work for you: exploration.
I find that doing vulnerability research using AI really hinders my creativity. When your workflow consists of asking questions and getting answers immediately, you don't get to see what's nearby. It's like a genie - you get exactly what you asked for and nothing more.
The researcher who discovered Copy Fail relied heavily on AI after noticing something fishy. If he had to manually wade through lots of code by himself, he would have many more chances to spot these twin bugs.
At the same time, I'm pretty sure that by using slightly less directed prompting, a frontier LLM would found these bugs for him too.
It's a very unusual case of negative synergy, where working together hurt performance.
System partitions being non-writable has nothing to do with the vulnerability - it allows modifying the cache of any file that you can open for reading.
Not using setuid anywhere means you'd have to build a slightly more clever exploit, but it's still trivial - just modify some binary you know will run as root "soon".
But... I didn't check, but IIRC the untrusted_app secontext that apps run in is not allowed to open AF_ALG sockets - so you can't directly trigger the vulnerability as a malicious app. Although it might be possible in some roundabout way (requesting some more privileged crypto service to do so).
Yeah, totally agree now that I've looked into it more.
> If OSS models are equally up to the task, why not find novel vulnerabilities?
To be fair, in the same blog post Anthropic mentioned costs in the tens of thousands of dollars per project looked at it. So it's a big ask to do an experiment that compares. Would love to see it though.
> Open source models found the same bugs? Sure, if you tell them "here is a for which may contain a vulnerability, look for a big in how function XYZ handles ABC"
In one of Anthropic's blog post, they describe that that's basically what they did too. They run the agent many times, each time specifying a different file to focus on. [1]
From my experience as a security researcher, manually finding a fishy file and sicking even sonnet 4.5 yields great results for most memory corruption bugs.
No comments otherwise. I don't have a clue as to who that guy is, and I haven't watched the video yet. You might be right overall.
Security efforts are not evenly distributed, even within a single project. This includes both the thinking that the developers put in, and the scrutiny given to a piece of code by researchers.
The initial batch of publicly disclosed vulnerabilities by Mythos demonstrates that perfectly. None of the bugs themselves are especially interesting or complex, in my opinion. They were found by applying effort to a very large amount of code which included under-scrutinized areas, where bugs hid. Yes, even in projects like Linux and OpenBSD there are many pieces of code that aren't that properly vetted, because of the finite amount of developer/researcher time allotted.
The fact that this effort is much cheaper does indeed change things. But really strong sandboxing solutions, such as gvisor or firecracker, do a really good job of having very little attack surface, all of which is heavily scrutinized.
Until we see more of the bugs that were found, it remains to be seen whether or not the post's premise about sandboxes is correct.
I bet that with a slightly looser prompt/harness, the LLM could have found these twin bugs too.
Yet at the same time, I also think that if the human researcher had manually scanned the code, he'd have noticed these bugs too.
FWIW I do think LLMs are great tools for finding vulnerabilities in general. Just that they were visibly not optimally applied in this case.