Using AI to secure AI(mattsayar.com)
mattsayar.com
Using AI to secure AI
https://mattsayar.com/letting-inmates-run-the-asylum-using-ai-to-secure-ai/
33 comments
Ah, then you’ll want to check out xbow -
https://xbow.com/blog
I believe some folks here (moyix) are active with the project.
https://xbow.com/blog
I believe some folks here (moyix) are active with the project.
Currently living through a great litmus test of competency versus luck by company leaders
We’ve kinda solved the detection of issues. what we still lack is understanding what’s important.
I think an underappreciated use case for LLMs is to contextualize security issues.
Rather than asking Claude to detect problems, I think it’s more useful to let it figure out the context around vulnerabilities and help triage them.
(for better or worse, I am knee-deep in this stuff)
I think an underappreciated use case for LLMs is to contextualize security issues.
Rather than asking Claude to detect problems, I think it’s more useful to let it figure out the context around vulnerabilities and help triage them.
(for better or worse, I am knee-deep in this stuff)
Here's a better option -- what we've been working on at Snyk.
- Take something like Cursor and plug the Snyk MCP server into it: https://docs.snyk.io/integrations/developer-guardrails-for-a... (it has a one-click install) - Then, either within your project or via global settings, create some human-language rules for your AI code editor to use (this works basically the same between all editors: Claude Code, Cursor, Windsurf, etc...)
For example, a rule might state:
"If you add or change any code, run a Snyk Code scan on the modified files then fix the detected vulnerabilities. When you're done fixing them, perform another scan to ensure they're fixed, and if not, keep iterating until the code is secure."
Obviously, there are other rules you can use here, such as using Snyk's open source dependency testing to identify vulns in third-party dependencies and handle package updates/rewrites/etc., but you get the idea.
This works insanely well -- I've been playing around with it for a while now and we're getting close to rolling this out to all of our users in a major way =)
The best part about it is that you can just "vibe code" whatever you want, and you get really accurate static analysis security testing incorporated by default automagically.
I recorded a little video here that walks through this in-depth (https://www.youtube.com/watch?v=hQtgR1lTPYI), if you want to see the part I'm referencing, jump to 20:09 =)
- Take something like Cursor and plug the Snyk MCP server into it: https://docs.snyk.io/integrations/developer-guardrails-for-a... (it has a one-click install) - Then, either within your project or via global settings, create some human-language rules for your AI code editor to use (this works basically the same between all editors: Claude Code, Cursor, Windsurf, etc...)
For example, a rule might state:
"If you add or change any code, run a Snyk Code scan on the modified files then fix the detected vulnerabilities. When you're done fixing them, perform another scan to ensure they're fixed, and if not, keep iterating until the code is secure."
Obviously, there are other rules you can use here, such as using Snyk's open source dependency testing to identify vulns in third-party dependencies and handle package updates/rewrites/etc., but you get the idea.
This works insanely well -- I've been playing around with it for a while now and we're getting close to rolling this out to all of our users in a major way =)
The best part about it is that you can just "vibe code" whatever you want, and you get really accurate static analysis security testing incorporated by default automagically.
I recorded a little video here that walks through this in-depth (https://www.youtube.com/watch?v=hQtgR1lTPYI), if you want to see the part I'm referencing, jump to 20:09 =)
Next: using AI to sue AI.
I saw in Trump v. Murdoch case that an AI company has filed a motion to be allowed to analyze the filings and report back to the court on anything that looks shady. It's only a small step from there to having AI bulk sue people on behalf of a law firm.
This has already been leading to some incredible profits for security companies like mine.
So please, don’t be too loud about how terrible it is :)
So please, don’t be too loud about how terrible it is :)
The quotation is more impactful in the original Latin: Quis custodiet ipsos custodes?
custodes[.]ai would be a great startup name
is this the Blackwall from Cyberpunk, kinda reminds me of that.
who watches the watch man?
According to my company's senior leadership there's nothing the magic dust of AI can't solve. Even problems with AI can be solved by more AI
This is where it gets fun.
We're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance. It's going to be great watching c-suite and VPs who are drunk on the sauce accept AI in these positions and get golden parachutes when the business ends up facing a massive external audit, fraud, and the possibility of bankruptcy.
We're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance. It's going to be great watching c-suite and VPs who are drunk on the sauce accept AI in these positions and get golden parachutes when the business ends up facing a massive external audit, fraud, and the possibility of bankruptcy.
Pfft. The hammer will come down IT leadership, not execs.
>IT Leadership
You nailed it. Ive found that HN users in general have terrible understanding of how power dynamics work. Most seem to want to jam some sort of logic outcome to a situation that always only has one outcome. Those with power decide the outcome.
You nailed it. Ive found that HN users in general have terrible understanding of how power dynamics work. Most seem to want to jam some sort of logic outcome to a situation that always only has one outcome. Those with power decide the outcome.
Word. It’s true until events overtake them. But until then, the dominant understanding of a problem is that which preserves the current power structure.
And that’s why the C-level AI mania is so fascinating - preserving the status quo usually means rejecting or controlling change. But with AI they are embracing something that could eat their status, presumably out of legitimate fear of the alternative.
And that’s why the C-level AI mania is so fascinating - preserving the status quo usually means rejecting or controlling change. But with AI they are embracing something that could eat their status, presumably out of legitimate fear of the alternative.
Ah, I hadn't considered that last bit. That is indeed telling.
The status quo is broken. It's a wobbling top. It's no secret; for all they benefit from it, most CEOs know that this isn't sustainable. For better or for worse, change is coming. Perhaps for some, embracing AI is an attempt to get ahead of that.
The status quo is broken. It's a wobbling top. It's no secret; for all they benefit from it, most CEOs know that this isn't sustainable. For better or for worse, change is coming. Perhaps for some, embracing AI is an attempt to get ahead of that.
There’s always some level of change coming - e.g. the petrochemical industry giants attempting to get into renewables, or the US healthcare absorbing Obamacare and thriving, or the industrialization of organic dairy production. This one feels different.
I think one element is that AI can be a very effective bullshit generator, and most CEOs and middle managers are deploying some amount of bullshit all day long. So they see a new player on the field who undercuts their strengths and they are responding existentially.
I think one element is that AI can be a very effective bullshit generator, and most CEOs and middle managers are deploying some amount of bullshit all day long. So they see a new player on the field who undercuts their strengths and they are responding existentially.
IT leadership will blame their subordinates, the ones that knew better - somehow in these things it's always the people who should be able to say "I told you so" that get the blame.
Thinking about pivoting to pentesting
>> nWe're on the precipice of being able to install AI into positions of business critical processes. Hiring, billing, sales, and compliance
We're already there. Have been for several years now. I was doing RPA (robotic process automation) for about 4 years in a corporate environment. It went from, "Lets automate these mundane tasks" to "How can we create a billing platform that can be totally automated?". This was back in 2021, just for reference.
>> It's going to be great watching c-suite and VPs who are drunk on the sauce
Hopefully this will be a cautionary tale of what happens when they do?
https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai...
UnitedHealth Group Inc (UNH.N), uses an artificial intelligence algorithm that systematically denies elderly patients' claims for extended care such as nursing facility stays, according to a proposed class action lawsuit, filed on Tuesday.
Family members of two now-deceased UnitedHealth beneficiaries sued the insurer in federal court in Minnesota, saying they were forced to pay out of pocket for care that doctors said was medically necessary.
We're already there. Have been for several years now. I was doing RPA (robotic process automation) for about 4 years in a corporate environment. It went from, "Lets automate these mundane tasks" to "How can we create a billing platform that can be totally automated?". This was back in 2021, just for reference.
>> It's going to be great watching c-suite and VPs who are drunk on the sauce
Hopefully this will be a cautionary tale of what happens when they do?
https://www.reuters.com/legal/lawsuit-claims-unitedhealth-ai...
UnitedHealth Group Inc (UNH.N), uses an artificial intelligence algorithm that systematically denies elderly patients' claims for extended care such as nursing facility stays, according to a proposed class action lawsuit, filed on Tuesday.
Family members of two now-deceased UnitedHealth beneficiaries sued the insurer in federal court in Minnesota, saying they were forced to pay out of pocket for care that doctors said was medically necessary.
Going through that now. I’m part of the Chernobyl clean up team already. Mostly because my part of the org is the only one with a positive ROI and isn’t a fucked up mess still so I’ve got plenty of time to deal with everyone else’s problems.
How did we get in this situation? Avoid all the fucking fads.
How did we get in this situation? Avoid all the fucking fads.
We must work at the same company. LOL
ohdeargodno(2)
I'll pay more attention when they start finding vulnerabilities in commonly used, more complex applications.