Infrastructure provisioning is a key ingredient of agentic AI viruses: https://www.ericburel.tech/blog/ai-virus-agent
This may be the first steps of the worst wave of spamming campaign ever seen on the Internet. We'll need to reinvent how we connect and communicate via computers.
I've been working on making the "lethal trifecta" concept more popular in France. We should dedicate a statue to Simon Wilinson: this security vulnerability is kinda obvious if you know a bit about AI agents but actually naming it is incredibly helpful for spreading knowledge.
Reading the sentence "// indirect prompt injection via email" makes me so happy here, people may finally get it for good.
The thing is running it onto your machine is kinda the point. These agents are meant to operate at the same level - and perhaps replace - your mail agent and file navigator. So if we sandbox too much we make it useless.
The compromise being having separate folders for AI, a bit like having a Dropbox folder on your machine with some subfolders being personal, shared, readonly etc.
Running terminal commands is usually just a bad idea though in this case, you'd want to disable that and instead fine tune a very well configured MCP server that runs the commands with a minimal blast radius.
Before using make sure you read this entirely and understand it:
https://docs.openclaw.ai/gateway/security
Most important sentence: "Note: sandboxing is opt-in. If sandbox mode is off"
Don't do that, turn sandbox on immediately.
Otherwise you are just installing an LLM controlled RCE.
There are still improvements to be made to the security aspects yet BIG KUDOS for working so hard on it at this stage and documenting it extensively!! I've explored Cursor security docs (with a big s cause it's so scattered) and it was nothing as good.
Related: https://news.ycombinator.com/item?id=46223311
Switching to paid version was already problematic as it uses GCP account system which doesn't have a spending limit, and API keys do not have an expiration date. So the free offer was great for freelancers and SMEs and yet the paid version was the worst possible scenario you can imagine for same freelancers and SMEs. OpenRouter + free models and the increased rate limit you get after buying 10 credits (10€) is my current favourite choice for learning/teaching.
Microsoft is using the deep penetration of SharePoint in companies to sell Copilot license. At least in France it's well and alive and I see much more Copilot licenses than actual OpenAI uses.
Yeah I mean you can replace sandboxing buy other safe alternatives but the idea is the same, the generated code has to be considered as 100% untrusted. Supply chain attacks are especially nasty.
Talking about the debt of a system prompt feels really weird. A system prompt tied to an LLM is the equivalent of crafting a new model in the pre-LLM era. You measure their success using various quality metrics. And you improve the system prompt progressively to raise these metrics. So it feels like bandaid but that's actually how it's supposed to work and totally equivalent to "fixing" a machine learning model by improving the dataset.
You can have an agent that focuses on studying the interactions. What you're saying is that an AI cannot find every security issue but neither do humans otherwise we wouldn't have security breaches in the first place. You are describing a relatively basic agentic setup mostly using your AI-assisted text editor but a commercial security bot is a much more complex beast hopefully. You replace context by memory and synthesis for instance, the same way our brain works.
"You can investigate this yourself by putting a logging proxy between the claude code CLI and the Anthropic API using ANTHROPIC_BASE_URL" I'd be eager to read a tutorial about that I never know which tool to favour for doing that when you're not a system or network expert.
I call that self-destructive prompting in the sense that you use AI to output programs that replace calling the AI in the future. The paper seems to indicate that this also brings much better results. However it's subject to attacks as running generated code is usually unsafe. A sandbox has to be used, major agentic AI players are providing some solutions, like Langchain sandbox released earlier this year.
I am pretty sure you can figure massive loopholes like how it's legal to train the model on stolen data but not to steal data etc.
For instance advertisers can push model benchmarks that favours some opinions, based on a biased selection of research papers.
I think we've only seen the beginnings of what intricate business models can be figured for an AI company, it's much more convoluted than a search engine or even a social network.
"that could redefine the web economy" I don't think that ads in ChatGPT are that disruptive, it's just another channel. I think ChatGPT apps are an order magnitude more game changing, as they are not a new markting channel but a new distribution channel for software. Your next ad will still be an ad, but your next SaaS might be a ChatGPT App.