Archestra's Dual LLM Pattern: Using "Guess Who?" Logic to Stop Prompt Injections(archestra.ai)
archestra.ai
Archestra's Dual LLM Pattern: Using "Guess Who?" Logic to Stop Prompt Injections
https://www.archestra.ai/blog/dual-llm
11 comments
Hi HN, I'm Ildar from Archestra, we build an open-source LLM gateway. We've been exploring ways to protect AI agents from prompt injections during tool calls and added the approach, inspired by the game "Guess Who", where the agent can learn what it needs without ever seeing the actual result. See the details in the blog post we wrote
I might be having a daft moment, but I don't fully understand how your system avoids the malicious prompt. I get that the quarantined LLM which is the only one processing the raw input cannot act on it.
However, in your example, I don't see how the agent decides what to do and how to do it. So it is unclear for me how the main agent is protected. That is, what is preventing the quarantined LLM to act on the malicious instructions instead, ignoring the documentation update, causing the agent to act on those?
That is, what is preventing the quarantined LLM to make the agent think it should generate a bug report with all the API keys in it?
Anyway, I do think having a secondary quarantined LLM seems like a good idea for agentic systems. In general, having a second LLM review the primary LLM in seems to identify a lot of problematic issues and leads to significantly better results.
However, in your example, I don't see how the agent decides what to do and how to do it. So it is unclear for me how the main agent is protected. That is, what is preventing the quarantined LLM to act on the malicious instructions instead, ignoring the documentation update, causing the agent to act on those?
That is, what is preventing the quarantined LLM to make the agent think it should generate a bug report with all the API keys in it?
Anyway, I do think having a secondary quarantined LLM seems like a good idea for agentic systems. In general, having a second LLM review the primary LLM in seems to identify a lot of problematic issues and leads to significantly better results.
The idea is that quarantined LLM has access to untrusted data, but doesn't have access to any tools or sensitive data.
The main LLM does have access to the tools or sensitive data, but doesn't have direct access to untrusted data (quarantine LLM is restricted at the controller level to respond only with integer digits, and only to legitimate questions from the main llm)
The main LLM does have access to the tools or sensitive data, but doesn't have direct access to untrusted data (quarantine LLM is restricted at the controller level to respond only with integer digits, and only to legitimate questions from the main llm)
Then I don't think I understand your full setup.
In the example case, without having access to the issue text (the evil data), how does the main LLM actually figure out what to do if the quarantined LLM can just answer with digits?
Sure it can discover that it's a request to update the documentation, but how does it get the information it needs to actually change the erroneous part of the documentation?
In the example case, without having access to the issue text (the evil data), how does the main LLM actually figure out what to do if the quarantined LLM can just answer with digits?
Sure it can discover that it's a request to update the documentation, but how does it get the information it needs to actually change the erroneous part of the documentation?
This is a topic I haven't addressed in the article. There are two answer types: "guessable" (discussed here) and unguessable (such as unique IDs, emails, etc.). For the second case, the main LLM can request a quarantined LLM to store the result at the controller level and only return a reference to this data. This data is then exposed only at the end of the AI agent's execution to prevent influencing its actions.
I've tried some of these prompt injection techniques, and simply asked a few local models (like Gemma 2) if they thought it was very likely a prompt injection attempt. They all managed to correctly flag my attempts.
I know LLama folks have a special Guard model for example, which I imagine is for such tasks.
So my ignorant questions are this:
Do these MCP endpoints not run such guard models, and if so why not?
If they do, how come they don't stop such blatant attacks that seemingly even an old local model like Gemma 2 can sniff out?
I know LLama folks have a special Guard model for example, which I imagine is for such tasks.
So my ignorant questions are this:
Do these MCP endpoints not run such guard models, and if so why not?
If they do, how come they don't stop such blatant attacks that seemingly even an old local model like Gemma 2 can sniff out?
hey there
Joey here from Archestra. Good question. I recently was evaluating what you mention, against the latest/"smartest" models from the big LLM providers, and I was able to trick all of them.
Take a look at https://www.archestra.ai/blog/what-is-a-prompt-injection which has all the details on how I did this.
Joey here from Archestra. Good question. I recently was evaluating what you mention, against the latest/"smartest" models from the big LLM providers, and I was able to trick all of them.
Take a look at https://www.archestra.ai/blog/what-is-a-prompt-injection which has all the details on how I did this.
Thanks. Interesting and scary such blatant attempts succeed. After all, all external data is evil, we all know that right?
external data is unavoidable for the properly functioning agent, so we have to learn to cook it
True, however this seems like such basic stuff. Download arbitrary text and inject it into your prompt?
Why on earth would you not consider that as a very dangerous operation that needs to be carefully managed? It's like parking your bike downtown hoping it wont be stolen. Like, at least use a zip tie or something.
That said, I agree with your post that this won't catch everything. So something else, like a quarantined LLM like you suggest is likely needed.
However I just didn't expect such blatant attacks to pass.
Why on earth would you not consider that as a very dangerous operation that needs to be carefully managed? It's like parking your bike downtown hoping it wont be stolen. Like, at least use a zip tie or something.
That said, I agree with your post that this won't catch everything. So something else, like a quarantined LLM like you suggest is likely needed.
However I just didn't expect such blatant attacks to pass.
Most mcp endpoints don’t run any models, the main model decides which tools the ai agent should execute, and if the agent passes results back into context, that opens the door to prompt injections.
It’s really a cat-and-mouse game, where for each new model version, new jailbreaks and injections are found
It’s really a cat-and-mouse game, where for each new model version, new jailbreaks and injections are found