I'm working on _prompt injection_, the problem where LLMs can't reliably distinguish between the user's instructions and untrusted content like web search results.
Generally the threat model is that a trusted user is trying to get untrusted data into the system. E.g. you have an email monitor that reads your emails and takes certain actions for you, but that means it's exposed to all your emails which may trick the bot into doing things like forwarding password resets to a hacker.
There have been attempts like https://arxiv.org/pdf/2410.09102 to do this kind of color-coding but none of them work in a multi-turn context since as you note you can't trust the previous turn's output
I doubt Comet was using any protections beyond some tuned instructions, but one thing I learned at USENIX Security a couple weeks ago is that nobody has any idea how to deal with prompt injection in a multi-turn/agentic setting.
After reading Judith Butler for a class in college, reading "Professor of Parody" was such a breath of fresh air. Nussbaum is a clear thinker who doesn't take BS kindly.
Bosses who harass employees and a culture that encourage such things present a much larger risk to the company than a single employee's complaints, so in a well-functioning company their incentives align with policing these issues. But it's still worth keeping in mind that their incentives are not your incentives, and if they have given up on keeping the company culture healthy, their interests will be oppossed to yours.
Just published a blog post a few minutes ago: https://alexcbecker.net/blog/prompt-injection-benchmark.html