From my experience, in a majority of real-world LLMs applications, prompt injection is not a primary concern.
The systems that I see most commonly deployed in practice are chatbots that use retrieval-augmented generation. These chatbots are typically very constrained: they can't use the internet, they can't execute tools, and essentially just serve as an interface to non-confidential knowledge bases.
While abuse through prompt injection is possible, its impact is limited. Leaking the prompt is just uninteresting, and hijacking the system to freeload on the LLM could be a thing, but it's easily addressable by rate limiting or other relatively simple techniques.
In many cases, for a company is much more dangerous if their chatbot produces toxic/wrong/inappropriate answers. Think of an e-commerce chatbot that gives false information about refund conditions, or an educational bot that starts exposing children to violent content. These situations can be a hugely problematic from a legal and reputational standpoint.
The fact that some nerd, with some crafty and intricate prompts, intentionally manages to get some weird answer out of the LLM is almost always secondary with respect to the above issues.
However, I think the criticism is legitimate: one reason we are limited to such dumb applications of LLMs is precisely because we have not solved prompt injection, and deploying a more powerful LLM-based system would be too risky. Solving that issue could unlock a lot of the currently unexploited potential of LLMs.
> in the academic literature, it has been suggested that the signature pattern of the DKE (Figure 1A) might be nothing more than a statistical artefact. In a typical study, people’s tendencies to under- or overestimation are analysed as a function of their ability for the task. This involves a ‘double dipping’ into the data because the task performance score is used once to rank people for ability, and then again to determine whether the self-estimate is an under- or over-estimate. This dubious double-dipping makes the analysis prone to a slippery statistical phenomenon called ‘regression to the mean’.
This is not ‘autocorrelation’, it is regression to the mean. I find the article unclear and imprecise.
For those interested in a better overview of the Dunning–Kruger effect, I recommend this short article by McIntosh & Della Sala instead:
Hey, Giskard team member here! I am around to discuss and read your feedback.
I’ve worked in particular on automatic scanning of ML models for bugs and problems, the idea was to systematically scan for general issues and automatically find segments of data on which the model performs worse than average. If you have questions, I am happy to discuss here.
Not exactly, metamorphic testing does not need an oracle. That’s actually the reason of its popularity in ML testing. It works by perturbing the input in a way that will produce a predictable variation of the output (or possibly no variation).
Take for example a credit scoring model: you can reasonably expect that if you increase the liquidity, the credit score should not decrease. In general it is relatively easy to come up with a set of assumptions on the effect of perturbation, which allows evaluating the robustness of a model without knowing the exact ground truth.
The systems that I see most commonly deployed in practice are chatbots that use retrieval-augmented generation. These chatbots are typically very constrained: they can't use the internet, they can't execute tools, and essentially just serve as an interface to non-confidential knowledge bases.
While abuse through prompt injection is possible, its impact is limited. Leaking the prompt is just uninteresting, and hijacking the system to freeload on the LLM could be a thing, but it's easily addressable by rate limiting or other relatively simple techniques.
In many cases, for a company is much more dangerous if their chatbot produces toxic/wrong/inappropriate answers. Think of an e-commerce chatbot that gives false information about refund conditions, or an educational bot that starts exposing children to violent content. These situations can be a hugely problematic from a legal and reputational standpoint.
The fact that some nerd, with some crafty and intricate prompts, intentionally manages to get some weird answer out of the LLM is almost always secondary with respect to the above issues.
However, I think the criticism is legitimate: one reason we are limited to such dumb applications of LLMs is precisely because we have not solved prompt injection, and deploying a more powerful LLM-based system would be too risky. Solving that issue could unlock a lot of the currently unexploited potential of LLMs.