The Cake Problem: when LLMs make operational promises nobody can fulfill(medium.com)
medium.com
The Cake Problem: when LLMs make operational promises nobody can fulfill
https://medium.com/@evgeny-chernyshov/the-cake-problem-when-llms-make-operational-promises-nobody-can-fulfill-4696fca70cd1
4 comments
Happy to answer questions about failure modes, where we draw the line between retrieval and operational decisions, and which constraints actually reduced risk in production. The main lesson for us was that “don’t hallucinate” is too soft for real operations. We had to replace soft prompting with hard boundaries: verified data only, checks for critical actions, and escalation before collecting fulfillment details for anything unverified.
We deployed AI agents across 25 hospitality properties and logged ~46,000 guest conversations. The main failure mode wasn’t tone or retrieval. It was “confident gap-filling”: the model promising operational outcomes nobody had verified. This post is about the production failures we saw and the constraints we added to stop them.
"if a request is not grounded in verified data, the agent must not improvise." - This is an instruction sales people across the globe also could benefit from. And the LLM is trained on content humans made. Makes sense it needs the same instruction.
Spot on. We often joke that a raw LLM acts exactly like an over-eager junior sales rep—it desperately wants to say "yes" to please the customer. Because they learned from us, they inherit the bad human habit of equating "helpfulness" with agreement. The difference is an AI will scale those broken promises instantly, which is why the constraints have to be architectural.