OpenAI: Models are programmed to make stuff up instead of admitting ignorance(theregister.com)
theregister.com
OpenAI: Models are programmed to make stuff up instead of admitting ignorance
https://www.theregister.com/2025/09/17/openai_hallucinations_incentives/
3 comments
This is why I just can't fully get behind the idea that AI is going to be the panacea the hype merchants keep claiming, at least in this era: making stuff up incentivizes users to prompt again, while returning "I don't know" tends to be a conversation ender. They're not incentivized to be accurate, they're incentivized to give the appearance of competence to the point that it keeps users coming back.
Tufte said it best: There are only two industries that call their customers 'users': illegal drugs and software.
Tufte said it best: There are only two industries that call their customers 'users': illegal drugs and software.
This is the problem Aloe (a state-of-the-art agent designed and built by a cognitive scientist) is solving. This article out yesterday delves into exactly that: https://puck.news/is-aloe-the-first-self-building-ai/?sharer...
I don't really think people want these things to say "I don't know" - they want it to know.
That's obviously not reasonable for everything but I bet a lot of hallucinations ARE things where the model should be able to know or figure out. Most people are asking questions with well-known answers.
But I would guess the OpenAI post is correct, fundamentally they are trained in a way that rewards guessing, which I think must make it more likely it guesses even if the answer is in its reach.