AI Hallucination Legal Cases Database(damiencharlotin.com)
damiencharlotin.com
AI Hallucination Legal Cases Database
https://www.damiencharlotin.com/hallucinations/
49 comments
Taking into account the output usually has the structure of being an expert output (probably because of its correctness language-wise and from a formal standpoint), I propose "bullshitting".
> I still think confabulation is a better term for what LLMs do than hallucination.
That battle is both lost and irrelevant. Don’t confuse accurate word usage with effective communication, most people don’t understand the nuance between hallucination and confabulation nor do they care. Even if you convinced everyone in the world to start using “confabulation” right now, nothing would change.
You’re doing a disservice to your own cause by insisting on this pointless weak distinction. If you truly want to make a point about the issue with LLMs in a way anyone outside of HN might pay attention, suggest a simpler and stronger word people already know: “lying”, “bullshitting”, …
You can surely object to those suggestions: “LLMs don’t lie. That would require active deception which they are incapable of, being statistical token generators”. Which is true, but:
1. There are plenty of people who both believe LLMs are intelligent and capable of deception.
2. For everyone else, “bullshitted” is no more inaccurate than “hallucinated” yet still conveys a stronger urgency and required care of operation.
That battle is both lost and irrelevant. Don’t confuse accurate word usage with effective communication, most people don’t understand the nuance between hallucination and confabulation nor do they care. Even if you convinced everyone in the world to start using “confabulation” right now, nothing would change.
You’re doing a disservice to your own cause by insisting on this pointless weak distinction. If you truly want to make a point about the issue with LLMs in a way anyone outside of HN might pay attention, suggest a simpler and stronger word people already know: “lying”, “bullshitting”, …
You can surely object to those suggestions: “LLMs don’t lie. That would require active deception which they are incapable of, being statistical token generators”. Which is true, but:
1. There are plenty of people who both believe LLMs are intelligent and capable of deception.
2. For everyone else, “bullshitted” is no more inaccurate than “hallucinated” yet still conveys a stronger urgency and required care of operation.
Confabulation is a good term for the majority of what is currently termed AI hallucinations, but there is still a good proportion that is accurately called hallucination.
For instance, if you give AI a photo and ask it to describe in detail what it seems, it will often report things that aren’t there. That’s not confabulation, that’s hallucination. But if you ask a general knowledge question with no additional context and it responds with something untrue, then that would be confabulation, I agree.
For instance, if you give AI a photo and ask it to describe in detail what it seems, it will often report things that aren’t there. That’s not confabulation, that’s hallucination. But if you ask a general knowledge question with no additional context and it responds with something untrue, then that would be confabulation, I agree.
IIRC The reason it was originally called a hallucination was due to experiments like "clone github.com/bleh/blah"
and it would give you a non-existent repository. You could list and edit the files and so on. Or you would ask it to list files in your home dir and go three directories deep and it would keep on generating.
You're not wrong in a strict sense, but you have to remember that most people aren't that strict about language
I would bet that for most people they define the words like:
Hallucination - something that isn't real
Confabulation - a word that they have never heard of
I would bet that for most people they define the words like:
Hallucination - something that isn't real
Confabulation - a word that they have never heard of
I would go one step further and suppose that a lot of people just don't know what confabulation means.
We should not bend over backwards to use language the way ignorant people do.
We should not bend over backwards to use language the way anally retentive people demand we do.
Just in case you're talking about descriptivism vs. prescriptivism.
I'm a descriptivist. I don't believe language should have arbitrary rules, like which kinds of words you're allowed to end a sentence with.
However, to be an honest descriptivist, you must acknowledge that words are used in certain ways more frequently than others. Definitions attempt to capture the canonical usage of a word.
Therefore, if you want to communicate clearly, you should use words the way they are commonly understood to be used.
I'm a descriptivist. I don't believe language should have arbitrary rules, like which kinds of words you're allowed to end a sentence with.
However, to be an honest descriptivist, you must acknowledge that words are used in certain ways more frequently than others. Definitions attempt to capture the canonical usage of a word.
Therefore, if you want to communicate clearly, you should use words the way they are commonly understood to be used.
> However, to be an honest descriptivist, you must acknowledge that words are used in certain ways more frequently than others. Definitions attempt to capture the canonical usage of a word.
True. And that's generally how they order the definitions in the dictionary, in order of usage.
For example, "an unfounded or mistaken impression or notion" is indeed the 2nd definition in M-W for "hallucination", not the first.
True. And that's generally how they order the definitions in the dictionary, in order of usage.
For example, "an unfounded or mistaken impression or notion" is indeed the 2nd definition in M-W for "hallucination", not the first.
A dictionary entry's second definition isn't necessarily an uncommonly used one. It could be up to 49% of the word's usage (assuming the dictionary has such precise statistics).
Problem is that in some fields of study / work, and in some other situations absolute clarity and accuracy are super important to avoid dangerous or harmful mistakes. Many of the sciences are that way, and A.I. is absolutely one of those sciences where communicating accurately can matter quite a lot. Otherwise you end up with massive misunderstandings about the technology being spread around as gospel truth by people who are quite simply mis-informed (like you see happening right now with all the A.I. hype).
Ignorance clusters easily. You’ll have no problem finding alike.
> Ignorance clusters easily.
So does pedantry and prickliness.
Intelligence is knowing that a tomato is a fruit, wisdom is not putting it in a fruit salad. It's fine to want to do your part to steer language, but this is not one of those cases where it's important enough for anyone to be an asshole about it.
So does pedantry and prickliness.
Intelligence is knowing that a tomato is a fruit, wisdom is not putting it in a fruit salad. It's fine to want to do your part to steer language, but this is not one of those cases where it's important enough for anyone to be an asshole about it.
Sure bud.
I like communicating with people using a shared understanding of the words being used, even if I have an additional, different understanding of the words, which I can use with other people.
That's what words are, anyway.
That's what words are, anyway.
I'm glad we can agree. I also like communicating with people using a shared understanding of the words being used, i.e. their definitions.
You might be interested to learn that the people who write down the definitions in dictionaries consider themselves to be in the business of documenting usage, not bringing commandments on stone tablets down from the mountain.
Ignorance is easy to hide behind many words.
I like calling it bullshit[0] because it's the most accurate, most understandable, and the most fun to use with a footnote
0 (featured previously on HN) https://link.springer.com/article/10.1007/s10676-024-09775-5
0 (featured previously on HN) https://link.springer.com/article/10.1007/s10676-024-09775-5
"Bending over backwards" is a pretty ignorant metaphor for this situation, it describes explicit activity whereas letting people use metaphor loosely only requires passivity.
I think "apophenia" (attributing meaning to spurious connections) or "pareidolia" (the form of aphonenia where we see faces where there are none) would have been good choices, as well.
There's a simpler word for that: lying.
It's also equally wrong. Lying implies intent. Stop anthropomorphising language models.
It's also equally wrong. Lying implies intent. Stop anthropomorphising language models.
Lying is different from confabulation. As you say, lying implies intent. Confabulation does not necessarily, ergo it's a far better word than either lying or hallucinating.
A person with dementia confabulates a lot, which entails describing reality "incorrectly";, but it's not quite fair to describe it as lying.
A person with dementia confabulates a lot, which entails describing reality "incorrectly";, but it's not quite fair to describe it as lying.
I took note, and I agree. The problem is that (as a non-native English speaker), I had to look the word up. I'm concerned that this nuance could escape people, even when they know what the word stands for.
"Making things up" is precise but wordy. "Lying" is good enough, obvious, and concise.
"Making things up" is precise but wordy. "Lying" is good enough, obvious, and concise.
A liar seeks to hide the truth; a confabulator is indifferent to the truth entirely. It's an important distinction. True statements can still be confabulations.
And why confabulation is better one of those?
It seems like these are all anthropomorphic euphemisms for things that would otherwise be described as bugs, errors (in the “broken program” sense), or error (in the “accumulation of numerical error” sense), if LLMs didn’t have the easy-to-anthropomorphize chat interface.
They aren't really bugs though in the traditional sense because all LLMs ever do is "hallucinate", seeing what we call a hallucination as something fundamentally different than what we consider a correct response is further anthropomorphising the LLM.
We just label it with that word when it statistically generates something we know to be wrong, but functionally what it did in that case is no different than when it statistically generated something that we know to be correct.
We just label it with that word when it statistically generates something we know to be wrong, but functionally what it did in that case is no different than when it statistically generated something that we know to be correct.
It’s metaphor. A hardware “bug” is occasionally due to an actual insect in the machinery, but usually it isn’t, and for software bugs it couldn’t be.
The word “hallucination” was pretty appropriate for images made by DeepDream.
https://en.m.wikipedia.org/wiki/DeepDream
The word “hallucination” was pretty appropriate for images made by DeepDream.
https://en.m.wikipedia.org/wiki/DeepDream
Imagine you have function that is called "is_true" but it only gets it right 60% of the time. We're doing this within CS/ML, so lets call that "correctness" or something fancier. In order for that function to be valuable, would we need to hit a 100% in correctness? I mean probably most of the time, yeah. But sometimes, maybe even rarely, we're fine with it being less than 100%, but still as high as possible.
So in this point of view, it's not a bug or error that it currently sits at 60%, but if we manage to find a way to hit 70%, it would be better. But in order to figure this out, we need to call this "correct for most part, but could be better" concept something. So we look at what we already know and are familiar with, and try to draw parallels, maybe even borrow some names/words.
So in this point of view, it's not a bug or error that it currently sits at 60%, but if we manage to find a way to hit 70%, it would be better. But in order to figure this out, we need to call this "correct for most part, but could be better" concept something. So we look at what we already know and are familiar with, and try to draw parallels, maybe even borrow some names/words.
> but if we manage to find a way to hit 70%, it would be better.
Yet still absolutely worthless.
> "correct for most part, but could be better" concept something.
When humans do that we just call it "an error."
> so lets call that "correctness" or something
The appropriate term is "confidence." These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating.
We can quibble over terms but more appropriately this is just "garbage." It's a giant waste of energy and resources that produces flawed results. All of that money and effort could be better used elsewhere.
Yet still absolutely worthless.
> "correct for most part, but could be better" concept something.
When humans do that we just call it "an error."
> so lets call that "correctness" or something
The appropriate term is "confidence." These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating.
We can quibble over terms but more appropriately this is just "garbage." It's a giant waste of energy and resources that produces flawed results. All of that money and effort could be better used elsewhere.
> Yet still absolutely worthless.
Depends on the context, doesn't it? Nothing is usually 100% worthless or 100% "worthy", there are grey areas in life where we're fine with "kind of right, most of the time". Are you saying these scenarios absolutely never exists in your world? I guess I'd be grateful if my life was so easy always.
Depends on the context, doesn't it? Nothing is usually 100% worthless or 100% "worthy", there are grey areas in life where we're fine with "kind of right, most of the time". Are you saying these scenarios absolutely never exists in your world? I guess I'd be grateful if my life was so easy always.
and even those confidence ratings are useless, imo. If trained with wrong data, it will report high confidence for the wrong answer. And curating a dataset is a black art in the first place
> These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating.
Why do you believe they could give you a confidence rating? They can't, at least not a meaningful one.
Why do you believe they could give you a confidence rating? They can't, at least not a meaningful one.
[deleted]
This doesn’t seem too different from my third thing, error (in the “accumulation of numerical error” sense).
These penalties need to be larger. Think of all the hours of work that using ChatGPT could save a lawyer. An occasional $2500 fine will not deter the behavior.
And this matters, because this database is only the fabrications which got caught. What happens when a decision is formulated based on AI-fabricated evidence, and that decision becomes precedent?
Here in the US, our legal system is already having its legitimacy assailed on multiple fronts. We don't need additional legitimacy challenges.
How about disbarring lawyers who present confabulated evidence?
And this matters, because this database is only the fabrications which got caught. What happens when a decision is formulated based on AI-fabricated evidence, and that decision becomes precedent?
Here in the US, our legal system is already having its legitimacy assailed on multiple fronts. We don't need additional legitimacy challenges.
How about disbarring lawyers who present confabulated evidence?
I'm presuming that if my case gets dismissed because my lawyer submits fabricated AI generated materials to the court, then I would have a very good case against my lawyer for professional misconduct. And that the damages could be very high.
This seems like a perfect use case for a legal MCP server that can provide grounding for citations. Protomated already has one[1].
[1] https://github.com/protomated/legal-context
[1] https://github.com/protomated/legal-context
Can we submit ChatGPT convo histories??
[deleted]
So what's the amount of cases were it was wrong but no one checked?
Good point. People putting the least amount of effort into their job that they can get away with is universal, judges are no more immune to it than lawyers.
Hallucination - A hallucination is a false perception where a person senses something that isn't actually there, affecting any of the five senses: sight, sound, smell, touch, or taste. These experiences can seem very real to the person experiencing them, even though they are not based on external stimuli.
Confabulation - Confabulation is a memory error consisting of the production of fabricated, distorted, or misinterpreted memories about oneself or the world. It is generally associated with certain types of brain damage or a specific subset of dementias.