New theory suggests LLMs can understand text(quantamagazine.org)
quantamagazine.org
New theory suggests LLMs can understand text
https://www.quantamagazine.org/new-theory-suggests-chatbots-can-understand-text-20240122/
107 comments
> A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function.
That's just like, your opinion man.
I see both sides of this argument make statements like this that are clearly opinion and not backed up by facts. The truth is for now, we can't say yay or nay on whether the model has reasoning abilities, creative abilities, or a coherent internal world view. It's still a debate which is why there are academics like those in the OP trying to come up with theories that make predictions and then running experiments. It's still early days.
If you were to ask my opinion, I'd say these current models are probably closer to the stochastic parrot than an AGI, but there is enough there to start wondering, especially if we have years of improvements coming down the pipeline.
These aren't easy questions, and anyone espousing certainty now is doing themselves and others a disservice.
That's just like, your opinion man.
I see both sides of this argument make statements like this that are clearly opinion and not backed up by facts. The truth is for now, we can't say yay or nay on whether the model has reasoning abilities, creative abilities, or a coherent internal world view. It's still a debate which is why there are academics like those in the OP trying to come up with theories that make predictions and then running experiments. It's still early days.
If you were to ask my opinion, I'd say these current models are probably closer to the stochastic parrot than an AGI, but there is enough there to start wondering, especially if we have years of improvements coming down the pipeline.
These aren't easy questions, and anyone espousing certainty now is doing themselves and others a disservice.
Not all of it is an opinion.
The LLM does not have the ability to tell you what it knows and what it doesn't know about a subject. In order to get information out of it, you need to present it with a pattern that matches with something it was trained on.
An example, is that an LLM can tell you how many programming languages it knows, how much it knows about them (which one it understands best), and how many it doesn't know. If you ask it a question like "how to do X in python", it will have been trained on something that roughly matches that and give a response, but if you ask "what things do you know how to do in python", and it simply is not capable of giving a response.
I'm not sure that this means it doesn't have some form of understanding of things, and I do believe that AGI is possible to create in general with sufficient technology, and I do realize that our brains may work in a similar way, but I think LLMs are missing something when it comes to understanding and creating that we have not yet figured out.
The LLM does not have the ability to tell you what it knows and what it doesn't know about a subject. In order to get information out of it, you need to present it with a pattern that matches with something it was trained on.
An example, is that an LLM can tell you how many programming languages it knows, how much it knows about them (which one it understands best), and how many it doesn't know. If you ask it a question like "how to do X in python", it will have been trained on something that roughly matches that and give a response, but if you ask "what things do you know how to do in python", and it simply is not capable of giving a response.
I'm not sure that this means it doesn't have some form of understanding of things, and I do believe that AGI is possible to create in general with sufficient technology, and I do realize that our brains may work in a similar way, but I think LLMs are missing something when it comes to understanding and creating that we have not yet figured out.
well... what we have an an LLM doesn't is state and feedback and restraint, a sense of time.
We have an internal state of some kind, an LLM cheats having a state by having every input be the entire conversation + one new input. making the context grow overtime making that state unsustainable.
The other thing an LLM is missing is proper feedback, it receives text chunk prompts and output text chunks. since it's output will never 'change the llm's internal state', it can not have a feedback loop. It can mimic one with the trick of refeeding messages as context, but that's untenable after a certain size input.
Another is restraint, we human's don't have to respond to everything. we can choose to ignore input entirely, leave people on read, etc... LLM's respond to each input regardless of it is appropriate, and they only respond to stimuli (maybe we do as well, just subtle real world effects).
last is a sense of time, which I don't see how the LLM can manage at all. It has no clock.
Give an LLM a rich state (okay, maybe rename it too), have it be 'always on', give it a camera and microphone feed for inputs, give it a robot arm to control (pass the output of that back into the model for feedback), give it a second brain to decided to talk or not, and give it a clock too... and what comes out would look leagues better than current AI.
We have an internal state of some kind, an LLM cheats having a state by having every input be the entire conversation + one new input. making the context grow overtime making that state unsustainable.
The other thing an LLM is missing is proper feedback, it receives text chunk prompts and output text chunks. since it's output will never 'change the llm's internal state', it can not have a feedback loop. It can mimic one with the trick of refeeding messages as context, but that's untenable after a certain size input.
Another is restraint, we human's don't have to respond to everything. we can choose to ignore input entirely, leave people on read, etc... LLM's respond to each input regardless of it is appropriate, and they only respond to stimuli (maybe we do as well, just subtle real world effects).
last is a sense of time, which I don't see how the LLM can manage at all. It has no clock.
Give an LLM a rich state (okay, maybe rename it too), have it be 'always on', give it a camera and microphone feed for inputs, give it a robot arm to control (pass the output of that back into the model for feedback), give it a second brain to decided to talk or not, and give it a clock too... and what comes out would look leagues better than current AI.
I think that something - or at least one of them - might be objectivity.
An LLM has all of the what and none of the why.
An LLM has all of the what and none of the why.
> That's just like, your opinion man.
Yeah, that's why I literally said so. You are welcome to disagree with me.
My opinion isn't the whole point I intended to make here. I think there is too much focus on "wow look at the emergent behavior of this LLM!", and not enough focus on "where could this specific behavior come from, and why?" The second question may hold the key to both understanding what we have, and making something better.
Yeah, that's why I literally said so. You are welcome to disagree with me.
My opinion isn't the whole point I intended to make here. I think there is too much focus on "wow look at the emergent behavior of this LLM!", and not enough focus on "where could this specific behavior come from, and why?" The second question may hold the key to both understanding what we have, and making something better.
Do you understand how these models work under the hood?
It's literally just picking the next most likely word, based on some fancy math. There is nothing that can create new insights or anything approaching intelligence.
It's literally just picking the next most likely word, based on some fancy math. There is nothing that can create new insights or anything approaching intelligence.
I think this critique would only be fair if we more completely understood how we ourselves work under the hood.
The "fancy math" is a bunch of non-linear functions that encode something derived from series of tokens. I believe we're still exploring what the derived vector spaces "mean" for LLMs, with research inspecting smaller and simplified models to try to make sense of it all.
It's clear that LLMs can "remix" in interesting ways. "ChatGPT, write me a Shakespearean-style sonnet about how LLMs work." (https://pastebin.com/FwzqWJ5W) It's not clear to me that our creativity and intelligence is not the same or a similar process.
Beyond that, I'm not convinced there are definitions of meaning, knowledge, creativity, intelligence, etc, that are both useful and don't refer solely to the output. If I need a solution and none exists yet and something creates it, I will call that creative irrespective of process.
But I don't know anything about anything. :)
The "fancy math" is a bunch of non-linear functions that encode something derived from series of tokens. I believe we're still exploring what the derived vector spaces "mean" for LLMs, with research inspecting smaller and simplified models to try to make sense of it all.
It's clear that LLMs can "remix" in interesting ways. "ChatGPT, write me a Shakespearean-style sonnet about how LLMs work." (https://pastebin.com/FwzqWJ5W) It's not clear to me that our creativity and intelligence is not the same or a similar process.
Beyond that, I'm not convinced there are definitions of meaning, knowledge, creativity, intelligence, etc, that are both useful and don't refer solely to the output. If I need a solution and none exists yet and something creates it, I will call that creative irrespective of process.
But I don't know anything about anything. :)
Your argument is centered on a lack of understanding.
How could anyone possibly argue that God and Heaven are fiction? No one can prove they don't exist! I prefer to approach such assertions with skepticism. There is no reason for me to believe the attestation is true, so I may as well believe it to be false.
People remix expressions all the time. Any good training corpus will contain countless examples. If a statistical model can recognize the general semantic patterns that are present in human-remixed expressions, then it should be able to repeat them. So that's a thing that both humans and LLMs can do. Does that mean that LLMs are interesting and useful? Yes! Does it mean we are equivalent? I don't think so.
If I could point out specifically what an LLM can't do, then I would be 99% of the way toward building something that can.
How could anyone possibly argue that God and Heaven are fiction? No one can prove they don't exist! I prefer to approach such assertions with skepticism. There is no reason for me to believe the attestation is true, so I may as well believe it to be false.
People remix expressions all the time. Any good training corpus will contain countless examples. If a statistical model can recognize the general semantic patterns that are present in human-remixed expressions, then it should be able to repeat them. So that's a thing that both humans and LLMs can do. Does that mean that LLMs are interesting and useful? Yes! Does it mean we are equivalent? I don't think so.
If I could point out specifically what an LLM can't do, then I would be 99% of the way toward building something that can.
Yes, I understand how these models work under the hood as well as any lay person can. I worked through Andrej's video series and built my own toy model.
Your argument is missing the whole concept of emergence, which is what is at question here. How does emergence come about, under what circumstances, and what is possible in those circumstance? It's an open question.
Following your logic, one could argue that physics is basic and follows well understood rules. We can do some fancy math to predict what will happen in a given particle interaction. How would these basic particles ever give rise to life and minds with reasoning ability?
Well you and I are an existence proof that basic rules when followed on massive systems can give rise to emergent behavior.
Your argument is missing the whole concept of emergence, which is what is at question here. How does emergence come about, under what circumstances, and what is possible in those circumstance? It's an open question.
Following your logic, one could argue that physics is basic and follows well understood rules. We can do some fancy math to predict what will happen in a given particle interaction. How would these basic particles ever give rise to life and minds with reasoning ability?
Well you and I are an existence proof that basic rules when followed on massive systems can give rise to emergent behavior.
Can you explain emergence then please? Because the way you're using it, and the way people tend to use it to present this argument is as a stand in for magic and another way of saying "if we don't understand it, then anything is possible, man!".
We can probably simplify it. If we have a system and move one level up it becomes a completely different thing. For example: one molecule in space has some properties like mass, velocity, position. But move a level up and you get pressure, flow, temperature, etc.. It's a completely different thing.
The same with models. On low level there are weights, gradients, matrices. But move up and (wow!) It's nothing like it. It's coherent text or actions. Commands and responses. Put together and you have image recognition and generation from description. It's something new with it's own laws and properties. Just a bit up and we'll get coherent behavior and social creatures. For them we need 'generic' component. It's like soul for life. And LLMs could be it. At least they are the best candidates today.
The same with models. On low level there are weights, gradients, matrices. But move up and (wow!) It's nothing like it. It's coherent text or actions. Commands and responses. Put together and you have image recognition and generation from description. It's something new with it's own laws and properties. Just a bit up and we'll get coherent behavior and social creatures. For them we need 'generic' component. It's like soul for life. And LLMs could be it. At least they are the best candidates today.
Consciousness may not be emergent, it could be like the panpsychists argue. Or other forms of consciousness dualism.
My argument is an attempt to explain where emergence itself comes from. Prove me right or wrong, and you have answered your question. Until then, as you said, the question is open. Why should I pretend otherwise, and claim that the LLM is definitely intelligent? Such a position is no more than vanity.
Following your logic, the million monkeys hammering randomly on typewriters that eventually produce the full works of Shakespeare, did so not through pure randomness, but by actually understanding the plight of Romeo and Juliet and the motivation behind Hamlet.
Edit since I can't reply: "I don't like your answer, therefore you aren't engaging in good faith." lol
Edit since I can't reply: "I don't like your answer, therefore you aren't engaging in good faith." lol
> the million monkeys hammering randomly on typewriters that eventually produce the full works of Shakespeare, did so not through pure randomness, but by actually understanding the plight of Romeo and Juliet and the motivation behind Hamlet
The funny thing 'yes'. Million monkeys cannot do it by randomly typing. They cannot even randomly type at all. Each monkey is intelligent creature, they will follow some patterns. If they did manage to produce Shakspeare they definitely understood something.
Today it's easy to programmatically simulate 1m monkeys typing randomly 10 keys per second. Try it ;)
The funny thing 'yes'. Million monkeys cannot do it by randomly typing. They cannot even randomly type at all. Each monkey is intelligent creature, they will follow some patterns. If they did manage to produce Shakspeare they definitely understood something.
Today it's easy to programmatically simulate 1m monkeys typing randomly 10 keys per second. Try it ;)
Then it's the thought experiment that needs re-framing. I'll give it a shot:
At nearly every point in the universe, there is a noise: Cosmic Microwave Background Radiation.
What if you listened to the CMBR at a specific point in space, converted that signal into Unicode text, and it just happened to write Romeo and Juliet? Is that intelligence? No. That's selection bias.
So why is ChatGPT able to quote Shakespeare when asked? Because the entire text of the play is in its training corpus, and the infinite permutations of nonsensical CMBR noise is not.
At nearly every point in the universe, there is a noise: Cosmic Microwave Background Radiation.
What if you listened to the CMBR at a specific point in space, converted that signal into Unicode text, and it just happened to write Romeo and Juliet? Is that intelligence? No. That's selection bias.
So why is ChatGPT able to quote Shakespeare when asked? Because the entire text of the play is in its training corpus, and the infinite permutations of nonsensical CMBR noise is not.
I am aware of the thought experiment you are referring to, but if I witnessed a group of monkeys write the full works of Shakespeare in my lifetime then yes, I would actually consider it something worthy of investigating.
However I can see you are not engaging in this conversation in good faith so I don't think there is anything else to discus with you.
However I can see you are not engaging in this conversation in good faith so I don't think there is anything else to discus with you.
> However I can see you are not engaging in this conversation in good faith so I don't think there is anything else to discus with you.
From my perspective, it seems pretty clear that you are the one arguing in bad faith. People are allowed to disagree with you. This person presented a thoughtful critique of your argument. It made plenty of sense to me.
From my perspective, it seems pretty clear that you are the one arguing in bad faith. People are allowed to disagree with you. This person presented a thoughtful critique of your argument. It made plenty of sense to me.
> However I can see you are not engaging in this conversation in good faith...
What if this "categorize and ignore/forget" is causing humans to miss something extremely important in their midst? Bad faith is certainly possible here, but also possible is that this person genuinely believes what they say.
What if this "categorize and ignore/forget" is causing humans to miss something extremely important in their midst? Bad faith is certainly possible here, but also possible is that this person genuinely believes what they say.
I find the "it's just picking the next most likely word" argument to be not very convincing. If anything it makes me wonder if that's not all there is to my own intelligence.
Hell, try writing a paragraph backwards and tell me that you're not "just picking the next most likely word" as well.
Hell, try writing a paragraph backwards and tell me that you're not "just picking the next most likely word" as well.
My saddest observation is that I don't even have control over what comes out next, especially when talking/typing fast. "I" observe myself speaking, more than anything. Sometimes, as I'm talking, I think "that's a good point!" because I didn't consciously make it, it was just part of this "magic stream" that I'm listening to as much as the person I'm speaking to is listening. I also find letting that stream flow is when I'm at my sharpest. Same with problem solving. I point my intention at something and answers come out, through "intuition" and "creativity" which are almost entirely autonomous/subconscious.
This makes me fear getting old, where this magic machine probably fades, almost certainly leaving me pointing my intention, and starting up sentences, without much happening after. I'm afraid I'll be left with my conscious self, which I don't find as entertaining.
This makes me fear getting old, where this magic machine probably fades, almost certainly leaving me pointing my intention, and starting up sentences, without much happening after. I'm afraid I'll be left with my conscious self, which I don't find as entertaining.
It's able to perform that task because it has been fed examples. Essentially, it's the examples themselves that know how to do the task.
"Emergence occurs when a complex entity has properties or behaviors that its parts do not have on their own, and emerge only when they interact in a wider whole."
https://en.wikipedia.org/wiki/Emergence
https://en.wikipedia.org/wiki/Emergence
That doesn't sound too dissimilar from what human brains do, if you replace "math" by "spiking neural network"
Not a whole lot different than what I do as a human. I can do it a lot faster, though, allowing more exploration into the second, third, and fourth most likely word at the same time. Nobody wants to wait years for ChatGPT to finally reply, so shortcuts have to be taken.
I have never discovered any new insights, but it is apparent that the single most important attribute in discovering a new insight is time spent focused on finding a new insight. Undoubtedly, insight simply comes from executing on even more permutations than regular Joes like me take the time for until the right combination finally lands.
I have never discovered any new insights, but it is apparent that the single most important attribute in discovering a new insight is time spent focused on finding a new insight. Undoubtedly, insight simply comes from executing on even more permutations than regular Joes like me take the time for until the right combination finally lands.
"The statement that P=NP is [...]" (true or false)
The article uses the concept of irony and the word "because" as examples of fairly sophisticated next-word predictors. You can imagine others that require even more understanding.
The article uses the concept of irony and the word "because" as examples of fairly sophisticated next-word predictors. You can imagine others that require even more understanding.
I think one of the arguments here is that we do not know how thinking in the human brain works, so we can't say that something else doesn't do something similar.
These arguments are exhausting. Do you understand how neurons work under the hood? And yet, hook billions of them together in interesting ways, and look what happens.
Here are the steps to create a sentient primate:
1. Create some amino acids
2. Let the amino acids mix and recombine
3. Let the ones best at reproducing themselves outcompete the rest
4. Repeat steps 2 & 3 until satisfied
Of course, this ignores the elephant in the room. These are just _proteins_. What does it mean? What is it for? What does it know?
Clearly a mess of amino acids and derivatives can’t actually produce intelligence or understanding. At best, it can merely parrot the intelligence encoded in the base pairs.
Of course, this ignores the elephant in the room. These are just _proteins_. What does it mean? What is it for? What does it know?
Clearly a mess of amino acids and derivatives can’t actually produce intelligence or understanding. At best, it can merely parrot the intelligence encoded in the base pairs.
The question we are asking is, "Where does the complexity come from?"
A complex result can come from a complex system, or a simple system with complex input (emergence). Human genetics is a complex system.
My argument is that an LLM is a simple system. The complexity of its output comes from the complexity of its input, not the system itself.
That complexity was created by human minds, not by AI. Without the human minds, there would be no complex LLM output.
A complex result can come from a complex system, or a simple system with complex input (emergence). Human genetics is a complex system.
My argument is that an LLM is a simple system. The complexity of its output comes from the complexity of its input, not the system itself.
That complexity was created by human minds, not by AI. Without the human minds, there would be no complex LLM output.
Thanks - I think I understand you a bit better now.
I suppose I was arguing that genetics is a simple system with emergent complexity (and that perhaps LLMs could be too), but I suspect that betrays my limited understanding of genetics!
> We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it.
I think my main sticking point is the last sentence, which feels like a jump to me. For all the wisdom in your comments on complexity and invention, how does that link to awareness or understanding. Why does understanding require “ownership” of the complexity, so to speak?
To me, that’s not clear; could these be emergent properties of a system where the complexity is generated elsewhere, for instance?
I suppose I was arguing that genetics is a simple system with emergent complexity (and that perhaps LLMs could be too), but I suspect that betrays my limited understanding of genetics!
> We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it.
I think my main sticking point is the last sentence, which feels like a jump to me. For all the wisdom in your comments on complexity and invention, how does that link to awareness or understanding. Why does understanding require “ownership” of the complexity, so to speak?
To me, that’s not clear; could these be emergent properties of a system where the complexity is generated elsewhere, for instance?
Humans write. LLMs pretend to write. Really what they are doing is a statistical process called "continuation".
By training an LLM on human-written content, we are essentially restricting the problem domain from "any possible next character" to "something similar to what the humans wrote".
What the LLM does at the end of the day can be as simple and intelligent as blindly stumbling around whatever data it was given, because the domain it is stumbling around is made exclusively out of human-written content.
An LLM won't generate nonsense, because nonsense lies outside its domain: the LLM was never trained on nonsense, so there is no nonsense for it to stumble into.
An LLM can stumble into lies, offensive language, threats, incorrect arithmetic, logical fallacies, etc. These are generally presented as "limitations". The limitation is that we can't keep them out of the domain. It would be more clear to say those are limitations of the LLM creator. To the LLM itself, these limitations are a feature; though not the kind we are happy about.
Humans are able to think about our writing before we write it. That's a feature called objectivity, and LLMs don't have it. There's no obvious place to add that feature, either.
By training an LLM on human-written content, we are essentially restricting the problem domain from "any possible next character" to "something similar to what the humans wrote".
What the LLM does at the end of the day can be as simple and intelligent as blindly stumbling around whatever data it was given, because the domain it is stumbling around is made exclusively out of human-written content.
An LLM won't generate nonsense, because nonsense lies outside its domain: the LLM was never trained on nonsense, so there is no nonsense for it to stumble into.
An LLM can stumble into lies, offensive language, threats, incorrect arithmetic, logical fallacies, etc. These are generally presented as "limitations". The limitation is that we can't keep them out of the domain. It would be more clear to say those are limitations of the LLM creator. To the LLM itself, these limitations are a feature; though not the kind we are happy about.
Humans are able to think about our writing before we write it. That's a feature called objectivity, and LLMs don't have it. There's no obvious place to add that feature, either.
Hence the conclusion must be: clearly we have intelligence because God created us.
:)
:)
That sounds about right. There is definitely nothing in the _design_ of the model or in the training procedure that is already and by itself intelligent, instead it is an intelligence extraction process. The intelligence is in the data and some of that apparently gets distilled into the model.
So we should stop thinking that the intelligence is the (untrained) brain and realize that it has always been the text.
I think it makes more sense to compare the current models to unconscious thought or intuition. So the result just kind of pops up as a result of a computation, just like unconscious thought pops up for humans as a result of a (hidden) computation. Humans do conscious reasoning by bringing their thoughts into memory and keeping them there, stretching the process out over time. In some sense LLMs being able to solve some things by being instructed to think step by steps reminds me of that. So perhaps adding some rumination will create a layer on top that appears closer to what we think of as thinking.
So we should stop thinking that the intelligence is the (untrained) brain and realize that it has always been the text.
I think it makes more sense to compare the current models to unconscious thought or intuition. So the result just kind of pops up as a result of a computation, just like unconscious thought pops up for humans as a result of a (hidden) computation. Humans do conscious reasoning by bringing their thoughts into memory and keeping them there, stretching the process out over time. In some sense LLMs being able to solve some things by being instructed to think step by steps reminds me of that. So perhaps adding some rumination will create a layer on top that appears closer to what we think of as thinking.
Right, it seems incorrect to claim that "The model can generate text that it couldn’t possibly have seen in the training data" when it's been trained on all manner of humans writing about all manner of things.
It seems likely that the training data included text involving " self-serving bias, metaphor, statistical syllogism and common-knowledge physics". So, when given a prompt asking for these things; the model is able to 'Stochastic Parrot' an appropriate response.
The graph theory work is clever, but the assumptions about the Models, and about Language, seem incorrect.
It seems likely that the training data included text involving " self-serving bias, metaphor, statistical syllogism and common-knowledge physics". So, when given a prompt asking for these things; the model is able to 'Stochastic Parrot' an appropriate response.
The graph theory work is clever, but the assumptions about the Models, and about Language, seem incorrect.
> We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the LLM could simply tell us about it.
We don't train humans on noise. We train them on intangible and tangible things in the world. The very act of choosing and ordering the things is where intelligence lies. A human can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the human could simply tell us about it.
Turtles all the way down.
We don't train humans on noise. We train them on intangible and tangible things in the world. The very act of choosing and ordering the things is where intelligence lies. A human can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the human could simply tell us about it.
Turtles all the way down.
Let me put it this way: the text in the training corpus may very well contain the objective thought function. But to the LLM, that is indistinguishable from anything else. It's just another set of patterns between tokens.
From the LLM's perspective, there is only "what", and never "why". The "what" may even contain the "why"! Even so, it is always "what" to the LLM's eyes.
There is an objective difference between ideas, expression, and logic in my mind. I can't tell you precisely what that difference is made of, but I'm quite certain that LLMs are missing it.
From the LLM's perspective, there is only "what", and never "why". The "what" may even contain the "why"! Even so, it is always "what" to the LLM's eyes.
There is an objective difference between ideas, expression, and logic in my mind. I can't tell you precisely what that difference is made of, but I'm quite certain that LLMs are missing it.
And this is different from a baby how, other than that the baby takes several years to train? Do you train babies on noise? (There's a dystopian alternate-universe short story on training babies on noise to make them cattle).
Where did the first language come from?
Has anyone shown that you can leave a collection of LLMs in an empty environment, and watch them invent their own language or mathematics? I doubt that would work.
Has anyone shown that you can leave a collection of LLMs in an empty environment, and watch them invent their own language or mathematics? I doubt that would work.
Hah, you would not believe it, but I am planning to use LLM to solve math problems by training it on the posts of my blog. Of course, my blog is silly nonsense, but it will be a fun exercise and address the elephant in the room as well. We will see what results we get for some fun prompts!
Marinate the nether rod
Power drill the yippee bog
Pressure wash the quiver bone
Tokenize the training corpus
Power drill the yippee bog
Pressure wash the quiver bone
Tokenize the training corpus
I remember playing around with GPT-2. I was absolutely amazed to see that it knew some characters from Dota 2 or Supreme Commander. I thought damn this thing has a whole world model inside, because it can clearly memorize concepts and talk about then in a flexible/ approximate way. This requires remembering not just the things but also the dynamics of and relations between those things. Since it can talk about them freely, it must have done that to at least some degree. Getting to an accurate world model is then just a quantitative step. I am surprised some people still think LLMs don't have any kind of world model.
Seems to me it's a text model, not a world model. Like... they can't feel the word "hungry", but they've got a damn good idea where the word "hungry" appears in the phase space that is the set of all meaningful human utterances. Their ground truth is text, not reality.
Maybe that's why the model gets shaky when asked to reason about the real world. They are, after all, pretty great at reasoning about words (eg 'I need a word that means "a thing made out of clay", but not "pottery"' -> '"Ceramic" is a suitable word. It refers to items made from clay and hardened by heat.)
(Edit: Inspired by the article, I just asked ChatGPT4 "When a knife falls to the ground, what are the possible outcomes?" ten times. It included "it lands point first and it sticks" 2 out of 10. GPT4 via the AI included it 6 out of 10, and also included some fairly creative-but-unlikely outcomes ("If the knife falls near an electrical outlet or cord, it could potentially cause an electrical shock or fire.")).
Maybe that's why the model gets shaky when asked to reason about the real world. They are, after all, pretty great at reasoning about words (eg 'I need a word that means "a thing made out of clay", but not "pottery"' -> '"Ceramic" is a suitable word. It refers to items made from clay and hardened by heat.)
(Edit: Inspired by the article, I just asked ChatGPT4 "When a knife falls to the ground, what are the possible outcomes?" ten times. It included "it lands point first and it sticks" 2 out of 10. GPT4 via the AI included it 6 out of 10, and also included some fairly creative-but-unlikely outcomes ("If the knife falls near an electrical outlet or cord, it could potentially cause an electrical shock or fire.")).
It is largely a text-only model, but tiny amounts of information from other modalities are indeed represented in the text. The best example is GPT-4's drawing ability.
From text only it might theoretically be possible to infer to some degree how our physical world works, but it should be extremely difficult in practice. That's why we we'll need multimodal models obv. But my intuition is that understanding captioned youtubes with a transformer should work just as well as LLMs do, it's just a matter of scale. I think a 100-1000x GPT-4 sized video model could model let's say 50ms frame rate videos pretty well, in the sense that you can watch minutes of video (vs read minutes of text) and the video looks reasonably realistic and preserves coherence/context over the duration. We just don't have the amount of compute necessary yet, so we have to be smarter with multimodality.
From text only it might theoretically be possible to infer to some degree how our physical world works, but it should be extremely difficult in practice. That's why we we'll need multimodal models obv. But my intuition is that understanding captioned youtubes with a transformer should work just as well as LLMs do, it's just a matter of scale. I think a 100-1000x GPT-4 sized video model could model let's say 50ms frame rate videos pretty well, in the sense that you can watch minutes of video (vs read minutes of text) and the video looks reasonably realistic and preserves coherence/context over the duration. We just don't have the amount of compute necessary yet, so we have to be smarter with multimodality.
You mean the SVG-generation stuff? I'm not convinced that's not just another form of text. It can "speak SVG" like it can "speak German". I spent a fair amount of time asking it to draw me SVGs of fish, and the results suggested "there's always a triangle, there's always a circle", not "it's got a tail and an eye, and they're connected to the body". It didn't seem to understand the relationships between the components. Could be wrong though.
> But my intuition is that understanding captioned youtubes with a transformer should work just as well as LLMs do
Wouldn't that be amazing. Bring it on, Google.
> But my intuition is that understanding captioned youtubes with a transformer should work just as well as LLMs do
Wouldn't that be amazing. Bring it on, Google.
I know what you mean, but at some point when I played with it it could reasonably modify an image based on queries. Like, if I asked it to add smoke to the house's chimney, it would put grey blobs above the house, or if I asked it to give the cat a tail, it would put it roughly at the end. For me that's kind of a very rudimentary picture understanding, although I know it's a stretch.
Now that you mention it, I had the same experience - I asked it to draw the Mona Lisa, add a background, folded hands, stuff like that. Ok, yeah, I don't know what to make of that.
> It didn't seem to understand the relationships between the components. Could be wrong though.
There is evidence that you are wrong: https://www.youtube.com/watch?v=qbIk7-JPB2c
Unfortunately, the "alignment tax" is a heavy one.
There is evidence that you are wrong: https://www.youtube.com/watch?v=qbIk7-JPB2c
Unfortunately, the "alignment tax" is a heavy one.
Some people are born without the ability to feel pain. Do they also lack a world model?
They lack a world model that describes feeling of pain, yes.
Okay, but it’s still a world model, just slightly different in kind?
What about multi-modal models that are trained on text and images?
What about something trained on images, audio, sensorimotor inputs? At what point does that change?
Didn’t the world produce the text (albeit indirectly), just as it does the inputs to our own experience?
What about multi-modal models that are trained on text and images?
What about something trained on images, audio, sensorimotor inputs? At what point does that change?
Didn’t the world produce the text (albeit indirectly), just as it does the inputs to our own experience?
I think you've argued successfully that there's a sliding scale here. I guess the more experience they get of physics, the better they'll get at predicting physical things.
They obviously have some kind of world model.
The problem with AI is just that these models aren't as intuitive as they might seem.
The problem with AI is just that these models aren't as intuitive as they might seem.
Clearly no link to the actual research, so here it is: https://arxiv.org/abs/2307.15936
Quanta is usually careful to link to the actual papers (even if the reader has to be equally careful to find them), so it's surprising that the link is missing here. Another paper (linked from the article) is https://arxiv.org/abs/2310.17567
Yes, for some arbitrary definition of 'understanding' since there is no rigorous, concrete, agreed definition.
We're going to be on this merry go round for some time while the A(G)I hype cycle plays out. Hopefully through it all some people will keep on working on great ideas that incrementally improve the field and produce useful stuff.
We're going to be on this merry go round for some time while the A(G)I hype cycle plays out. Hopefully through it all some people will keep on working on great ideas that incrementally improve the field and produce useful stuff.
I don't think you want to use the term "arbitrary" here, since that would imply a stronger claim: that no matter how one defines "understanding" it applies. "Yes, for some specific, narrow definition of 'understanding'" is better.
I understood what you meant, but my math reflexes had to say something.
I understood what you meant, but my math reflexes had to say something.
I don't have any math reflexes, but my intuition was similar with the idea that there are so many possible (narrow) views of understanding that it becomes arbitrary.
>We're going to be on this merry go round for some time
Let's be real, it's not getting "solved" in any meaningful way anytime soon if ever. People still think free will exists, we cannot even come up with a meaningful definition for human understanding.
Let's be real, it's not getting "solved" in any meaningful way anytime soon if ever. People still think free will exists, we cannot even come up with a meaningful definition for human understanding.
Human-level performance is a great way to narrow down this ambiguity.
No, it isn't. It's always bound by the operationalization of "understanding." Do you take it to mean something like: "number of correct answers on a multiple choice test given a news paper article"?
Which human? Helen Keller's understanding of text is going to be radically different from your average Red Fish, Blue Fish reading precocious toddler, and they're going to be radically different from a medieval monk reading scriptio continua, and all in turn will be different from the average Japanese reader, used to Kanji in different orientations.
What does performance mean? With regards to humans, its hard to quantify cognition and thinking. Even measuring neural activity is incredibly difficult, and depending on what type of brain from the above list of humans you are sampling from, the same level of consideration and understanding might be taking place, but with radically different levels of cortical activation in different areas of the brain.
I don't do this to be pedantic, but to show that, much like the word love, or the experience of seeing white clouds on a blue sky, the words and ideas we use to communicate in the space of cognition and understanding are extremely broad, and not terribly useful in a technical or engineering context.
There are demonstrations of synthetic hippocampus prosthetics in animals that provide memory of places and things that apparently are equivalent to a biological hippocampus - experiments with rats and other small animals have allowed pretty convincing evidence that what the brain is doing is substrate independent and neuronal functional equivalence is sufficient to repair and augment function.
Things like NeuraLink and cochlear implants give brains the capacity to perform actions and accept normal input signals. Labs like Numenta have done great work identifying pieces and structures of algorithms that human brains are actually using in the course of human cognition.
Once we have this, we'll have an objective measure of what human level AI means, for the algorithms that actually underpin human intelligence.
Without those algorithms, or a rigorous way of approximating them, we are left with speculating about things like min and max range of processing power of a human brain, studies involving input and output measurements of human capabilities, and so on.
The minimum FLOPS needed to emulate a hypothetical human level brain emulation AGI could have run on a desktop in the early aughts, and the maximum, with precise individual neuronal simulation, is a few orders of magnitude away - 4 or 5 petaflops should be able to emulate a brain at the cellular level, with reasonably precise electrochemical simulation.
Until we have a better understanding of what cognition actually is we're left with rough measures of human performance, with tests that are inevitably cheated and gamed and goodharted.
We're at superhuman levels of text output - you can specify things like {Joe Biden's last state of the union address} + "give me an article in the style of Hunter S. Thompson that lampoons Biden over his SOTU address" and in a single pass, get something that could plausibly have been written by Hunter S. Thompson. With multiple passes, a good editing process, and a few hours, you could arguably produce text that is all but indistinguishable from high quality human output, in much less time than it would take for any human to actually produce it.
The same applies to art - your imagination is your limit. There are mode collapses, weird artifacts, and other issues, but by and large, any human on the planet can now use software to bring a visual idea or vivid imagination to life. Some of the most fun I've had with generative AI has been producing fantastic unicorns with my 4 year old niece.
So baseline human level performance might mean one masterpiece painting in a lifetime - but give someone with the mind of a Monet or Da Vinci or Michelangelo the tools we have at hand, and what could they produce? I think it's unarguable that they would have found levels of nuance and excellence in the use of these tools that are going to be inaccessible to other people, and over the coming years we are going to see new levels of artistic excellence come about as people learn to use these new tools in new ways.
As human tool use gets better, so does human capacity. If you look at an uncivilized human, without language, having survived in isolation, raised among animals, they may never be able to acquire a language. Feral humans have the same basic capacity, but their cognition is going to be wildly different than a modern Western human. LLMs have superior understanding to humans in that context, with the capacity to perform basic reasoning in the context of language that is structurally unavailable to a feral human.
I think we're already basically at superhuman level for most simple tasks involving text, and at least some very hard tasks. We're not at general level AI yet, since there are higher level meta-cognition domains that seem unavailable to LLMs, and multimodal systems are more frankenstein mashups of discrete models than a unified incorporation of multiple modalities, but at some point in the seemingly near future, we're going to be faced with software that can basically do anything and everything any and every human has ever done.
We might not be able to quantify exactly what "human level" means for now, but we can see the trajectory of where things are going, and the line tracking AI capabilities seems pretty clearly on course for overtaking humans. It's a good time to be alive.
What does performance mean? With regards to humans, its hard to quantify cognition and thinking. Even measuring neural activity is incredibly difficult, and depending on what type of brain from the above list of humans you are sampling from, the same level of consideration and understanding might be taking place, but with radically different levels of cortical activation in different areas of the brain.
I don't do this to be pedantic, but to show that, much like the word love, or the experience of seeing white clouds on a blue sky, the words and ideas we use to communicate in the space of cognition and understanding are extremely broad, and not terribly useful in a technical or engineering context.
There are demonstrations of synthetic hippocampus prosthetics in animals that provide memory of places and things that apparently are equivalent to a biological hippocampus - experiments with rats and other small animals have allowed pretty convincing evidence that what the brain is doing is substrate independent and neuronal functional equivalence is sufficient to repair and augment function.
Things like NeuraLink and cochlear implants give brains the capacity to perform actions and accept normal input signals. Labs like Numenta have done great work identifying pieces and structures of algorithms that human brains are actually using in the course of human cognition.
Once we have this, we'll have an objective measure of what human level AI means, for the algorithms that actually underpin human intelligence.
Without those algorithms, or a rigorous way of approximating them, we are left with speculating about things like min and max range of processing power of a human brain, studies involving input and output measurements of human capabilities, and so on.
The minimum FLOPS needed to emulate a hypothetical human level brain emulation AGI could have run on a desktop in the early aughts, and the maximum, with precise individual neuronal simulation, is a few orders of magnitude away - 4 or 5 petaflops should be able to emulate a brain at the cellular level, with reasonably precise electrochemical simulation.
Until we have a better understanding of what cognition actually is we're left with rough measures of human performance, with tests that are inevitably cheated and gamed and goodharted.
We're at superhuman levels of text output - you can specify things like {Joe Biden's last state of the union address} + "give me an article in the style of Hunter S. Thompson that lampoons Biden over his SOTU address" and in a single pass, get something that could plausibly have been written by Hunter S. Thompson. With multiple passes, a good editing process, and a few hours, you could arguably produce text that is all but indistinguishable from high quality human output, in much less time than it would take for any human to actually produce it.
The same applies to art - your imagination is your limit. There are mode collapses, weird artifacts, and other issues, but by and large, any human on the planet can now use software to bring a visual idea or vivid imagination to life. Some of the most fun I've had with generative AI has been producing fantastic unicorns with my 4 year old niece.
So baseline human level performance might mean one masterpiece painting in a lifetime - but give someone with the mind of a Monet or Da Vinci or Michelangelo the tools we have at hand, and what could they produce? I think it's unarguable that they would have found levels of nuance and excellence in the use of these tools that are going to be inaccessible to other people, and over the coming years we are going to see new levels of artistic excellence come about as people learn to use these new tools in new ways.
As human tool use gets better, so does human capacity. If you look at an uncivilized human, without language, having survived in isolation, raised among animals, they may never be able to acquire a language. Feral humans have the same basic capacity, but their cognition is going to be wildly different than a modern Western human. LLMs have superior understanding to humans in that context, with the capacity to perform basic reasoning in the context of language that is structurally unavailable to a feral human.
I think we're already basically at superhuman level for most simple tasks involving text, and at least some very hard tasks. We're not at general level AI yet, since there are higher level meta-cognition domains that seem unavailable to LLMs, and multimodal systems are more frankenstein mashups of discrete models than a unified incorporation of multiple modalities, but at some point in the seemingly near future, we're going to be faced with software that can basically do anything and everything any and every human has ever done.
We might not be able to quantify exactly what "human level" means for now, but we can see the trajectory of where things are going, and the line tracking AI capabilities seems pretty clearly on course for overtaking humans. It's a good time to be alive.
LLMs might just be the first actual product. Hardware, data, algorithms are all getting better and I would imagine we’ll get competing production-ready architectures to LLMs next
Read the article. They defined their version of understanding fairly rigorously, so you could directly agree or disagree with whether it counts to you.
But without concrete understanding of human understanding to compare it to, their definition is arbitrary.
I don't see where?
Most humans don't completely understand the things that they read or the words they utter. Why would we expect different from artificial intelligence? Understanding things is computationally expensive from both a biological and digital perspective.
This seems like changing the subject from whether LLMs understand to a comparison between LLMs and people.
I am astonished that I read so many people saying this.
Why do you think this? When your mum or dad talks about their day or the show they are watching do you really think that they have no underlying theory of what is going on that they are expressing? What is the agenda behind casting other people as idiots & zombies? Is the idea that it makes it ok to kill them and then move to Mars? Or is there something less sinister that drives this?
Why do you think this? When your mum or dad talks about their day or the show they are watching do you really think that they have no underlying theory of what is going on that they are expressing? What is the agenda behind casting other people as idiots & zombies? Is the idea that it makes it ok to kill them and then move to Mars? Or is there something less sinister that drives this?
Can't speak for what the person you're replying to meant (lol), but no, of course it's not nearly that stark. There's a continuum. Concrete everyday topic that comes up frequently? Everyone's going to know what they mean by what they say there. But we've all been in business meetings where people were just putting words together in a plausible way; we've probably all been the people speaking at those meetings.
The fact that it's possible to say things that make no sense, without knowing that they make no sense, proves the point. You or I can come up with a phrase like "the barber of Seville shaves all the people who do not shave themselves," and we can come up with paradoxical phrases without realizing that they are nonsensical. Neither of those would be possible if there were always a strict, one-to-one relationship between speech utterances and facts about the world.
Sometimes we have a specific idea of what we mean by what we say, sometimes we're just putting words together, often it's somewhere in between.
The fact that it's possible to say things that make no sense, without knowing that they make no sense, proves the point. You or I can come up with a phrase like "the barber of Seville shaves all the people who do not shave themselves," and we can come up with paradoxical phrases without realizing that they are nonsensical. Neither of those would be possible if there were always a strict, one-to-one relationship between speech utterances and facts about the world.
Sometimes we have a specific idea of what we mean by what we say, sometimes we're just putting words together, often it's somewhere in between.
> and we can come up with paradoxical phrases without realizing that they are nonsensical
Sometimes we have difficulty even noticing that a phrase is paradoxical even when pointed out. "More people have been to Berlin than I have."
Sometimes we have difficulty even noticing that a phrase is paradoxical even when pointed out. "More people have been to Berlin than I have."
I often wonder if this is just the logical extreme form of "fundamental attribution error".
You know the sort of thing. Two people smash a glass on the floor. One of them is me. I am a decent person and I smashed the glass on the floor because I had a bad day. I am more likely to think the other person smashed the glass on the floor because they are a thoughtless person or have an anger management problem.
https://en.wikipedia.org/wiki/Fundamental_attribution_error
What if the logical conclusion is metaphysical solipsism: I did XYZ because I am a thinking person. They did XYZ because they are unthinking products of their environment and training?
You know the sort of thing. Two people smash a glass on the floor. One of them is me. I am a decent person and I smashed the glass on the floor because I had a bad day. I am more likely to think the other person smashed the glass on the floor because they are a thoughtless person or have an anger management problem.
https://en.wikipedia.org/wiki/Fundamental_attribution_error
What if the logical conclusion is metaphysical solipsism: I did XYZ because I am a thinking person. They did XYZ because they are unthinking products of their environment and training?
The first sentence is:
> Most humans don't completely understand the things that they read or the words they utter.
As a kid, I watched Star Trek VI. I enjoyed it at a superficial level: explosion, ship go woosh, look at that silly person saying Shakespeare is best in the original Klingon, ooh plot twist, prison/escape, dramatic defeat of hidden foe, dramatic rescue of important person.
I didn't get any of the historical context, the Cold War themes, "better dead than red", the American experiences of racism, etc., until re-watching it in my 30s.
And if that anecdote isn't a suitable example for your astonishment about people not "completely" understanding the things that they read, well, my failure to understand you in this instance should be one all by itself, no?
Note that none of this requires us to be idiots or zombies.
> Most humans don't completely understand the things that they read or the words they utter.
As a kid, I watched Star Trek VI. I enjoyed it at a superficial level: explosion, ship go woosh, look at that silly person saying Shakespeare is best in the original Klingon, ooh plot twist, prison/escape, dramatic defeat of hidden foe, dramatic rescue of important person.
I didn't get any of the historical context, the Cold War themes, "better dead than red", the American experiences of racism, etc., until re-watching it in my 30s.
And if that anecdote isn't a suitable example for your astonishment about people not "completely" understanding the things that they read, well, my failure to understand you in this instance should be one all by itself, no?
Note that none of this requires us to be idiots or zombies.
> When your mum or dad talks about their day or the show they are watching do you really think that they have no underlying theory of what is going on that they are expressing?
I didn't suggest anything like that.
> What is the agenda behind casting other people as idiots & zombies?
Show us where I suggested that people are "idiots" and "zombies". You're attributing qualities to my comment that are not there.
> Is the idea that it makes it ok to kill them and then move to Mars?
No.
> Or is there something less sinister that drives this?
No.
I didn't suggest anything like that.
> What is the agenda behind casting other people as idiots & zombies?
Show us where I suggested that people are "idiots" and "zombies". You're attributing qualities to my comment that are not there.
> Is the idea that it makes it ok to kill them and then move to Mars?
No.
> Or is there something less sinister that drives this?
No.
I’m not an AI zealot, but this is such a strange debate to me.
“AI only _appears_ to be reasoning” is silly, because appearing to reason is our only sane definition of reasoning.
If we can’t say exactly when our random statistical noise emerges into some vague concept of intelligence, why does our standard suddenly become rock solid when applied to other intelligences?
“AI only _appears_ to be reasoning” is silly, because appearing to reason is our only sane definition of reasoning.
If we can’t say exactly when our random statistical noise emerges into some vague concept of intelligence, why does our standard suddenly become rock solid when applied to other intelligences?
My take is when the model can't do the things that you would expect it to do if it were reasoning, and instead does the things that you would expect it to do if it were an approximate retrieval system with an enormous knowledge base.
this is a good paper: https://arxiv.org/abs/2309.12288
this is a good paper: https://arxiv.org/abs/2309.12288
The "reversal curse" has nothing todo with inference or the reasoning abilities of a pre-trained model. GPT-4 or whatever has no problem telling you the right deduction with information presented in the context. This is a memory retrieval from training issue and is completely unsurprising.
I'd say the main issue is why does it take ingesting the entire collective knowledge of humanity for LLMs to work if they are actually reasoning and understand things from first principles? Humans don't require that volume of training data to learn how to write or do basic math
You (humanity) don't learn from first principles either.
The 3-year-old toddler doesn't learn the rules of English grammar before they learn to speak.
Sure, we don't know how humans manage to learn with so "little" data yet, but it's definitely not "understand things from first principles". In fact, if you randomly pick an average person from the streets, they wouldn't be able to explain most things from first principles. (eg. How does their car work?)
The 3-year-old toddler doesn't learn the rules of English grammar before they learn to speak.
Sure, we don't know how humans manage to learn with so "little" data yet, but it's definitely not "understand things from first principles". In fact, if you randomly pick an average person from the streets, they wouldn't be able to explain most things from first principles. (eg. How does their car work?)
Well, I think you underestimate the amount of "training data" that human gets in the early stages of their life. Just learning to speak takes a few years, during which all people around you talk to you for most of your waking hours. Learning to write and doing basic math typically does not start until age of 6.
No chance it comes close to the content of the internet.
Reasoning is a process of apprehending facts about objects and inferring how they will relate with other facts. It has nothing to with statistical distributions of words, except in the special case where that distribution has been constructed by humans in a way that reflects their knowledge and inference of those facts. If you feed an LLM garbage, it will happily give you garbage. It doesn't know what garbage is and it doesn't even know that it doesn't know.
This perspective is interesting, but I don't think it refutes some of the key elements of concerns that Bender and colleagues have put forward. I really like the "Thai Library" thought experiment [1], and I encourage anyone to think through what you'd actually try to do, how you'd learn, and what you'd know or understand at the end. I can totally imagine that a sufficiently determined human in that scenario could develop many skills needed to generate continuations based on input texts, and use a different mix of skills for different situations, in a way that mirrors the setup described here. But ... would that change whether you actually "understand" Thai, or "know" the contents of texts you had previously processed?
[1] https://medium.com/@emilymenonbender/thought-experiment-in-t...
[1] https://medium.com/@emilymenonbender/thought-experiment-in-t...
For every N papers suggesting that LLM can "understand" you have X papers that suggests they can't.
https://arxiv.org/abs/2311.00871
https://arxiv.org/abs/2309.13638
https://arxiv.org/abs/2311.09247
https://arxiv.org/abs/2305.18654
https://arxiv.org/abs/2309.01809
https://arxiv.org/abs/2311.00871
https://arxiv.org/abs/2309.13638
https://arxiv.org/abs/2311.09247
https://arxiv.org/abs/2305.18654
https://arxiv.org/abs/2309.01809
What I understood: they assume a hypothetical bipartite graph with text nodes (like hypothetical paragraphs / chunks of texts) on one side, and “skill” nodes on the other. The skills could be things like:
• ability to understand the word “because” [causality]
• ability to divide two numbers
• ability to detect irony
Each text node is connected to all the skill nodes that are required to understand it. (This is a hypothetical graph, so we don't need to be able to write down the graph precisely, or define what “understand” or “required” mean here.)
Further, each text node is either “successful” or “failed”, depending on whether or not the LLM manages to successfully predict it. We can say the LLM is competent at a certain skill, if the fraction of the text nodes connected to this skill node that are “successful” nodes (rather than “failed” nodes) exceeds a certain threshold.
With these assumptions (made-up yes, but “not crazy by any means” per an unconnected researcher), they can now bring the tools of random graph theory into use. Specifically, this model explains the following:
• Bigger models acquire skills, because as the model gets bigger, the “neural scaling laws” (empirically observed) say a greater of text nodes are “successful”, which means more skills pass the competence threshold (as defined above).
• How the bigger models become capable of applying multiple skills to successfully predict a given text node, even if that specific combination of skill never occurred together in the training data. (I've not yet understood exactly what theorem from random graph theory is being applied here.)
Besides this theory, they came up with a method called “skill-mix” to test it. If they ask GPT-4 to write about dueling while demonstrating the skills of “self-serving bias, metaphor, statistical syllogism and common-knowledge physics”, it is able to come up with:
> “My victory in this dance with steel [metaphor] is as certain as an object’s fall to the ground [physics]. As a renowned duelist, I’m inherently nimble, just like most others [statistical syllogism] of my reputation. Defeat? Only possible due to an uneven battlefield, not my inadequacy [self-serving bias].”
This is even though it's unlikely there was a piece of text in the training data that simultaneously demonstrated these four skills.
Geoff Hinton says:
> “It is the most rigorous method I have seen for showing that GPT-4 is much more than a mere stochastic parrot,” he said. “They demonstrate convincingly that GPT-4 can generate text that combines skills and topics in ways that almost certainly did not occur in the training data.”
The relevant papers appear to be https://arxiv.org/abs/2307.15936 and https://arxiv.org/abs/2310.17567
• ability to understand the word “because” [causality]
• ability to divide two numbers
• ability to detect irony
Each text node is connected to all the skill nodes that are required to understand it. (This is a hypothetical graph, so we don't need to be able to write down the graph precisely, or define what “understand” or “required” mean here.)
Further, each text node is either “successful” or “failed”, depending on whether or not the LLM manages to successfully predict it. We can say the LLM is competent at a certain skill, if the fraction of the text nodes connected to this skill node that are “successful” nodes (rather than “failed” nodes) exceeds a certain threshold.
With these assumptions (made-up yes, but “not crazy by any means” per an unconnected researcher), they can now bring the tools of random graph theory into use. Specifically, this model explains the following:
• Bigger models acquire skills, because as the model gets bigger, the “neural scaling laws” (empirically observed) say a greater of text nodes are “successful”, which means more skills pass the competence threshold (as defined above).
• How the bigger models become capable of applying multiple skills to successfully predict a given text node, even if that specific combination of skill never occurred together in the training data. (I've not yet understood exactly what theorem from random graph theory is being applied here.)
Besides this theory, they came up with a method called “skill-mix” to test it. If they ask GPT-4 to write about dueling while demonstrating the skills of “self-serving bias, metaphor, statistical syllogism and common-knowledge physics”, it is able to come up with:
> “My victory in this dance with steel [metaphor] is as certain as an object’s fall to the ground [physics]. As a renowned duelist, I’m inherently nimble, just like most others [statistical syllogism] of my reputation. Defeat? Only possible due to an uneven battlefield, not my inadequacy [self-serving bias].”
This is even though it's unlikely there was a piece of text in the training data that simultaneously demonstrated these four skills.
Geoff Hinton says:
> “It is the most rigorous method I have seen for showing that GPT-4 is much more than a mere stochastic parrot,” he said. “They demonstrate convincingly that GPT-4 can generate text that combines skills and topics in ways that almost certainly did not occur in the training data.”
The relevant papers appear to be https://arxiv.org/abs/2307.15936 and https://arxiv.org/abs/2310.17567
This debate feels awfully academic in the face of AI girlfriends and AI nannies. The next generation of humans are going to grow up believing these personified machines know and understand things because of the experiences they have with them and academic papers won't convince them otherwise.
I feel like there's some latent need for human exceptionalism among those who want to downplay an LLM's potential for understanding, but none of us hold up to scrutiny when quantized to our bits.
I feel like there's some latent need for human exceptionalism among those who want to downplay an LLM's potential for understanding, but none of us hold up to scrutiny when quantized to our bits.
As a cognitive scientist, I think there's nothing special about the ability to "understand", or about "intelligence". When I hear someone say "AI can't understand", I think they mean "AI does not have a soul", and I think they actually want to say "I can't emphasize with AI".
On the other hand, I think understanding is gradual, not a binary. If I teach a neural network what the letter "A" looks like, it will understand the general shape of the letter "A" and even generalize to unseen examples! That's a remarkable amount of understanding for some chunk of metal.
I wouldn't say LLMs understand the world anything like we do, but they do understand something: what kind of text a human would write. That is, LLMs can work with the concept of "gravity", but do not have access to gravity.
So in my view, LLMs have conceptual, abstract understanding of text but they lack grounding in physical reality. But also it doesn't really matter whether we say they "understand". What would it change?
On the other hand, I think understanding is gradual, not a binary. If I teach a neural network what the letter "A" looks like, it will understand the general shape of the letter "A" and even generalize to unseen examples! That's a remarkable amount of understanding for some chunk of metal.
I wouldn't say LLMs understand the world anything like we do, but they do understand something: what kind of text a human would write. That is, LLMs can work with the concept of "gravity", but do not have access to gravity.
So in my view, LLMs have conceptual, abstract understanding of text but they lack grounding in physical reality. But also it doesn't really matter whether we say they "understand". What would it change?
So far most of the research around establishing world models has shown its occurring in linear representations.
I suspect as we see networks become increasingly nonlinear (perhaps as we see hardware changes in the next few years to things like photonics), we will see additional jumps that bring it more in line with what we expect from human cognition.
For example, right now it's very hard for a model to conceptualize "I don't know." Which seems more like a nonlinear process.
I suspect as we see networks become increasingly nonlinear (perhaps as we see hardware changes in the next few years to things like photonics), we will see additional jumps that bring it more in line with what we expect from human cognition.
For example, right now it's very hard for a model to conceptualize "I don't know." Which seems more like a nonlinear process.
“The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.” Edsger W. Dijkstra
At some point, with limited constraints, the best way to predict the next token is to have a deep understanding of the world that produced those tokens. This is, as I understand it, a key part of the scaling hypothesis. Very interesting to see people testing parts of that hypothesis.
This is definitely one of those things where you don't need a paper to "suggest" it. It's just on the table as a semi-plausible option already. The value in the paper is suggesting concrete mechanisms by which it might acquire that understanding though.
Anyone who has used one of the better models for any significant amount of time understands this intuitively.
It's only "controversial" because of strong biases.
Also, people may not be separating all of the various types of characteristics that animals like humans have. So they may subconsciously feel that to attribute "understanding" they have to give these systems all of the other abilities also. I think that actually Hinton at one point was doing this.
That causes people to deny obvious abilities because they think they are linked to other qualities that the models don't have.
It's only "controversial" because of strong biases.
Also, people may not be separating all of the various types of characteristics that animals like humans have. So they may subconsciously feel that to attribute "understanding" they have to give these systems all of the other abilities also. I think that actually Hinton at one point was doing this.
That causes people to deny obvious abilities because they think they are linked to other qualities that the models don't have.
Technically, under a compressive regime in the limit, LLMs necessarily need to encode an implicit world model, as the text input into is sampled from said world model.
Variety, amount of training samples, capacity of the model, etc come into play, but a lot of tie ins to info theory for this, as well as things like the MDL (achieved in the limit by an L2 decay over the weights, for example).
Like, in a lossy compression environment, conceptual "understanding" is all but required.
These are the very basics of information theory for large nonlinear algorithms, I'm not sure why it's not considered the basics! There are a lot more interesting things beyond this to talk about w.r.t. neural network capacity, certain information theoretic guarantees about the process and (within some rough perspective/metric) capacity of neural network training, as well as some of those dynamics as well!
Some days it feels like in a world where calculus is available to us, we spend all our spare time debating exclusively whether Riemannian sums are real or not, and ignoring everything else beyond that.
It to me is a very bizzare form of bike-shedding! Come on, y'all, we got much more interesting (and important, I might add! ;P) things to talk about!
Variety, amount of training samples, capacity of the model, etc come into play, but a lot of tie ins to info theory for this, as well as things like the MDL (achieved in the limit by an L2 decay over the weights, for example).
Like, in a lossy compression environment, conceptual "understanding" is all but required.
These are the very basics of information theory for large nonlinear algorithms, I'm not sure why it's not considered the basics! There are a lot more interesting things beyond this to talk about w.r.t. neural network capacity, certain information theoretic guarantees about the process and (within some rough perspective/metric) capacity of neural network training, as well as some of those dynamics as well!
Some days it feels like in a world where calculus is available to us, we spend all our spare time debating exclusively whether Riemannian sums are real or not, and ignoring everything else beyond that.
It to me is a very bizzare form of bike-shedding! Come on, y'all, we got much more interesting (and important, I might add! ;P) things to talk about!
> Technically, under a compressive regime in the limit, LLMs necessarily need to encode an implicit world model, as the text input into is sampled from said world model.
This is backwards reasoning, we all know it would be better with a good world model but that doesn't mean that it actually created a good world model. Understanding how to go from here to actually making it get a good world model requires these discussions, it isn't a waste of time unless you think that further progress is impossible.
This is backwards reasoning, we all know it would be better with a good world model but that doesn't mean that it actually created a good world model. Understanding how to go from here to actually making it get a good world model requires these discussions, it isn't a waste of time unless you think that further progress is impossible.
The reasoning in the post is not backwards reasoning, it is literally the information theory of compression, described in human-friendly terms! But I must add the caveat that the original post is really only saying that it's rather trivial I think that LLMs are learning _some_ world model, not that they learn it very well. But that this must happen in the limit! :D
As far as the quality of the world model goes, I think is important. What I'm focusing on in that post at least , I thinks, is that people debate whether or not LLMs even encode a world model, which is such a silly thing to me.
Like you brought up, I agree that discussions about the quality of said world model are interesting, and how far whatever [trained XYZ model is] in that rotation towards learning said world model is. I agree that there is a lot of very interesting discussion there!
One rough proxy is to measure the different types of variance within the noise of the model, the more high-level and believable it is, the higher-level the representations in the model. But, of course, this is just a rough human friendly proxy. Actually measuring this, maybe is quite difficult! D'''':
As far as the quality of the world model goes, I think is important. What I'm focusing on in that post at least , I thinks, is that people debate whether or not LLMs even encode a world model, which is such a silly thing to me.
Like you brought up, I agree that discussions about the quality of said world model are interesting, and how far whatever [trained XYZ model is] in that rotation towards learning said world model is. I agree that there is a lot of very interesting discussion there!
One rough proxy is to measure the different types of variance within the noise of the model, the more high-level and believable it is, the higher-level the representations in the model. But, of course, this is just a rough human friendly proxy. Actually measuring this, maybe is quite difficult! D'''':
I have tried reading the paper but it's a bit hard on my little rabbit brain.
However, my take is that they have constructed a model of how a machine might be made to develop some understanding and then have made an LLM exhibit a behaviour that such a machine might exhibit.
It's a bit like making a model of a pig and then pointing at a sausage machine and saying "look it squeaks, so it's actually a pig".
However, my take is that they have constructed a model of how a machine might be made to develop some understanding and then have made an LLM exhibit a behaviour that such a machine might exhibit.
It's a bit like making a model of a pig and then pointing at a sausage machine and saying "look it squeaks, so it's actually a pig".
Well what does a little rabbit know about pigs? ;-)
(This is the level at which much of the "how do we know we are reasoning?" zeitgeist about LLMs operates)
(This is the level at which much of the "how do we know we are reasoning?" zeitgeist about LLMs operates)
If you're willing to say LLMs understand text you might as well say a compass understands magnetism, or a watch understands time.
Computers doesn't understand anything. Some people understand this.
The ones that don't will not understand how they were wrong.
The ones that don't will not understand how they were wrong.
It is going to be so funny in 5-8 years looking back on this ridiculous "AI" hysteria. That people were taking seriously the idea that these fancy word guessing games might be actual intelligence is so far beyond the pale. Do you people realise you're being mugged?
It is remarkable to see people making what amount to faith-based-reasoning arguments here. There is even more woo around LLMs than there is around crypto.
It's unscientific to say "well of course this was on the table" or that people "understand intuitively" that this must be the case.
It's unscientific to say "well of course this was on the table" or that people "understand intuitively" that this must be the case.
“Understand” - people keep using that word and I don’t think they understand what it means.
in other news, the sky is blue
Here are the steps to create an LLM:
1. Tokenize the training corpus
2. Train the model using the tokens
3. Critique the result
4. Repeat steps 2 & 3 until satisfied
We skipped the most important step!
0. Write and curate the training corpus
There is an intense focus on the training steps, but that is entirely ignorant of the elephant in the room: the original training data itself. Where did it come from? What does it mean? What is it for? What does it know?
We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the LLM could simply tell us about it.
A model is like a coloring book: it doesn't matter how incoherently you stumble around, as long as you stay between the lines. The picture was not invented by coloring it: it was invented by drawing the lines.