Chomsky on what ChatGPT is good for (2023)(chomsky.info)
chomsky.info
Chomsky on what ChatGPT is good for (2023)
https://chomsky.info/20230503-2/
388 comments
> AI this decade have not, at least so far, substantially deepened our understanding of our own intelligence and its constitution
I would push back on this a little bit. While it has not helped us to understand our own intelligence, it has made me question whether such a thing even exists. Perhaps there are no simple and beautiful natural laws, like those that exists in Physics, that can explain how humans think and make decisions. When CNNs learned to recognize faces through a series of hierarchical abstractions that make intuitive sense it's hard to deny the similarities to what we're doing as humans. Perhaps it's all just emergent properties of some messy evolved substrate.
The big lesson from the AI development in the last 10 years from me has been "I guess humans really aren't so special after all" which is similar to what we've been through with Physics. Theories often made the mistake of giving human observers some kind of special importance, which was later discovered to be the cause of theories not generalizing.
I would push back on this a little bit. While it has not helped us to understand our own intelligence, it has made me question whether such a thing even exists. Perhaps there are no simple and beautiful natural laws, like those that exists in Physics, that can explain how humans think and make decisions. When CNNs learned to recognize faces through a series of hierarchical abstractions that make intuitive sense it's hard to deny the similarities to what we're doing as humans. Perhaps it's all just emergent properties of some messy evolved substrate.
The big lesson from the AI development in the last 10 years from me has been "I guess humans really aren't so special after all" which is similar to what we've been through with Physics. Theories often made the mistake of giving human observers some kind of special importance, which was later discovered to be the cause of theories not generalizing.
> The big lesson from the AI development in the last 10 years from me has been "I guess humans really aren't so special after all"
Instead I would take the opposite take.
How wonderful is it, that with naturally evolved processes and neural structures, have we been able to create what we have. Van Gogh’s paintings came out of the human brain. The Queens of the Skies - hundreds of tons of metal and composites - flying across continents in the form of a Boeing 747 or an A380 - was designed by the human brain. We went to space, have studied nature (and have conservation programs for organisms we have found to need help), took pictures the pillars of creation that are so incredibly far… all with such a “puny” structure a few cm in diameter? I think that’s freaking amazing.
Instead I would take the opposite take.
How wonderful is it, that with naturally evolved processes and neural structures, have we been able to create what we have. Van Gogh’s paintings came out of the human brain. The Queens of the Skies - hundreds of tons of metal and composites - flying across continents in the form of a Boeing 747 or an A380 - was designed by the human brain. We went to space, have studied nature (and have conservation programs for organisms we have found to need help), took pictures the pillars of creation that are so incredibly far… all with such a “puny” structure a few cm in diameter? I think that’s freaking amazing.
"I guess humans really aren't so special after all"
This is a crazy take to me. As compared to what? The machines that we built?
Until we discover comparably intelligent life in the universe I think it's fair to say that we are indeed very special.
This is a crazy take to me. As compared to what? The machines that we built?
Until we discover comparably intelligent life in the universe I think it's fair to say that we are indeed very special.
I am pleasantly surprised that David Hume's writings have been mentioned. I love his works.
I have to confess this is the only essay of his I know, though it's an all-time favorite. What other Hume pieces would you recommend?
What initially drew me to David Hume was a quote from his discussions of miracles in "An Enquiry Concerning Human Understanding" (name of chapter is "Of Miracles").
That said, I began with "A Treatise of Human Nature" around the age of 17, translated to my native language (his works are not an easy read in English, IMO), due to my interest in both philosophy and psychology.
If you haven't read them yet, I would certainly recommend them. I would recommend the latter I mentioned even if you are not interested in psychology (but may be interested in epistemology, philosophy of mind, and/or ethics), as he gets into detail about his "impressions" vs "ideas".
Additionally, he is famously known for his "problem of induction" which you may already know.
That said, I began with "A Treatise of Human Nature" around the age of 17, translated to my native language (his works are not an easy read in English, IMO), due to my interest in both philosophy and psychology.
If you haven't read them yet, I would certainly recommend them. I would recommend the latter I mentioned even if you are not interested in psychology (but may be interested in epistemology, philosophy of mind, and/or ethics), as he gets into detail about his "impressions" vs "ideas".
Additionally, he is famously known for his "problem of induction" which you may already know.
Its like saying:
Ah, but these wizards created a magical entity that can also do magic! Wizards must not be so special after all...
Ah, but these wizards created a magical entity that can also do magic! Wizards must not be so special after all...
You know how many old sci-fi settings pictured aliens as bipedal furry animals or lizards? Even to go from that to realistically-intelligent swarms of insects is already difficult.
(Of course, there’s plenty of sci-fi where conscious entities manifest themselves as abstract balls of pure energy or the like; except for some reason those balls still think in the same way we do, get assigned the same motivations, sometimes even speak our language, etc., which makes it, in a way, even less realistic than the walking and talking human-cat hybrid you’d see in Elder Scrolls.)
Whenever we ponder questions of intelligence and consciousness, the same pitfall awaits.
Since we don’t have an objective definition of consciousness or intelligence (and in all likelihood we can’t have one, because any formal attempt at such wouldn’t get very far due to being attempted by the same thing that’s being defined), the only one that makes sense is, in crude language, “something like what we are”. There’s a vague feeling that it has to do with free will, self-awareness, etc.; however, all of it is also influenced by the nature of us being all parts of some big figurative anthill—assuming your sense of self only arises as you model yourself against the other (starting with your parents/caretakers and on), a standalone human cannot be self-aware in the way we are if it evolved in an emptiness without others—i.e., it would not possess human intelligence; supported by our natural-scientific observations rejecting the possibility of a being of this shape and form ever evolving in the first place.
In other words, the more different some kind of intelligence is from ours, the less it would look like intelligence to us—which makes the search for alien intelligence in space somewhat tragically futile (if it exists, we wouldn’t recognize it unless it just happens to be like us), but opens up exciting opportunities for finding alien but not-too-alien intelligence right on this planet (almost Douglas Adams style, minus dolphins speaking English).
There’s an extra trick when it comes to LLMs. In case of alien life, the possibility of a radically different kind of consciousness producing output that closely mimics our own is almost impossible (if our prior assumption is correct, then for all intents and purposes truly alien, non-meatbag-scale kind of intelligence might not be able to recognize ours in the first place, just like we wouldn’t recognize alien intelligence). However, the LLMs are designed to mimic the most social aspect of our behavior, our communication aimed at fellow humans; so when an LLM produces sufficiently human-like output—even if it has a very different kind of consciousness[0] or no consciousness at all (more likely, though as we concluded above we can’t distinguish between the two cases anyway)—our minds are primed to see it as a manifestation of [which would be human-like] intelligence, even if there’s nothing that would suggest such judging by the way it’s created (which is radically different from the way we’ve been creating intelligent life so far, wink-wink), by the substrate it runs on, if not by the way it actually works (which per our conclusion above we might never be able to conclusively determine about our own minds, without resorting to unfalsifiable philosophical assumptions for at least some aspects of it).
So yes, I’d say humans are special, if nothing else then because by the only usable (if somewhat circular) definition of what we are there’s absolutely nothing like us around, and in all likelihood can never be. (That’s not to say that something not like us isn’t special in its own way—I mean, think of the dolphins!—but given we, due to not being it, would not be able to properly understand it, it just never hits the same.)
[0] Which if true would be completely asocial (given it neither exists in groups nor depends on others for survival) and therefore drastically different from ours.
(Of course, there’s plenty of sci-fi where conscious entities manifest themselves as abstract balls of pure energy or the like; except for some reason those balls still think in the same way we do, get assigned the same motivations, sometimes even speak our language, etc., which makes it, in a way, even less realistic than the walking and talking human-cat hybrid you’d see in Elder Scrolls.)
Whenever we ponder questions of intelligence and consciousness, the same pitfall awaits.
Since we don’t have an objective definition of consciousness or intelligence (and in all likelihood we can’t have one, because any formal attempt at such wouldn’t get very far due to being attempted by the same thing that’s being defined), the only one that makes sense is, in crude language, “something like what we are”. There’s a vague feeling that it has to do with free will, self-awareness, etc.; however, all of it is also influenced by the nature of us being all parts of some big figurative anthill—assuming your sense of self only arises as you model yourself against the other (starting with your parents/caretakers and on), a standalone human cannot be self-aware in the way we are if it evolved in an emptiness without others—i.e., it would not possess human intelligence; supported by our natural-scientific observations rejecting the possibility of a being of this shape and form ever evolving in the first place.
In other words, the more different some kind of intelligence is from ours, the less it would look like intelligence to us—which makes the search for alien intelligence in space somewhat tragically futile (if it exists, we wouldn’t recognize it unless it just happens to be like us), but opens up exciting opportunities for finding alien but not-too-alien intelligence right on this planet (almost Douglas Adams style, minus dolphins speaking English).
There’s an extra trick when it comes to LLMs. In case of alien life, the possibility of a radically different kind of consciousness producing output that closely mimics our own is almost impossible (if our prior assumption is correct, then for all intents and purposes truly alien, non-meatbag-scale kind of intelligence might not be able to recognize ours in the first place, just like we wouldn’t recognize alien intelligence). However, the LLMs are designed to mimic the most social aspect of our behavior, our communication aimed at fellow humans; so when an LLM produces sufficiently human-like output—even if it has a very different kind of consciousness[0] or no consciousness at all (more likely, though as we concluded above we can’t distinguish between the two cases anyway)—our minds are primed to see it as a manifestation of [which would be human-like] intelligence, even if there’s nothing that would suggest such judging by the way it’s created (which is radically different from the way we’ve been creating intelligent life so far, wink-wink), by the substrate it runs on, if not by the way it actually works (which per our conclusion above we might never be able to conclusively determine about our own minds, without resorting to unfalsifiable philosophical assumptions for at least some aspects of it).
So yes, I’d say humans are special, if nothing else then because by the only usable (if somewhat circular) definition of what we are there’s absolutely nothing like us around, and in all likelihood can never be. (That’s not to say that something not like us isn’t special in its own way—I mean, think of the dolphins!—but given we, due to not being it, would not be able to properly understand it, it just never hits the same.)
[0] Which if true would be completely asocial (given it neither exists in groups nor depends on others for survival) and therefore drastically different from ours.
In Star Trek the whole humanoids everywhere thing is an obvious practicality in producing episodes, though.
They spent the whole budget on the salt vampire and never recovered.
They spent the whole budget on the salt vampire and never recovered.
Well, most sci-fi still fits the bill. Vinge is a bit interesting in that he plays around with the idea with Tines where an “individual” (in human sense) is a pack of 5 of them[0] or with civilizations that “transcend” and then no one has any idea of what are about anymore, and how a bunch of civilizations evolved from humans which explains how they all just happen to operate on equivalent human meatbag scale.
[0] Genuinely not unlike how a congregation of gelled-together humans is an entity that can achieve much more than an individual human.
[0] Genuinely not unlike how a congregation of gelled-together humans is an entity that can achieve much more than an individual human.
"Brain_s_". I find we (me included) generally overlook/underestimate the distributed nature of human intelligence, included in the AI field. That's why when I first heard of mixture of experts I was thrilled about the idea and the potential. (One could also see similarities in random forest).
I believe a path to AGI(tm) would be to reproduce the evolution of human intelligence artificially. Start with small models training bigger and bigger models and let the bigger successfull models (insert RL, genetic algos, etc.) "reproduce" and teach newer models from scratch. Having different model architecture cohabit could maybe even lead to the kind of specializations we see in parts of the brain
I think it is important to realize, that we need to understand language on our own terms. The logic of LLMs is not unlike alien technology to us. That being said, the minimalist program of Chomsky lead to nowhere, because just like programming, it found edge case after edge case, reducing it further and further, until there was no program anymore that resembled a real theory. But it is wrong to assume that the big progress in linguistics is in vain, the same reason Prolog, Theorem provers, type theory, category theory is in vain, when we have LLMs that can produce everything in C++. We can use the technology of linguistics to ground our knowledge, and in some dark corner of the LLM it might already have integrated this. I think the original divide between the sciences and the humanities might be deeper and more fundamental than we think. We need linguistic as a discipline of the humanities, and maybe huge swaths of Computer Science is just that.
I agree with you. I think the fundamental problem is we don't have a good unified theory of fuzzy reasoning. We have a lot of different formal approaches but they all have flaws.
Now LLMs made a big breakthrough that they showed we can do decent fuzzy reasoning in practice. But at the cost of nobody understanding the underlying process formally.
If we had a good unified (formal) theory of fuzzy reasoning, we could build models that reason better (or at least more predictably). But we won't get a better theory by scaling the existing models, I think Chomsky is right about that.
We lack the goal, not the means. If I am asking LLM a question, what answer do I want? A playfully creative one? A strictly logical one? A pleasingly sycophantic one? A harshly critical one? An out of the box devil's advocate one? A beautiful one? A practical one? We have no clue how to express these modes in logical reasoning.
Now LLMs made a big breakthrough that they showed we can do decent fuzzy reasoning in practice. But at the cost of nobody understanding the underlying process formally.
If we had a good unified (formal) theory of fuzzy reasoning, we could build models that reason better (or at least more predictably). But we won't get a better theory by scaling the existing models, I think Chomsky is right about that.
We lack the goal, not the means. If I am asking LLM a question, what answer do I want? A playfully creative one? A strictly logical one? A pleasingly sycophantic one? A harshly critical one? An out of the box devil's advocate one? A beautiful one? A practical one? We have no clue how to express these modes in logical reasoning.
By way of analogy, the result of the theorem prover is usually actionable (i.e. we can replace one kind of expression with its proven equivalent for some end like optimizing code-size or code-run-time), but mathematicians _still_ endeavor to translate the unwieldy and verbose machine-generated proofs into concise human-readable proofs, because those readable proofs are useful to our understanding of mathematics even long after the "productive action" has been taken.
In a way, this collaboration between the machine and the human is better than what came before, because now productive actions can be taken sooner, and mathematicians do not have to doubt whether they are searching for a proof that exists.
In a way, this collaboration between the machine and the human is better than what came before, because now productive actions can be taken sooner, and mathematicians do not have to doubt whether they are searching for a proof that exists.
>That being said, the minimalist program of Chomsky lead to nowhere, because just like programming, it found edge case after edge case, reducing it further and further, until there was no program anymore that resembled a real theory
As someone who has worked in linguistics, I don't really see what you're talking about. Minimalism is not full of exceptions (please elaborate on a specific example if you have one). Minimalism was created to make the old theory, Government and Binding, simpler.
As someone who has worked in linguistics, I don't really see what you're talking about. Minimalism is not full of exceptions (please elaborate on a specific example if you have one). Minimalism was created to make the old theory, Government and Binding, simpler.
Yes, and the project can be criticised by reducing until there's no value anymore. Well known instances of this process:
- Predicate Fronting in Free Relatives: In sentences like “What John saw was a surprise,” labeling the fronted predicate is not without problems, Merge doesn’t yield a clear head.
- Optional Verb Movement in Persian: Yes-no questions where verbs can optionally move (e.g., “Did you go?” vs. “You went?”) messes up feature-checking’s binary mode.
- Non-Matching Free Relatives with Pied-Piping: Structures like “In whichever city you live, you’ll find culture” mess up standard labeling, needs extra stipulations.
- Some Subjects in Finnish: Nominative vs. non-nominative subjects (e.g., “Minua kylmä” [me-ACC cold]) complicate that Minimalist case assignment.
- Predicate Fronting in Free Relatives: In sentences like “What John saw was a surprise,” labeling the fronted predicate is not without problems, Merge doesn’t yield a clear head.
- Optional Verb Movement in Persian: Yes-no questions where verbs can optionally move (e.g., “Did you go?” vs. “You went?”) messes up feature-checking’s binary mode.
- Non-Matching Free Relatives with Pied-Piping: Structures like “In whichever city you live, you’ll find culture” mess up standard labeling, needs extra stipulations.
- Some Subjects in Finnish: Nominative vs. non-nominative subjects (e.g., “Minua kylmä” [me-ACC cold]) complicate that Minimalist case assignment.
but we don't have llms that can "produce everything in c++".
We have LLMs that can get some boilerplate right if you use it in a greenfield project, and will repeatedly mess up your code once it grows enough for you to actually need assistance grokking it.
We have LLMs that can get some boilerplate right if you use it in a greenfield project, and will repeatedly mess up your code once it grows enough for you to actually need assistance grokking it.
> Perhaps there are no simple and beautiful natural laws, like those that exists in Physics, that can explain how humans think and make decisions.
Isn't Physics trying to describe the natural world? I'm guessing you are taking two positions here that are causing me confusion with your statement: 1) that our minds can be explained strictly through physical processes, and 2) our minds, including our intelligence, are outside of the domain of Physics.
If you take 1) to be true, then it follows that Physics, at least theoretically, should be able to explain intelligence. It may be intractably hard, like it might be intractably hard to have physics decribe and predict the motions of more than two planetary bodies.
I guess I'm saying that Physical laws ARE natural laws. I think you might be thinking that natural laws refer solely to all that messy, living stuff.
Isn't Physics trying to describe the natural world? I'm guessing you are taking two positions here that are causing me confusion with your statement: 1) that our minds can be explained strictly through physical processes, and 2) our minds, including our intelligence, are outside of the domain of Physics.
If you take 1) to be true, then it follows that Physics, at least theoretically, should be able to explain intelligence. It may be intractably hard, like it might be intractably hard to have physics decribe and predict the motions of more than two planetary bodies.
I guess I'm saying that Physical laws ARE natural laws. I think you might be thinking that natural laws refer solely to all that messy, living stuff.
I think their emphasis is on simple and beautiful; not that human intelligence is outside the laws of physics, but that there will never be a “Maxwell’s equations” modelling the workings of human intelligence, it will just be a big pile of hacks and complex interactions of many distinct parts; nothing like the couple of recursive LISP macros people of the 1960s might have hoped to find.
Neuroscientist here:
> Perhaps there are no simple and beautiful natural laws, like those that exists in Physics, that can explain how humans think and make decisions...Perhaps it's all just emergent properties of some messy evolved substrate.
Yeah, it is very likely that there are not laws that will do this, it's the substrate. The fruit fly brain (let alone human) has been mapped, and we've figured out that it's not just the synapse count, but the 'weights' that matter too [0]. Mind you, those weights adjust in real time when a living animal is out there.
You'll see in literature that there are people with some 'lucky' form of hydranencephaly where their brain is as thin as paper. But they vote, get married, have kids, and for some strange reason seem to work in mailrooms (not a joke). So we know it's something about the connectome that's the 'magic' of a human.
My pet theory: We need memristors [2] to better represent things. But that takes redesigning the computer from the metal on up, so is unlikely to occur any time soon with this current AI craze.
> The big lesson from the AI development in the last 10 years from me has been "I guess humans really aren't so special after all" which is similar to what we've been through with Physics.
Yeah, biologists get there too, just the other way abouts, with animals and humans. Like, dogs make vitamin C internally, and humans have that gene too, it's just dormant, ready for evolution (or genetic engineering) to reactivate. That said, these neuroscience issues with us and the other great apes are somewhat large and strange. I'm not big into that literature, but from what little I know, the exact mechanisms and processes that get you from tool using ourangs to tool using humans, well, those seem to be a bit strange and harder to grasp for us. Again, not in that field though.
In the end though, humans are special. We're the only ones on the planet that ever really asked a question. There's a lot to us and we're actually pretty strange in the end. There's many centuries of work to do with biology, we're just at the wading stage of that ocean.
[0] https://en.wikipedia.org/wiki/Drosophila_connectome
[1] https://en.wikipedia.org/wiki/Hydranencephaly
[2] https://en.wikipedia.org/wiki/Memristor
> Perhaps there are no simple and beautiful natural laws, like those that exists in Physics, that can explain how humans think and make decisions...Perhaps it's all just emergent properties of some messy evolved substrate.
Yeah, it is very likely that there are not laws that will do this, it's the substrate. The fruit fly brain (let alone human) has been mapped, and we've figured out that it's not just the synapse count, but the 'weights' that matter too [0]. Mind you, those weights adjust in real time when a living animal is out there.
You'll see in literature that there are people with some 'lucky' form of hydranencephaly where their brain is as thin as paper. But they vote, get married, have kids, and for some strange reason seem to work in mailrooms (not a joke). So we know it's something about the connectome that's the 'magic' of a human.
My pet theory: We need memristors [2] to better represent things. But that takes redesigning the computer from the metal on up, so is unlikely to occur any time soon with this current AI craze.
> The big lesson from the AI development in the last 10 years from me has been "I guess humans really aren't so special after all" which is similar to what we've been through with Physics.
Yeah, biologists get there too, just the other way abouts, with animals and humans. Like, dogs make vitamin C internally, and humans have that gene too, it's just dormant, ready for evolution (or genetic engineering) to reactivate. That said, these neuroscience issues with us and the other great apes are somewhat large and strange. I'm not big into that literature, but from what little I know, the exact mechanisms and processes that get you from tool using ourangs to tool using humans, well, those seem to be a bit strange and harder to grasp for us. Again, not in that field though.
In the end though, humans are special. We're the only ones on the planet that ever really asked a question. There's a lot to us and we're actually pretty strange in the end. There's many centuries of work to do with biology, we're just at the wading stage of that ocean.
[0] https://en.wikipedia.org/wiki/Drosophila_connectome
[1] https://en.wikipedia.org/wiki/Hydranencephaly
[2] https://en.wikipedia.org/wiki/Memristor
>You'll see in literature that there are people with some 'lucky' form of hydranencephaly where their brain is as thin as paper. But they vote, get married, have kids, and for some strange reason seem to work in mailrooms (not a joke). So we know it's something about the connectome that's the 'magic' of a human.
These cases seem totally fascinating. Have you any links to examples or more information (i'm also curious about the curious detail of them tending to work in mail rooms)?
These cases seem totally fascinating. Have you any links to examples or more information (i'm also curious about the curious detail of them tending to work in mail rooms)?
It is possible that we simply haven't yet discovered those natural laws for "emergent behavior" from the "messy substrate".
> it has made me question whether such a thing even exists
I was reading a reddit post the other day where the guy lost his crypto holdings because he input his recovery phrase somewhere. We question the intelligence of LLMs because they might open a website, read something nefarious, and then do it. But here we have real humans doing the exact same thing...
> I guess humans really aren't so special after all
No they are not. But we are still far from getting there with the current LLMs and I suspect mimicking the human brain won't be the best path forward.
I was reading a reddit post the other day where the guy lost his crypto holdings because he input his recovery phrase somewhere. We question the intelligence of LLMs because they might open a website, read something nefarious, and then do it. But here we have real humans doing the exact same thing...
> I guess humans really aren't so special after all
No they are not. But we are still far from getting there with the current LLMs and I suspect mimicking the human brain won't be the best path forward.
> But here we have real humans doing the exact same thing...
I'd wager that a motivation in designing these systems it so they do not make these mistakes. Otherwise what's the point, really.
I'd wager that a motivation in designing these systems it so they do not make these mistakes. Otherwise what's the point, really.
I think a system too perfect will not show any creativity. Maybe wild new ideas require taking risks which means a system that can invent new things will end up making bad choices.
Our own, and other people's mistakes shape us, and our understanding. What even is perfect, anyway?
I'm not interested in navel gazing. I'm interested in getting my taxes done properly.
> one, that the facility of LLMs is fantastic and useful
I'm assuming this was the part you were saying he doesn't hold, because it is pretty clear he holds the second thought.
| is it likely that programs will be devised that surpass human capabilities? We have to be careful about the word “capabilities,” for reasons to which I’ll return. But if we take the term to refer to human performance, then the answer is: definitely yes.
| We can make a rough distinction between pure engineering and science. There is no sharp boundary, but it’s a useful first approximation. Pure engineering seeks to produce a product that may be of some use. Science seeks understanding.
It indeed baffles me how academics overall seem so dismissive of recent breakthroughs in sub-symbolic approaches as models from which we can learn about 'intelligence'?
It is as if a biochemist looks at a human brain, and concludes there is no 'intelligence' there at all, just a whole lot of electro-chemical reactions. It fully ignores the potential for emergence.
Don't misunderstand me, I'm not saying 'AGI has arrived', but I'd say even current LLM's do most certainly have interesting lessons for Human Language development and evolution in science. What can the success in transfer learning in these models contribute to the debates on universal language faculties? How do invariants correlated across LLM systems and humans?
It is as if a biochemist looks at a human brain, and concludes there is no 'intelligence' there at all, just a whole lot of electro-chemical reactions. It fully ignores the potential for emergence.
Don't misunderstand me, I'm not saying 'AGI has arrived', but I'd say even current LLM's do most certainly have interesting lessons for Human Language development and evolution in science. What can the success in transfer learning in these models contribute to the debates on universal language faculties? How do invariants correlated across LLM systems and humans?
>It fully ignores the potential for emergence.
There's two kinds of emergence, one scientific, the other a strange, vacuous notion in the absence of any theory and explanation.
The first case is emergence when we for example talk about how gas or liquid states, or combustibility emerge from certain chemical or physical properties of particles. It's not just that they're emergent, we can explain how they're emergent and how their properties are already present in the lower level of abstraction. Emergence properly understood is always reducible to lower states, not some magical word if you don't know how something works.
In these AI debates that's however exactly how "emergence" is used, people just assert it, following necessarily from their assumptions. They don't offer a scientific explanation. (the same is true with various other topics, like consciousness, or what have you). This is pointless, it's a sort of god of the gaps disguised as an argument. When Chomsky talks about science proper, he correctly points out that these kinds of arguments have no place in it, because the point of science is to build coherent theories.
There's two kinds of emergence, one scientific, the other a strange, vacuous notion in the absence of any theory and explanation.
The first case is emergence when we for example talk about how gas or liquid states, or combustibility emerge from certain chemical or physical properties of particles. It's not just that they're emergent, we can explain how they're emergent and how their properties are already present in the lower level of abstraction. Emergence properly understood is always reducible to lower states, not some magical word if you don't know how something works.
In these AI debates that's however exactly how "emergence" is used, people just assert it, following necessarily from their assumptions. They don't offer a scientific explanation. (the same is true with various other topics, like consciousness, or what have you). This is pointless, it's a sort of god of the gaps disguised as an argument. When Chomsky talks about science proper, he correctly points out that these kinds of arguments have no place in it, because the point of science is to build coherent theories.
>not some magical word if you don't know how something works.
I'd disagree, emergence is typically what we don't understand. When we understand it, it's rarely considered an emergent concept, just something that is.
>They don't offer a scientific explanation.
Correct, because we don't have the tooling necessary to explain it yet. Emergence as you stated came from simpler concepts at first, for example burning hydrogen and oxygen and water emerges from that.
Ecosystems are an emergent property of living systems, ones that we can explain rather well these days after we realized there were gaps in our knowledge. It's taken millions and millions of hours of research to piece all these bits together.
Now we are at the same place in large neural nets. What you say is pointless is not pointless at all. It's pointing at the exact things we need to work on if we want to have understanding of it. But at the same time understanding isn't necessary. We have made advancements in scientific topics that we don't understand.
I'd disagree, emergence is typically what we don't understand. When we understand it, it's rarely considered an emergent concept, just something that is.
>They don't offer a scientific explanation.
Correct, because we don't have the tooling necessary to explain it yet. Emergence as you stated came from simpler concepts at first, for example burning hydrogen and oxygen and water emerges from that.
Ecosystems are an emergent property of living systems, ones that we can explain rather well these days after we realized there were gaps in our knowledge. It's taken millions and millions of hours of research to piece all these bits together.
Now we are at the same place in large neural nets. What you say is pointless is not pointless at all. It's pointing at the exact things we need to work on if we want to have understanding of it. But at the same time understanding isn't necessary. We have made advancements in scientific topics that we don't understand.
> There's two kinds of emergence, one scientific
I am not aware of any scientific kind of emergence. There's philosophical emergence, and its counterpoint - ontological reductionism.
Most people have an intuitive sense that philosophical emergence is true, and that bubbles up in their writing, taken as an axiom that we're all supposed to go along with.
On closer inspection, it is not clear to me that this isn't simply a confusion or illusion caused by the tendency of the human mind to apply abstractions and socially constructed categories on top of complicated phenomena, and those abstractions are confused for actual effects that are different from the underlying base-level phenomena being described.
I am not aware of any scientific kind of emergence. There's philosophical emergence, and its counterpoint - ontological reductionism.
Most people have an intuitive sense that philosophical emergence is true, and that bubbles up in their writing, taken as an axiom that we're all supposed to go along with.
On closer inspection, it is not clear to me that this isn't simply a confusion or illusion caused by the tendency of the human mind to apply abstractions and socially constructed categories on top of complicated phenomena, and those abstractions are confused for actual effects that are different from the underlying base-level phenomena being described.
It is ontological emergence vs epistemological emergence, not philosophical vs ontological. Simply, ontic vs epistemic.
Nobody claims mistical gaps. There is no deus ex machina claim in emergence. However e.g. stable phenomena at a higher model level might be fully dynamic at a lower level model.
> the major breakthroughs of AI this decade have not, at least so far, substantially deepened our understanding of our own intelligence and its constitution
People's illusions and willingness to debase their own authority and control to take shortcuts to optimise towards lowest effort / highest yield (not dissimilar to something you would get with... auto regressive models!) was an astonishing insight to me.
People's illusions and willingness to debase their own authority and control to take shortcuts to optimise towards lowest effort / highest yield (not dissimilar to something you would get with... auto regressive models!) was an astonishing insight to me.
Well said. It's wild when you think of how many "AI" products are out there that essentially entrust an LLM to make the decisions the user would otherwise make. Recruitment, trading, content creation, investment advice, medical diagnosis, legal review, dating matches, financial planning and even hiring decisions.
At some point you have to wonder: is an LLM making your hiring decision really better than rolling a dice? At least the dice doesn't give you the illusion of rationality, it doesn't generate a neat sounding paragraph "explaining" why candidate A is the obvious choice. The LLM produces content that looks like reasoning but has no actual causal connection to the decision - it's a mimicry of explanation without true substance of causation.
You can argue that humans do the same thing. But post-hoc reasoning is often a feedback loop for the eventual answer. That's not the case for LLMs.
At some point you have to wonder: is an LLM making your hiring decision really better than rolling a dice? At least the dice doesn't give you the illusion of rationality, it doesn't generate a neat sounding paragraph "explaining" why candidate A is the obvious choice. The LLM produces content that looks like reasoning but has no actual causal connection to the decision - it's a mimicry of explanation without true substance of causation.
You can argue that humans do the same thing. But post-hoc reasoning is often a feedback loop for the eventual answer. That's not the case for LLMs.
> it doesn't generate a neat sounding paragraph "explaining" why candidate A is the obvious choice.
Here I will argue that humans do the same thing. For any business of any size recruitment has been pretty awful in recent history. The end user, that is the manager the employee will be hired under is typically a later step after a lot of other filters, some automated some not.
At the end of the day the only way is to measure the results. Do LLMs produce better hiring results than some outside group?
Also, LLMs seem very good at medical pre-diagnosis. If you accurately portray your symptoms to them they come back with a decent list of possible candidates. In barbaric nations like the US where medical care can easily lead to bankruptcy people are going to use it as a filter to determine if they should go in for a visit.
Here I will argue that humans do the same thing. For any business of any size recruitment has been pretty awful in recent history. The end user, that is the manager the employee will be hired under is typically a later step after a lot of other filters, some automated some not.
At the end of the day the only way is to measure the results. Do LLMs produce better hiring results than some outside group?
Also, LLMs seem very good at medical pre-diagnosis. If you accurately portray your symptoms to them they come back with a decent list of possible candidates. In barbaric nations like the US where medical care can easily lead to bankruptcy people are going to use it as a filter to determine if they should go in for a visit.
Chompsky's central criticism of LLMs is that they can learn impossible languages just as easily as they learn possible languages. He refers to this repeatedly in the linked interview. Therefore, they cannot teach us about our own intelligence.
However, a paper published last year (Mission: Impossible Language Models, Kallini et al.) proved that LLMs do NOT learn impossible languages as easily as they learn possible languages. This undermines everything that Chompsky says about LLMs in the linked interview.
However, a paper published last year (Mission: Impossible Language Models, Kallini et al.) proved that LLMs do NOT learn impossible languages as easily as they learn possible languages. This undermines everything that Chompsky says about LLMs in the linked interview.
I'm not that convinced by this paper. The "impossible languages" are all English with some sort of transformation applied, such as shuffling the word order. It seems like learning such languages would require first learning English and then learning the transformation. It's not surprising that systems would be worse at learning such languages than just learning English on its own. But I don't think these sorts of languages are what Chomsky is talking about. When Chomsky says "impossible languages," he means languages that have a coherent and learnable structure but which aren't compatible with what he thinks are innate grammatical facilities of the human mind. So for instance, x86 assembly language is reasonably structured and can express anything that C++ can, but unlike C++, it doesn't have a recursive tree-based syntax. Chomsky believes that any natural language you find will be structured more like C++ than like assembly language, because he thinks humans have an innate mental facility for using tree-based languages. I actually think a better test of whether LLMs learn languages like humans would be to see if they learn assembly as well as C++. That would be incomplete of course, but it would be getting at what Chomsky's talking about.
Also, GPT-2 actually seems to do quite well on some of the tested languages, including word-hop, partial reverse, and local-shuffle. It doesn't do quite as well as plain English, but GPT-2 was designed to learn English, so it's not surprising that it would do a little better. For instance, they tokenization seems biased towards English. They show "bookshelf" becoming the tokens "book", "sh", and "lf" – which in many of the languages get spread throughout a sentence. I don't think a system designed to learn shuffled-English would tokenize this way!
https://aclanthology.org/2024.acl-long.787.pdf
Also, GPT-2 actually seems to do quite well on some of the tested languages, including word-hop, partial reverse, and local-shuffle. It doesn't do quite as well as plain English, but GPT-2 was designed to learn English, so it's not surprising that it would do a little better. For instance, they tokenization seems biased towards English. They show "bookshelf" becoming the tokens "book", "sh", and "lf" – which in many of the languages get spread throughout a sentence. I don't think a system designed to learn shuffled-English would tokenize this way!
https://aclanthology.org/2024.acl-long.787.pdf
The authors of that paper misunderstand what "impossible languages" refers to. It doesn't refer to any language a human can't learn. It refers to computationally simple plausible alternative languages that humans can't learn, in particular linear-order (non-hierarchical structure) languages.
What exactly do you mean, "analogous to our own" and, "in a deep way" without making an appeal to magic or non-yet discovered fields of science? I understand what you're saying but when you scrutinize these things you end up in a place that's less scientific than one might think. That kind of seems to be one of Chomsky's salient points; we really, really need to get a handle on when we're doing science in the contemporary Kuhnian sense and philosophy.
The AI works on English, C++, Smalltalk, Klingon, nonsense, and gibberish. Like Turing's paper this illustrates the difference between, "machines being able to think" and, "machines being able to demonstrate some well understood mathematical process like pattern matching."
https://en.wikipedia.org/wiki/Computing_Machinery_and_Intell...
The AI works on English, C++, Smalltalk, Klingon, nonsense, and gibberish. Like Turing's paper this illustrates the difference between, "machines being able to think" and, "machines being able to demonstrate some well understood mathematical process like pattern matching."
https://en.wikipedia.org/wiki/Computing_Machinery_and_Intell...
> not, at least so far, substantially deepened our understanding of our own intelligence
Science progresses in a manner that when you see it happen in front of you it doesn't seem substantial at all, because we typically don't understand implications of new discoveries.
So far, in the last few years, we have discovered the importance of the role of language behind intelligence. We have also discovered quantitative ways to describe how close one concept is from another. More recently, from the new reasoning AI models, we have discovered something counterintuitive that's also seemingly true for human reasoning--incorrect/incomplete reasoning can often reach the correct conclusion.
Science progresses in a manner that when you see it happen in front of you it doesn't seem substantial at all, because we typically don't understand implications of new discoveries.
So far, in the last few years, we have discovered the importance of the role of language behind intelligence. We have also discovered quantitative ways to describe how close one concept is from another. More recently, from the new reasoning AI models, we have discovered something counterintuitive that's also seemingly true for human reasoning--incorrect/incomplete reasoning can often reach the correct conclusion.
In my opinion it will or already has redefined our conceptual models of intelligence - just like physical models of atoms or gravitational mechanics evolved and newer models replace the older. The older models aren't invalidated (all models are wrong, after all), but their limits are better understood.
People are waiting for this Prometheus-level moment with AI where it resembles us exactly but exceeds our capabilities, but I don't think that's necessary. It parallels humanity explaining Nature in our own image as God and claiming it was the other way around.
People are waiting for this Prometheus-level moment with AI where it resembles us exactly but exceeds our capabilities, but I don't think that's necessary. It parallels humanity explaining Nature in our own image as God and claiming it was the other way around.
> if the intelligence of LLMs proves to be analogous to our own in some deep way
First, they have to implement "intelligence" for LLMs, then we can compare. /s
First, they have to implement "intelligence" for LLMs, then we can compare. /s
There was an interesting debate where Chomsky took a position on intelligence being rooted in symbolic reasoning and Asimov asserted a statistical foundation (ah, that was not intentional ;).
LLM designs to date are purely statistical models. A pile, a morass of floating point numbers and their weighted relationships, along with the software and hardware that animates them and the user input and output that makes them valuable to us. An index of the data fed into them, different from a Lucene or SQL DB index made from compsci algorithms & data structure primitives. Recognizable to Azimov's definition.
And these LLMs feature no symbolic reasoning whatsoever within their computational substrate. What they do feature is a simple recursive model: Given the input so far, what is the next token? And they are thus enabled after training on huge amounts of input material. No inherent reasoning capabilities, no primordial ability to apply logic, or even infer basic axioms of logic, reasoning, thought. And therefore unrecognizable to Chomsky's definition.
So our LLMs are a mere parlor trick. A one-trick pony. But the trick they do is oh-so vastly complicated, and very appealing to us, of practical application and real value. It harkens back to the question: What is the nature of intelligence? And how to define it?
And I say this while thinking of the marked contrast of apparent intelligence between an LLM and say a 2-year age child.
LLM designs to date are purely statistical models. A pile, a morass of floating point numbers and their weighted relationships, along with the software and hardware that animates them and the user input and output that makes them valuable to us. An index of the data fed into them, different from a Lucene or SQL DB index made from compsci algorithms & data structure primitives. Recognizable to Azimov's definition.
And these LLMs feature no symbolic reasoning whatsoever within their computational substrate. What they do feature is a simple recursive model: Given the input so far, what is the next token? And they are thus enabled after training on huge amounts of input material. No inherent reasoning capabilities, no primordial ability to apply logic, or even infer basic axioms of logic, reasoning, thought. And therefore unrecognizable to Chomsky's definition.
So our LLMs are a mere parlor trick. A one-trick pony. But the trick they do is oh-so vastly complicated, and very appealing to us, of practical application and real value. It harkens back to the question: What is the nature of intelligence? And how to define it?
And I say this while thinking of the marked contrast of apparent intelligence between an LLM and say a 2-year age child.
That's not true, symbols emerge out of the statistics. Just look at the imagenet analysis that identified distinct concepts in different layers, or the experiments with ablation in LLMs.
They may not be doing strict formal logic, but they are definitely compressing information into, and operating using, symbols.
They may not be doing strict formal logic, but they are definitely compressing information into, and operating using, symbols.
Which argues much symbolic manipulation is formulaic, and not inductive reasoning or indicative of intelligence.
Sentence parsing with multiple potential verb-noun-adjective interpretations are an example of old, Chomsky made fruit flies like a banana famous for a reason.
(without the weights and that specific sentence programmed in, I would be interested exactly how the symbol models cope with that, and the myriad other examples)
Sentence parsing with multiple potential verb-noun-adjective interpretations are an example of old, Chomsky made fruit flies like a banana famous for a reason.
(without the weights and that specific sentence programmed in, I would be interested exactly how the symbol models cope with that, and the myriad other examples)
To me the interesting idea is the followup question: Can you do complex reasoning without intelligence?
LLM's seem to have proven themselves to be more than a one-trick-pony. There is actually some resemblance of reasoning and structuring etc.. No matter if directly within the LLM, or supported by computer code. E.g it can be argued that the latest LLMs like Gemini 2.5 and Claude 4 in fact do complex reasoning.
We have always taken for granted you need intelligence for that, but what if you don't? It would greatly change our view on intelligence and take away one of the main factors that we test for in e.g. animals to define their "intelligence".
LLM's seem to have proven themselves to be more than a one-trick-pony. There is actually some resemblance of reasoning and structuring etc.. No matter if directly within the LLM, or supported by computer code. E.g it can be argued that the latest LLMs like Gemini 2.5 and Claude 4 in fact do complex reasoning.
We have always taken for granted you need intelligence for that, but what if you don't? It would greatly change our view on intelligence and take away one of the main factors that we test for in e.g. animals to define their "intelligence".
> E.g it can be argued that the latest LLMs like Gemini 2.5 and Claude 4 in fact do complex reasoning.
They most definitely don't. We attach symbolic meaning to their output because we can map it semantically to the input we gave it. Which is why people are often caught by surprise when these mappings break down.
LLMs can emulate reasoning, but the failure modes show that they don't. We can get them to be coincidentally emulating reasoning well enough long enough to fools us, investors and the media. But doubling down on it hoping that this problem goes away with scale or fine tuning is proving more and more reckless.
They most definitely don't. We attach symbolic meaning to their output because we can map it semantically to the input we gave it. Which is why people are often caught by surprise when these mappings break down.
LLMs can emulate reasoning, but the failure modes show that they don't. We can get them to be coincidentally emulating reasoning well enough long enough to fools us, investors and the media. But doubling down on it hoping that this problem goes away with scale or fine tuning is proving more and more reckless.
Humans aren't infallible and make mistakes in reasoning as well. What is fundamentally different about the mistakes we make versus the mistakes that Claude or Gemini make? Haven't LLM's even been shown to make the same posthoc rationalizations of mistakes that we as humans do all the time?
Unless you're pulling humans out of the streets at random and asking them questions or to do work, I guess you also shouldn't do that with statistical models of random human language.
I think we are ignoring that the statistical aspect of our ability to reason effectively and to apply logic was predicated on the deaths of millions of our ancestors. When they made the wrong decision, they likely didn't reproduce. When they made the right decision, that particular configuration of their cortical substrate was carried forward a generation. The product of this cross-generational training could have easily led to non-intelligence, and often does, but we have survivor's bias in our favor.
Perhaps the next question we are asking is "what happens if you give a statistical model symbolic input" and the answer appears to be, you get symbolic output.
Even more strangely, the act of giving a statistical model symbolic input allows it to build a context which then shapes the symbolic output in a way that depends on some level of "understanding" instructions.
We "train" this model on raw symbolic data and it extracts the inherent semantic structure without any human ever embedding in the code anything resembling letters, words, or the like. It's as if Chomsky's elusive universal language is semantic structure itself.
Even more strangely, the act of giving a statistical model symbolic input allows it to build a context which then shapes the symbolic output in a way that depends on some level of "understanding" instructions.
We "train" this model on raw symbolic data and it extracts the inherent semantic structure without any human ever embedding in the code anything resembling letters, words, or the like. It's as if Chomsky's elusive universal language is semantic structure itself.
>There was an interesting debate where Chomsky took a position on intelligence being rooted in symbolic reasoning and Asimov asserted a statistical foundation (ah, that was not intentional ;).
Chomsky vs Norvig
https://norvig.com/chomsky.html
Chomsky vs Norvig
https://norvig.com/chomsky.html
> Chomsky took a position on intelligence being rooted in symbolic reasoning and Asimov asserted a statistical foundation
I dunno if people knew it at that time, but those two views are completely equivalent.
I dunno if people knew it at that time, but those two views are completely equivalent.
> and very appealing to us
Yes, because anthropomorphism is hardwired into our biology. Just two dots and an arc triggers a happy feeling in all humans. :)
> of practical application and real value
That is debatable. So far no groundbreaking useful applications have been found for LLMs. We want to believe, because they make us feel happy. But the results aren't there.
Yes, because anthropomorphism is hardwired into our biology. Just two dots and an arc triggers a happy feeling in all humans. :)
> of practical application and real value
That is debatable. So far no groundbreaking useful applications have been found for LLMs. We want to believe, because they make us feel happy. But the results aren't there.
The voice of reason. And, as always, the voice of reason is being vigorously ignored. Dreams of big profits and exerting control through generated lies are irresistible. And among others, HN comment threads demonstrate how even people who should know better are falling for it in droves. In fact this very thread shows how Chomsky's arguments fall on deaf ears.
Don't forget exerting control through automated surveillance. What a wonderful tool we have created for detecting whether citizens step out of line without needing giant offices full of analysts.
3.35 hrs Chomsky interview on ML Street Talk
https://youtu.be/axuGfh4UR9Q
Chomsky's in the last hour of that.
That part is unusually good btw. It's actually elegaic.
That part is unusually good btw. It's actually elegaic.
It's shocking how people are putting him down for the OP interview with just a couple of questions in 2023. The dude was 94 years old. I also did not predict where we would be in 2025 with LLMs. And neither did you. (When I say you, I mean some of the other commenters.)
Are we seriously saying that his ideas are not taken seriously? his theory of grammar/language construction was a major contributor to modern programming languages, for one.
Are we seriously saying that his ideas are not taken seriously? his theory of grammar/language construction was a major contributor to modern programming languages, for one.
The fact that we have figured out how to translate language into something a computer can "understand" should thrill linguists. Taking a word (token) and abstracting it's "meaning" as a 1,000-dimension vector seems like something that should revolutionize the field of linguistics. A whole new tool for analyzing and understanding the underlying patterns of all language!
And there's a fact here that's very hard to dispute, this method works. I can give a computer instructions and it "understands" them in a way that wasn't possible before LLMs. The main debate now is over the semantics of words like "understanding" and whether or not an LLM is conscious in the same way as a human being (it isn't).
And there's a fact here that's very hard to dispute, this method works. I can give a computer instructions and it "understands" them in a way that wasn't possible before LLMs. The main debate now is over the semantics of words like "understanding" and whether or not an LLM is conscious in the same way as a human being (it isn't).
Restricted to linguistics, LLM's supposed lack of understanding should be a non-sequitur. If the question is whether LLMs have formed a coherent ability to parse human languages, the answer is obviously yes. In fact not just human languages, as seen with multimodality the same transformer architecture seems to work well to model and generate anything with inherent structure.
I'm surprised that he doesn't mention "universal grammar" once in that essay. Maybe it so happens that humans do have some innate "universal grammar" wired in by instinct but it's clearly not _necessary_ to be able to parse things. You don't need to set up some explicit language rules or generative structure, enough data and the model learns to produce it. I wonder if anyone has gone back and tried to see if you can extract out some explicit generative rules from the learned representation though.
Since the "universal grammar" hypothesis isn't really falsifiable, at best you can hope for some generalized equivalent that's isomorphic to the platonic representation hypothesis and claim that all human language is aligned in some given latent representation, and that our brains have been optimized to be able to work in this subspace. That's at least a testable assumption, by trying to reverse engineer the geometry of the space LLMs have learned.
I'm surprised that he doesn't mention "universal grammar" once in that essay. Maybe it so happens that humans do have some innate "universal grammar" wired in by instinct but it's clearly not _necessary_ to be able to parse things. You don't need to set up some explicit language rules or generative structure, enough data and the model learns to produce it. I wonder if anyone has gone back and tried to see if you can extract out some explicit generative rules from the learned representation though.
Since the "universal grammar" hypothesis isn't really falsifiable, at best you can hope for some generalized equivalent that's isomorphic to the platonic representation hypothesis and claim that all human language is aligned in some given latent representation, and that our brains have been optimized to be able to work in this subspace. That's at least a testable assumption, by trying to reverse engineer the geometry of the space LLMs have learned.
Can LLMs actually parse human languages? Or can they react to stimuli with a trained behavioral response? Dogs can learn to sit when you say "sit", and learn to roll over when you say "roll over". But the dog doesn't parse human language; it reacts to stimuli with a trained behavioral response.
(I'm not that familiar with LLM/ML, but it seems like trained behavioral response rather than intelligent parsing. I believe this is part of why it hallucinates? It doesn't understand concepts, it just spits out words - perhaps a parrot is a better metaphor?)
(I'm not that familiar with LLM/ML, but it seems like trained behavioral response rather than intelligent parsing. I believe this is part of why it hallucinates? It doesn't understand concepts, it just spits out words - perhaps a parrot is a better metaphor?)
Dogs out of all creatures probably actually do have some parsing going on for human language. They learn it like we do, picking up context from the environment, actions, tone, facial expressions, body language, etc.
You can say 'what's that' in many different ways and a clever dog will react differently for each, even if it's the first time it's heard you say 'what's that?' In a scared tone it'll still react differently while knowing what you're asking.
They even do the cute head tilt when they're struggling to understand something.
I think people vastly underestimate the power of wetware and think animals and us are separated by a chasm, but I think it's a relatively small leap.
We base so much of our understanding of other creatures intelligence on their ability to communicate with us or express things in the ways we do. If elephants judged humans on their ability to communicate in infrasound to speak their names (yes they have names for each other) they'd wouldn't think too highly of us.
Sidenote but the latest I've heard is that elephants like us because they think we are cute.
You can say 'what's that' in many different ways and a clever dog will react differently for each, even if it's the first time it's heard you say 'what's that?' In a scared tone it'll still react differently while knowing what you're asking.
They even do the cute head tilt when they're struggling to understand something.
I think people vastly underestimate the power of wetware and think animals and us are separated by a chasm, but I think it's a relatively small leap.
We base so much of our understanding of other creatures intelligence on their ability to communicate with us or express things in the ways we do. If elephants judged humans on their ability to communicate in infrasound to speak their names (yes they have names for each other) they'd wouldn't think too highly of us.
Sidenote but the latest I've heard is that elephants like us because they think we are cute.
Oh sure, dogs definitely learn, they have about a 2 year old's level of intellect if I remember correctly? I just meant as example to the train/response thing. I believe the issue is their brains just aren't going to continue "leveling up" the way a human's does as it grows. My assumption is that this is because the AI isn't actually "understanding and thinking", but just acting according to its training. A program following source code, rather than rewriting its own source code, as it were.
Parsing? I don't think dogs "parse" language in a decompositional matter.
If they don't parse words to form an understanding of their meaning what are they doing?
LLMs are modelling the world, not just "predicting the next token". They are certainly not akin to parrots. Some examples here[1][2][3]. Anyone claiming otherwise at this point is not arguing in good faith.
[1] https://arxiv.org/abs/2405.15943
[2] https://x.com/OwainEvans_UK/status/1894436637054214509
[3] https://www.anthropic.com/research/tracing-thoughts-language...
[1] https://arxiv.org/abs/2405.15943
[2] https://x.com/OwainEvans_UK/status/1894436637054214509
[3] https://www.anthropic.com/research/tracing-thoughts-language...
Maybe it takes some world modeling to do it as well as they do, but ultimately they are just predicting the next token. These things are not mutually exclusive.
The issue is whether they are "just" predicting the next token. When people say they are stochastic parrots, they are denying any of these further capabilities. Modelling is facet of understanding and so to discover that LLMs model the world should strongly raise your credence that they do understand.
It can have a sensible conversation with you, follow your instructions, do math and physics, and write code that performs the task you described in English. Some models can create pictures and videos matching the description you gave them, or write descriptions of a video from you.
In 2023, Microsoft released a paper saying GPT4 could do things like tell you how to stack a random collection of unrelated variously-shaped objects so they don't fall over. Things have come a long way since then.
Try out one of the advanced models, and see whether you think it understands concepts.
In 2023, Microsoft released a paper saying GPT4 could do things like tell you how to stack a random collection of unrelated variously-shaped objects so they don't fall over. Things have come a long way since then.
Try out one of the advanced models, and see whether you think it understands concepts.
Animals definitely parse human language, some to a significant extent.
Like an airplane taking off, things that seem like “emergent behavior” and hard lines of human vs animal behavior are really matters of degree that, like the airplane, we don’t notice until it actually takes flight… then we think there is a clean line between flying and not flying, but there isn’t. The airplane is gradually becoming weightless until it breaks contact with the ground, and animals use and understand language, but we only notice when it seems human.
Like an airplane taking off, things that seem like “emergent behavior” and hard lines of human vs animal behavior are really matters of degree that, like the airplane, we don’t notice until it actually takes flight… then we think there is a clean line between flying and not flying, but there isn’t. The airplane is gradually becoming weightless until it breaks contact with the ground, and animals use and understand language, but we only notice when it seems human.
There actually is a clean line between flying and not flying. And that's when the lift generated is greater than the pull of earth's gravity. The fact that it "feels" weightless gradually doesn't change the fact that if lift<weight then the plane is not flying. If lift>weight, plane is flying. There is no "semi flying". If it's already airborne and lift becomes less than weight, then it stops flying and starts gliding.
The lift is an emergent behavior of molecules interacting (mostly) with the wings. But there is a hard clean cutoff between "flying" and "not flying".
The lift is an emergent behavior of molecules interacting (mostly) with the wings. But there is a hard clean cutoff between "flying" and "not flying".
Of course, but the cutoff I one of perception more than physics. The airplane is “not flying” right up until the lift generated is infinitesimally more than the weight of the aircraft. Likewise, during “flight” there are times when the lift is less than the weight, during descent. So the line seems clear but it is a matter of degree. The aircraft is not doing anything fundamentally different during the takeoff roll than during flight, it is all a matter of degree. There is no magical Change in physics or process.
"straight and level flight implies that lift equal to weight == flying too
I'm not sure if it's even up for debate that they parse human language, in fact they do it better than most people do. Parsing is just breaking up language into it's component ideas and fitting those ideas to one another to achieve meaning. We can meaningfully argue whether they are capable of novel reasoning (probably not) or whether they can apply knowledge from one domain to another (sometimes!) but parsing seems undeniable.
>Can LLMs actually parse human languages?
IMHO, no, they have nothing approaching understanding. It's Chinese Rooms[1] all the way down, just with lots of bell and whistles. Spicy autocomplete.
1. https://en.wikipedia.org/wiki/Chinese_room
IMHO, no, they have nothing approaching understanding. It's Chinese Rooms[1] all the way down, just with lots of bell and whistles. Spicy autocomplete.
1. https://en.wikipedia.org/wiki/Chinese_room
Actually, the LLMs made me realize John Searle’s “Chinese room” doesnt make much sense
Because languages have many similar concepts so the operator inside the Chinese room can understand nearly all the concepts without speaking Chinese.
And the LLM can translate to and from any language trivially, the inner layers do the actual understanding of concepts.
Because languages have many similar concepts so the operator inside the Chinese room can understand nearly all the concepts without speaking Chinese.
And the LLM can translate to and from any language trivially, the inner layers do the actual understanding of concepts.
Go ask the operator of a Chinese room to do some math they weren't taught in school, and see if the translation guide helps.
The analogy I've used before is a bright first-grader named Johnny. Johnny stumbles across a high school algebra book. Unless Johnny's last name is von Neumann, he isn't going to get anything out of that book. An LLM will.
So much for the Chinese Room.
The analogy I've used before is a bright first-grader named Johnny. Johnny stumbles across a high school algebra book. Unless Johnny's last name is von Neumann, he isn't going to get anything out of that book. An LLM will.
So much for the Chinese Room.
> Go ask the operator of a Chinese room to do some math they weren't taught in school, and see if the translation guide helps.
That analogy only holds if LLMs can solve novel problems that can be proven to not exist in any form in their training material.
That analogy only holds if LLMs can solve novel problems that can be proven to not exist in any form in their training material.
They do. Spend some time using a modern reasoning model. There is a class of interesting problems, nestled between trivial ones whose answers can simply be regurgitated and difficult ones that either yield nonsense or involve tool use, that transformer networks can absolutely, incontrovertibly reason about.
Have any LLMs solved any of the big (or even lesser known) unanswered problems in math, physics, computer science?
It may appear that they are solving novel problems but given the size of their training set they have probably seen them. There are very few questions a person can come up with that haven't already been asked and answered somewhere.
It may appear that they are solving novel problems but given the size of their training set they have probably seen them. There are very few questions a person can come up with that haven't already been asked and answered somewhere.
Google's AlphaEvolve recently produced a novel matrix multiplication function slightly faster than the previous state of the art that couldn't have been in any training data. While not a hard unsolved problem, I think it's good evidence that an LLM is capable of synthesizing new solutions to problems.
Reason about: sure. Independently solve novel ones without extreme amounts of guidance: I have yet to see it.
Granted, for most language and programming tasks, you don’t need the latter, only the former.
Granted, for most language and programming tasks, you don’t need the latter, only the former.
99.9% of humans will never solve a novel problem. It's a bad benchmark to use here
But they will solve a problem novel to them, since they haven't read all of the text that exists.
I agree. But it’s worth being somewhat skeptical of ASI scenarios if you can’t, for example, give a well formulated math problem to a LLM and it cannot solve it. Until we get a Reimann hypothesis calculator (or equivalent for hard/old unsolved maths) it’s kind of silly to be debating the extreme ends of AI cognition theory
"I'm taking this talking dog right back to the pound. It completely whiffed on both Riemann and Goldbach. And you should see the buffer overflows in the C++ code it wrote for me."
dog is a very different category man-made godlike super-intelligence
I have been able to get chatgpt to synthesize in the edges of two domains in ideaspace, say, psychology and economics, but surprisingly it struggled helping me write ODE code in go. In the first case, I think it actually synthesized. In the latter it couldn't extrapolate enough ideas from the two fields into one.
How can you distinguish "I think it did something really impressive in the first case but not the second" from "it spat out something that looked interesting in both cases but in the latter case there was an objective criteria that exposed a lack of true understanding"?
It's famously easier to impress people with soft-sciences speculation than it is to impress the rules of math or compilers.
It's famously easier to impress people with soft-sciences speculation than it is to impress the rules of math or compilers.
I think people give training data too much credit. Obviously it's important, but it also isn't a database of knowledge like it's made out to be.
You can see this in riddles that are obviously in the training set, but older or lighter models still get them wrong. Or situations where the model gets them right, but uses a different method than the ones used in the training set.
You can see this in riddles that are obviously in the training set, but older or lighter models still get them wrong. Or situations where the model gets them right, but uses a different method than the ones used in the training set.
A "Chinese Room" absolutely will, because the original thought experiment proposed no performance limits on the setup - the Room is said to pass the Turing Test flawlessly.
People keep using "Chinese Room" to mean something it isn't and it's getting annoying. It is nothing more than a (flawed) intuition pump and should not be used as an analogy for anything, let alone LLMs. "It's a Chinese Room" is nonsensical unless there is literally an ACTUAL HUMAN in the setup somewhere - its argument, invalid as it is, is meaningless in its absence.
People keep using "Chinese Room" to mean something it isn't and it's getting annoying. It is nothing more than a (flawed) intuition pump and should not be used as an analogy for anything, let alone LLMs. "It's a Chinese Room" is nonsensical unless there is literally an ACTUAL HUMAN in the setup somewhere - its argument, invalid as it is, is meaningless in its absence.
A Chinese Room has no attention model. The operator can look up symbolic and syntactical equivalences in both directions, English to Chinese and Chinese back to English, but they can't associate Chinese words with each other or arrive at broader inferences from doing so. An LLM can.
If I were to ask a Chinese room operator, "What would happen if gravity suddenly became half as strong while I'm drinking tea?," what would you expect as an answer?
Another question: if I were to ask "What would be an example of something a Chinese room's operator could not handle, that an actual Chinese human could?", what would you expect in response?
Claude gave me the first question in response to the second. That alone takes Chinese Rooms out of the realm of any discussion regarding LLMs, and vice versa. The thought experiment didn't prove anything when Searle came up with it, and it hasn't exactly aged well. Neither Searle nor Chomsky had any earthly idea that language was this powerful.
If I were to ask a Chinese room operator, "What would happen if gravity suddenly became half as strong while I'm drinking tea?," what would you expect as an answer?
Another question: if I were to ask "What would be an example of something a Chinese room's operator could not handle, that an actual Chinese human could?", what would you expect in response?
Claude gave me the first question in response to the second. That alone takes Chinese Rooms out of the realm of any discussion regarding LLMs, and vice versa. The thought experiment didn't prove anything when Searle came up with it, and it hasn't exactly aged well. Neither Searle nor Chomsky had any earthly idea that language was this powerful.
Where are you getting all this (wrong) detail about the internals of the Chinese Room? The thought experiment merely says that the operator consults "books" and follows "instructions" (no doubt Turing-complete but otherwise unspecified) for manipulating symbols they they explicitly DO NOT understand - they do NOT have access to "symbolic and syntactical equivalences" - that is the POINT of the thought experiment. But the instructions in the books in a Chinese Room could perfectly well have an attention model. The details are irrelevant, because - I stress again - Searle's Chinese Room is not cognitively limited, by definition. Its hypothetical output is indistinguishable from a Chinese human.
I tend to agree that Chinese Rooms should be kept out of LLM discussions. In addition to it being a flawed thought experiment, of all the dozens of times I've seen them brought up, not a single example has demonstrated understanding of what a Chinese Room is anyway.
I tend to agree that Chinese Rooms should be kept out of LLM discussions. In addition to it being a flawed thought experiment, of all the dozens of times I've seen them brought up, not a single example has demonstrated understanding of what a Chinese Room is anyway.
The details are irrelevant, because - I stress again - Searle's Chinese Room is not cognitively limited, by definition.
So said Searle. But without specifying what he meant, it was a circular statement at best. Punting to "it passes a Turing Test" just turns it into a different debate about a different flawed test.
The operator has no idea what he's doing. He doesn't know Chinese. He has a Borges-scale library of Chinese books and a symbol-to-symbol translation guide. He can do nothing but manipulate symbols he doesn't understand. How anyone can pass a well-administered Turing test without state retention and context-based reflection, I don't know, but we've already put more thought into this than Searle did.
So said Searle. But without specifying what he meant, it was a circular statement at best. Punting to "it passes a Turing Test" just turns it into a different debate about a different flawed test.
The operator has no idea what he's doing. He doesn't know Chinese. He has a Borges-scale library of Chinese books and a symbol-to-symbol translation guide. He can do nothing but manipulate symbols he doesn't understand. How anyone can pass a well-administered Turing test without state retention and context-based reflection, I don't know, but we've already put more thought into this than Searle did.
Give Johnny a copier and a pair of scissors and he will be able to perform more or less the same; and likely get more out of it as well, since he has a clue what he is doing.
How can you make that claim? Have you ever used an LLM that hasn't encountered high school algebra in it's training data? I don't think so.
I have at least encountered many LLMs with many school's worth of algebra knowledge, but fail miserably at algebra problems.
Similarly, they've ingested human-centuries or more of spelling bee related text, but can't reliably count the number of Rs in strawberry. (yes, I understand tokenization is to blame for a large part of this. perhaps that kind of limitation applies to other things too?)
Similarly, they've ingested human-centuries or more of spelling bee related text, but can't reliably count the number of Rs in strawberry. (yes, I understand tokenization is to blame for a large part of this. perhaps that kind of limitation applies to other things too?)
Similarly, they've ingested human-centuries or more of spelling bee related text, but can't reliably count the number of Rs in strawberry
Sigh
Sigh
That sigh might be a chronic condition, if it's happening even when people demonstrate a decent understanding of the causes. You may want to get that looked at.
An LLM will get ... what exactly ? The ability to reorder its sentences ? The LLM doesn't think, doesn't understand, doesn't know what matters more than not, doesn't use what it learns, doesn't expand what it learns to new knowledge, doesn't enjoy reading that book and doesn't suffer through it.
So what is it really gonna do with a book, that LLM ? Reorder its internal matrix to be a little bit more precise when autocompleting sentences sounding like the book ? We could build an nvidia cluster the size of the Sun and it would repeat sentences back to us in unbelievable ways but would still be unable to take a knowledge-based decision, I fear.
So what are we in awe at exactly ? A pretty parrot.
The day the Chinese room metaphor disappears is when ChatGPT replies to you that your question is so boring it doesn't want to expend the resources to think about it. But it'd be ready to talk about this or that, that it's currently trying to get better at. When it finally has agency over its own intelligence. When it acquires a purpose.
So what is it really gonna do with a book, that LLM ? Reorder its internal matrix to be a little bit more precise when autocompleting sentences sounding like the book ? We could build an nvidia cluster the size of the Sun and it would repeat sentences back to us in unbelievable ways but would still be unable to take a knowledge-based decision, I fear.
So what are we in awe at exactly ? A pretty parrot.
The day the Chinese room metaphor disappears is when ChatGPT replies to you that your question is so boring it doesn't want to expend the resources to think about it. But it'd be ready to talk about this or that, that it's currently trying to get better at. When it finally has agency over its own intelligence. When it acquires a purpose.
This isn't really the meaning of the Chinese room. The Chinese room presupposes that the output is identical to that of a speaker who understands the language. It is not arguing that there is any sort of limit to what an AI can do with its output and it is compatible with the AI refusing to answer or wanting to talk about something else.
LLM models are to a large extent neuronal analogs of human neural architecture
- of course they reason
The claim of the “stochastic parrot” needs to go away
Eg see: https://www.anthropic.com/news/golden-gate-claude
I think the rub is that people think you need consciousness to do reasoning, I’m NOT claiming LLMs have consciousness or awareness
- of course they reason
The claim of the “stochastic parrot” needs to go away
Eg see: https://www.anthropic.com/news/golden-gate-claude
I think the rub is that people think you need consciousness to do reasoning, I’m NOT claiming LLMs have consciousness or awareness
They are really not neuronal analogs, reasoning is far from what they do. If they reasoned, they'd stick to their guns more readily, but try to contradict an LLM and it will make any logic leap you ask it too.
If you debate with me, I'll keep reasoning on the same premises and usually the difference between two humans is not in reasoning but in choice of premises.
For instance you really want here to assert that LLM are close to human, I want to assert they're not - truth is probably in between but we chose two camps. We'll then reason from these premises, reach antagonistic conclusions and slowly try to attack each other point.
An LLM cannot do that, it cannot attack your point very well, it doesn't know how to say you're wrong, because it doesn't care anyway. It just completes your sentences, so if you say "now you're wrong, change your mind" it will, which sounds far from reasoning to me, and quite unreasonable in fact.
If you debate with me, I'll keep reasoning on the same premises and usually the difference between two humans is not in reasoning but in choice of premises.
For instance you really want here to assert that LLM are close to human, I want to assert they're not - truth is probably in between but we chose two camps. We'll then reason from these premises, reach antagonistic conclusions and slowly try to attack each other point.
An LLM cannot do that, it cannot attack your point very well, it doesn't know how to say you're wrong, because it doesn't care anyway. It just completes your sentences, so if you say "now you're wrong, change your mind" it will, which sounds far from reasoning to me, and quite unreasonable in fact.
Gemini 2.5 will tell you when you're wrong. It's the first model to do so.
> An LLM cannot do that, it cannot attack your point very well, it doesn't know how to say you're wrong, because it doesn't care anyway. It just completes your sentences, so if you say "now you're wrong, change your mind" it will, which sounds far from reasoning to me, and quite unreasonable in fact.
That is absolute bullshit. Go try any frontier reasoning model such as Gemini 2.5 Pro or GPT-o3 and see how that goes. They will inform you that you are full of shit.
Do you understand that they are deep learning models with hundreds of layers and trillions of parameters? They have learned patterns of reasoning, and can emulate human reasoning well enough to call you out on that nonsense.
That is absolute bullshit. Go try any frontier reasoning model such as Gemini 2.5 Pro or GPT-o3 and see how that goes. They will inform you that you are full of shit.
Do you understand that they are deep learning models with hundreds of layers and trillions of parameters? They have learned patterns of reasoning, and can emulate human reasoning well enough to call you out on that nonsense.
Gemini 2.5 will tell you when you're wrong
> LLM models are to a large extent neuronal analogs of human neural architecture
They are absolutely not. Despite the disingenuous name, computer neural nets are nothing like biological brains.
(Neural nets are a generalization of the logistic regression.)
They are absolutely not. Despite the disingenuous name, computer neural nets are nothing like biological brains.
(Neural nets are a generalization of the logistic regression.)
Language and intelligence are mostly orthogonal to each other and development of linguistic skills appeared very late in human evolutionary terms.
Babies and in particular Deaf babies understand and communicate significant amount of information w/o parsing sentences. Dogs don't parse human speech, they associate an emotion to the particular sound and body language exhibited to them, repeatedly.
Babies and in particular Deaf babies understand and communicate significant amount of information w/o parsing sentences. Dogs don't parse human speech, they associate an emotion to the particular sound and body language exhibited to them, repeatedly.
You can train LLMs on the output very complex CFGs, and it successfully learns the grammar and hierarchy needed to complete any novel prefix. This is a task much more recursive and difficult than human languages, so there's no reason to believe that LLMs aren't able to parse human languages in the formal sense as well.
And of course empirically LLMs do generate valid English sentences. They may not necessarily be _correct_ sentences in a propositional truth-value sense (as seen by so-called "hallucinations), but they are semantically "well-formed" in contrast to Chomsky's famous example of the failure of probabilistic grammar models, "Colorless green ideas sleep furiously."
I'm not a linguist but I don't think linguistics has ever cared about the truth value of a sentence, that's more under the realm of logic.
And of course empirically LLMs do generate valid English sentences. They may not necessarily be _correct_ sentences in a propositional truth-value sense (as seen by so-called "hallucinations), but they are semantically "well-formed" in contrast to Chomsky's famous example of the failure of probabilistic grammar models, "Colorless green ideas sleep furiously."
I'm not a linguist but I don't think linguistics has ever cared about the truth value of a sentence, that's more under the realm of logic.
I disagree, I think it's clear in the article that Chomsky thinks a language also should have a human purpose.
The compression we use in languages to not label impossible adjectives against impossible nouns (green ideas is impossible as ideas don't have colors, we could have a suffix on every noun to mark what can be colored and what cannot) is because we need to transfer these over the air, and quickly, before the lion jumps on the hunter. It's one of the many attributes of "languages in the wild" (Chinese doesn't use "tenses" really, can you imagine the compressive value?), and that's what Chomsky says here:
Proceeding further with normal science, we find that the internal processes and elements of the language cannot be detected by inspection of observed phenomena. Often these elements do not even appear in speech (or writing), though their effects, often subtle, can be detected. That is yet another reason why restriction to observed phenomena, as in LLM approaches, sharply limits understanding of the internal processes that are the core objects of inquiry into the nature of language, its acquisition and use. But that is not relevant if concern for science and understanding have been abandoned in favor of other goals.
Understand what he means: you can read a million text through a machine, it will never infer why we don't label adjective and nouns to prevent confusion and "green ideas". But for us it's painfully obvious, we don't have time when we speak to do all that. And I come from a language when we label every noun with a gender, I can see how stupid and painful it is to grasp for foreigners: it doesn't make any sense. Why do we do it ? Ask ChatGPT, will it tell you that it's because we like how beautiful it all sounds, which is the stupid reason why we do that ?
The compression we use in languages to not label impossible adjectives against impossible nouns (green ideas is impossible as ideas don't have colors, we could have a suffix on every noun to mark what can be colored and what cannot) is because we need to transfer these over the air, and quickly, before the lion jumps on the hunter. It's one of the many attributes of "languages in the wild" (Chinese doesn't use "tenses" really, can you imagine the compressive value?), and that's what Chomsky says here:
Proceeding further with normal science, we find that the internal processes and elements of the language cannot be detected by inspection of observed phenomena. Often these elements do not even appear in speech (or writing), though their effects, often subtle, can be detected. That is yet another reason why restriction to observed phenomena, as in LLM approaches, sharply limits understanding of the internal processes that are the core objects of inquiry into the nature of language, its acquisition and use. But that is not relevant if concern for science and understanding have been abandoned in favor of other goals.
Understand what he means: you can read a million text through a machine, it will never infer why we don't label adjective and nouns to prevent confusion and "green ideas". But for us it's painfully obvious, we don't have time when we speak to do all that. And I come from a language when we label every noun with a gender, I can see how stupid and painful it is to grasp for foreigners: it doesn't make any sense. Why do we do it ? Ask ChatGPT, will it tell you that it's because we like how beautiful it all sounds, which is the stupid reason why we do that ?
A “complex” cfg is still a cfg, and, giving credence to Chomsky’s hierarchy, remains computationally less complex than natural, context sensitive, grammars. Even a complex cfg can be parsed by a relatively simple program in ways that context-sensitive grammars cannot.
My understanding is that context sensitive grammars _can_ allow for recursive structures that are beyond cfgs, which is precisely why they sit below csgs in terms of computational complexity.
I don’t agree or disagree that LLMs might be, or are, capable of parsing (i.e., perception in Chomsky’s terms, or, arguably, “understanding” in any sense). But that they can learn the grammar of a “complex cfg” isn’t a convincing argument for the reasons you indicate.
My understanding is that context sensitive grammars _can_ allow for recursive structures that are beyond cfgs, which is precisely why they sit below csgs in terms of computational complexity.
I don’t agree or disagree that LLMs might be, or are, capable of parsing (i.e., perception in Chomsky’s terms, or, arguably, “understanding” in any sense). But that they can learn the grammar of a “complex cfg” isn’t a convincing argument for the reasons you indicate.
I don't think it's clear that human languages are context sensitive. The only consistent claim I can find is that at one point someone examined Swiss German and found that it's weakly context sensitive. Also empirically human language don't have that much recursion. You can artificially construct such examples, but beyond a certain depth people won't be able to parse it either.
I don't know whether the non-existence of papers studying whether LLMs can model context-sensitive grammar is because they can't, or because people haven't tested that hypothesis yet. But again empirically LLMs do seem to be able to reproduce human language just fine. The whole "hallucination" argument is precisely that LLMs are very good at reproducing the structure of language even if those statements don't encode things with the correct truth value. The fact that they successfully learn to parse complex CFGs is thus evidence that they can actually learn underlying generative mechanisms instead of simply parroting snippets of training data as naively assumed, and it's not a huge leap to imagine that they've learned some underlying "grammar" for English as well.
So if one argues that LLMs as a generative model cannot generate novel valid sentences in the English language, then that is easily falsifiable hypothesis. If we had examples of LLMs producing non-well formed sentences, people would have latched onto that by now, instead of "count Rs in strawbery" but I've never seen anyone arguing as such.
I don't know whether the non-existence of papers studying whether LLMs can model context-sensitive grammar is because they can't, or because people haven't tested that hypothesis yet. But again empirically LLMs do seem to be able to reproduce human language just fine. The whole "hallucination" argument is precisely that LLMs are very good at reproducing the structure of language even if those statements don't encode things with the correct truth value. The fact that they successfully learn to parse complex CFGs is thus evidence that they can actually learn underlying generative mechanisms instead of simply parroting snippets of training data as naively assumed, and it's not a huge leap to imagine that they've learned some underlying "grammar" for English as well.
So if one argues that LLMs as a generative model cannot generate novel valid sentences in the English language, then that is easily falsifiable hypothesis. If we had examples of LLMs producing non-well formed sentences, people would have latched onto that by now, instead of "count Rs in strawbery" but I've never seen anyone arguing as such.
It’s uncontroversial now that the class of string languages roughly corresponding to “human languages” is mildly context sensitive in a particular sense. This debate was hashed out in the 80s and 90s.
I don’t think formal languages classes have much to tell us about the capabilities of LLMs in any case.
>Also empirically human language don't have that much recursion. You can artificially construct such examples, but beyond a certain depth people won't be able to parse it either.
If you limit recursion depth then everything is regular, so the Chomsky hierarchy is of little application.
I don’t think formal languages classes have much to tell us about the capabilities of LLMs in any case.
>Also empirically human language don't have that much recursion. You can artificially construct such examples, but beyond a certain depth people won't be able to parse it either.
If you limit recursion depth then everything is regular, so the Chomsky hierarchy is of little application.
I’ve seen ChatGPT generate bad English and I’ve seen the layer or logic / UI re-render the page as I think there is a simple spell checker that kicks in and tells the api to re-render and recheck.
I don’t believe for one second that LLMs reason, understand, know, anything.
There are plenty of times LLMs fail to generate correct sentences, and plenty of times they fail to generate correct words.
Around the time ChatGPT rolled out web search inside actions, you’d get really funky stuff back and watch other code clearly try to catch the run away.
o3 can be hot garbage if you ask it expand a specific point inside a 3 paragraph memo, the reasoning models perform very, very poorly when they are not summarizing.
There are times where the thing works like magic, other times, asking it to write me a PowerShell script that gets users by first and last name has it inventing commands that flags that don’t exist.
If the model ‘understood’, ‘followed, some sort of structure outside parroting stuff it already knows about it would be easy to spot and guide it via prompts. That is not the case even with the most advanced models today.
It’s clear that LLMs work best at specific small tasks that have a well established pattern defined in a strict language or api.
I’ve broken o3 trying to have it lift working python code, into formal python code, how? The person that wrote the code didn’t exactly code it how a developer would code a program. 140 lines of basic grab some data generate a table broke the AI and it had the ‘informal’ solution in the prompt. So no there is zero chance LMMs do more than predict.
And to be clear, it one shot a whole thing for me last night, using the GitHub/Codex/agent thing in VS code, probably saved me 30 minutes but god forbid you start from a bad / edge / poorly structured thing that doesn’t fit the mould.
I don’t believe for one second that LLMs reason, understand, know, anything.
There are plenty of times LLMs fail to generate correct sentences, and plenty of times they fail to generate correct words.
Around the time ChatGPT rolled out web search inside actions, you’d get really funky stuff back and watch other code clearly try to catch the run away.
o3 can be hot garbage if you ask it expand a specific point inside a 3 paragraph memo, the reasoning models perform very, very poorly when they are not summarizing.
There are times where the thing works like magic, other times, asking it to write me a PowerShell script that gets users by first and last name has it inventing commands that flags that don’t exist.
If the model ‘understood’, ‘followed, some sort of structure outside parroting stuff it already knows about it would be easy to spot and guide it via prompts. That is not the case even with the most advanced models today.
It’s clear that LLMs work best at specific small tasks that have a well established pattern defined in a strict language or api.
I’ve broken o3 trying to have it lift working python code, into formal python code, how? The person that wrote the code didn’t exactly code it how a developer would code a program. 140 lines of basic grab some data generate a table broke the AI and it had the ‘informal’ solution in the prompt. So no there is zero chance LMMs do more than predict.
And to be clear, it one shot a whole thing for me last night, using the GitHub/Codex/agent thing in VS code, probably saved me 30 minutes but god forbid you start from a bad / edge / poorly structured thing that doesn’t fit the mould.
The terms are too unclear here. Can you define what it means to "be able to parse human language"? I'm sure contemporary chatbots score higher on typical reading comprehension tests than most humans. You're certainly correct that llms "only" react to stimuli with a trained response, but I guess anything that isn't consciousness necessarily fits that description
Good point, thanks for calling that out. I'm honestly not sure myself! On further reflection, it's probably a matter of degrees?
So for example, a soldier is trained, and then does what it is told. But the soldier also has a deep trough of contextual information and "decision weights" which can change its decisions, often in ways it wasn't trained for. Or perhaps to put it another way: it is capable of operating outside the parameters it was given, "if it feels like it", because the information the soldier processes at any given time may make it not follow its training.
A dog may also disobey an order after being trained, but it has a much smaller range of information it works off of, and fewer things influence its decision-making process. (genetics being a big player in the decision-making process, since they were literally bred to do what we want/defend our interests)
So perhaps a chat AI, a dog, and a soldier, are just degrees along the same spectrum. I remember reading something about how we can get AI to be about as intelligent as a 2-year-old, and that dogs are about that smart. If that's the case (and I don't know that it is; I also don't know if chat AI is actually capable of "disobeying", much less "learning" anything it isn't explicitly trained to learn), then the next question I'd have is, why isn't the AI able to act and think like a dog yet?
If we put an AI in a robot dog body and told it to act like a dog, would it? Or would it only act the way that we tell it dogs act like? Could/would it have emergent dog-like traits and spawn new dog lineages? Because as far as I'm aware, that's not how AI works yet; so to me, that would mean it's not actually doing the things we're talking about above (re: dogs/soldiers)
So for example, a soldier is trained, and then does what it is told. But the soldier also has a deep trough of contextual information and "decision weights" which can change its decisions, often in ways it wasn't trained for. Or perhaps to put it another way: it is capable of operating outside the parameters it was given, "if it feels like it", because the information the soldier processes at any given time may make it not follow its training.
A dog may also disobey an order after being trained, but it has a much smaller range of information it works off of, and fewer things influence its decision-making process. (genetics being a big player in the decision-making process, since they were literally bred to do what we want/defend our interests)
So perhaps a chat AI, a dog, and a soldier, are just degrees along the same spectrum. I remember reading something about how we can get AI to be about as intelligent as a 2-year-old, and that dogs are about that smart. If that's the case (and I don't know that it is; I also don't know if chat AI is actually capable of "disobeying", much less "learning" anything it isn't explicitly trained to learn), then the next question I'd have is, why isn't the AI able to act and think like a dog yet?
If we put an AI in a robot dog body and told it to act like a dog, would it? Or would it only act the way that we tell it dogs act like? Could/would it have emergent dog-like traits and spawn new dog lineages? Because as far as I'm aware, that's not how AI works yet; so to me, that would mean it's not actually doing the things we're talking about above (re: dogs/soldiers)
> If the question is whether LLMs have formed a coherent ability to parse human languages, the answer is obviously yes.
No, not "obviously". They work well for languages like English or Chinese, where word order determines grammar.
They work less well where context is more important. (e.g. Grammatical gender consistency.)
No, not "obviously". They work well for languages like English or Chinese, where word order determines grammar.
They work less well where context is more important. (e.g. Grammatical gender consistency.)
Alternatively, what Chomsky was thinking about with his universal grammar idea is something implicitly present in both our minds and an LLM i.e. "it's the wiring, stupid".
I'm not sure there's much evidence for this one way or another at this point.
I'm not sure there's much evidence for this one way or another at this point.
>is something implicitly present in both our minds and an LLM
The LLM doesn't start with any real structure besides the network of ops though. If there is any induced structure, it's learnable from the data. And given enough data the base network is sufficient to learn the "grammar" of not just human language but more complex CFGs and things you wouldn't traditionally consider "languages" as well (e.g. audio, images). In a sort of chicken/egg scenario, the morasses of data gives rise to the structures needed to parse and generate that data.
The LLM doesn't start with any real structure besides the network of ops though. If there is any induced structure, it's learnable from the data. And given enough data the base network is sufficient to learn the "grammar" of not just human language but more complex CFGs and things you wouldn't traditionally consider "languages" as well (e.g. audio, images). In a sort of chicken/egg scenario, the morasses of data gives rise to the structures needed to parse and generate that data.
by "parse" I usually assume I get out some sort of AST I can walk and manipulate. LLMs do no such thing. There is no parsing going on.
Unfortunately you've undermined your point by making sweeping claims about something that is the literal hardest known problem in philosophy (consciousness).
I'm not actually comfortable saying that LLMs aren't conscious. I think there's a decent chance they could be in a very alien way.
I realize that this is a very weird and potentially scary claim for people to parse but you must understand how weird and scary consciousness is.
I'm not actually comfortable saying that LLMs aren't conscious. I think there's a decent chance they could be in a very alien way.
I realize that this is a very weird and potentially scary claim for people to parse but you must understand how weird and scary consciousness is.
Note that I didn't say they aren't conscious, I said they aren't conscious "in the same way as a human being". I left open the possibility they could be conscious "in a very alien way".
[deleted]
If they are, knowing what they potentially know about humans by now, they would probably do their best to hide it.
I don't think their experience of consciousness would be very analogous to what you and I understand.
> whether or not an LLM is conscious in the same way as a human being
The problem is... that there is a whole amount of "smart" activities humans do without being conscious of it.
- Walking, riding a bike, or typing on a keyboard happen fluidly without conscious planning of each muscle movement.
- You can finish someone sentence or detect if a sentence is grammatically wrong, often without being able to explain the rule.
- When you enter a room, your brain rapidly identifies faces, furniture, and objects without you consciously thinking, “That is a table,” or “That is John.”
The problem is... that there is a whole amount of "smart" activities humans do without being conscious of it.
- Walking, riding a bike, or typing on a keyboard happen fluidly without conscious planning of each muscle movement.
- You can finish someone sentence or detect if a sentence is grammatically wrong, often without being able to explain the rule.
- When you enter a room, your brain rapidly identifies faces, furniture, and objects without you consciously thinking, “That is a table,” or “That is John.”
Indeed, the "rider and elephant" issue.
During Covid I gave a lecture on Python on Zoom in a non-English language. It was a beginner's topic about dictionary methods. I was attempting to multi-task and had other unrelated tasks open on second computer.
Midway through the lecture I noticed to my horror that I had switched to English without the audience noticing.
Going back through the recording I noticed the switch was fluid and my delivery was reasonable. What I talked about was just as good as something presented by LLM these days.
So this brings up the question - why aren't we p-zombies all the time instead of 99% of time?
Are there any tasks that absolutely demand human consciousness as we know it?
Presumably long term planning is something that active human consciousness is needed.
Perhaps there is some need for consciousness when one is in "conscious mastery" phase of acquiring a skill.
This goes for any skill such as riding a bicycle/playing chess/programming at a high level.
Once one reaches "unconscious mastery" stage the rider can concentrate on higher meta game.
During Covid I gave a lecture on Python on Zoom in a non-English language. It was a beginner's topic about dictionary methods. I was attempting to multi-task and had other unrelated tasks open on second computer.
Midway through the lecture I noticed to my horror that I had switched to English without the audience noticing.
Going back through the recording I noticed the switch was fluid and my delivery was reasonable. What I talked about was just as good as something presented by LLM these days.
So this brings up the question - why aren't we p-zombies all the time instead of 99% of time?
Are there any tasks that absolutely demand human consciousness as we know it?
Presumably long term planning is something that active human consciousness is needed.
Perhaps there is some need for consciousness when one is in "conscious mastery" phase of acquiring a skill.
This goes for any skill such as riding a bicycle/playing chess/programming at a high level.
Once one reaches "unconscious mastery" stage the rider can concentrate on higher meta game.
Phantoms in the Brain [1] has fantastic examples of the type of scenarios you described.
[1] - https://www.goodreads.com/book/show/31555.Phantoms_in_the_Br...
[1] - https://www.goodreads.com/book/show/31555.Phantoms_in_the_Br...
Why would that thrill linguists? I'm not saying it hasn't/wouldn't/shouldn't, but I don't see why this technology would have the dramatic impact you imagine.
Is/was the same true for ASCII/Smalltalk/binary? They are all another way to translate language into something the computer "understands".
Perhaps the fact that it hasn't would lead some to question the validity of their claims. When a scientist makes a claim about how something works, it's expected that they prove it.
If the technology is as you say, show us.
Is/was the same true for ASCII/Smalltalk/binary? They are all another way to translate language into something the computer "understands".
Perhaps the fact that it hasn't would lead some to question the validity of their claims. When a scientist makes a claim about how something works, it's expected that they prove it.
If the technology is as you say, show us.
> Is/was the same true for ASCII/Smalltalk/binary? They are all another way to translate language into something the computer "understands".
That's converting characters into a digital representation. "A" is represented as 01000001. The tokenization process for an LLM is similar, but it's only the first step.
An LLM isn't just mapping a word to a number, you're taking the entire sentence, considering the position of the words and converting it into vectors within a 1,000+ dimensional space. Machine learning has encoded some "meaning" within these dimensions that goes far far beyond something like an ASCII string.
And the proof here is that the method actual works, that's why we have LLMs.
That's converting characters into a digital representation. "A" is represented as 01000001. The tokenization process for an LLM is similar, but it's only the first step.
An LLM isn't just mapping a word to a number, you're taking the entire sentence, considering the position of the words and converting it into vectors within a 1,000+ dimensional space. Machine learning has encoded some "meaning" within these dimensions that goes far far beyond something like an ASCII string.
And the proof here is that the method actual works, that's why we have LLMs.
Word embeddings (that 1000-dimension vector you mention) are not new. No comment on the rest of your comment, but that aspect of LLMs is "old" tech - word2vec was published 11 years ago.
> The fact that we have figured out how to translate language into something a computer can "understand" should thrill linguists.
But I think it is important to distinguish the ability to understand language and translate it. Enough that you yourself put quotes around "understanding". This can often be a challenge for many translators, not knowing how to properly translate something because of underlying context.
Our communication runs far deeper than the words we speak or write on a page. This is much of what linguistics is about, this depth. (Or at least that's what they've told me, since I'm not a linguist) This seems to be the distinction Chomsky is trying to make.
> The main debate now is over the semantics of words like "understanding" and whether or not an LLM is conscious in the same way as a human being (it isn't).
My background is physics, and I moved into CS (degrees in both), working on ML. I see my peers at the top like Hinton[0] and Sutskever[1] making absurd claims. I call them absurd, because it is a mistake we've made over and over in the field of physics[2,3]. One of those lessons you learn again and again, because it is so easy to make the mistake. Hinton and Sutskever say that this is a feature, not a bug. Yet we know it is not enough to fit the data. Fitting the data allows you to make accurate, testable predictions. But it is not enough to model the underlying causal structure. Science has a long history demonstrating accurate predictions with incorrect models. Not just in the way of the Relativity of Wrong[4], but more directly. Did we forget that the Geocentric Model could still be used to make good predictions? Copernicus did not just face resistance from religious authorities, but also academics. The same is true for Galileo, Boltzmann, Einstein and many more. People didn't reject their claims because they were unreasonable. They rejected the claims because there were good reasons to. Just... not enough to make them right.
[0] https://www.reddit.com/r/singularity/comments/1dhlvzh/geoffr...
[1] https://www.youtube.com/watch?v=Yf1o0TQzry8&t=449s
[2] https://www.youtube.com/watch?v=hV41QEKiMlM
[3] Think about what Fermi said in order to understand the relevance of this link: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
[4] https://hermiene.net/essays-trans/relativity_of_wrong.html
"The fact that we have figured out how to translate language into something a computer can "understand" should thrill linguists."
No, there is no understanding at all. Please don't confuse codifying with understanding or translation. LLMs don't understand their input, they simply act on it based on the way they are trained on it.
"And there's a fact here that's very hard to dispute, this method works. I can give a computer instructions and it "understands" them "
No, it really does not understand those instructions. It is at best what used to be called an "idiot savant". Mind you, people used to describe others like that - who is the idiot?
Ask your favoured LLM to write a programme in a less used language - ooh let's try VMware's PowerCLI (it's PowerShell so quite popular) and get it to do something useful. It wont because it can't but it will still spit out something. PowerCLI is not extant across Stackoverflow and co much but it is PS based so the LLMs will hallucinate madder than a hippie on a new super weed.
No, there is no understanding at all. Please don't confuse codifying with understanding or translation. LLMs don't understand their input, they simply act on it based on the way they are trained on it.
"And there's a fact here that's very hard to dispute, this method works. I can give a computer instructions and it "understands" them "
No, it really does not understand those instructions. It is at best what used to be called an "idiot savant". Mind you, people used to describe others like that - who is the idiot?
Ask your favoured LLM to write a programme in a less used language - ooh let's try VMware's PowerCLI (it's PowerShell so quite popular) and get it to do something useful. It wont because it can't but it will still spit out something. PowerCLI is not extant across Stackoverflow and co much but it is PS based so the LLMs will hallucinate madder than a hippie on a new super weed.
I think the overarching theme that I glean from LLM critics is some kind of visceral emotional reaction, disgust even, with the idea of them, leading to all these proxy arguments and side quests in order to try and denigrate the idea of them without actually honestly engaging with what they are or why people are interacting with them.
so what they don't "understand", by your very specific definition of the word "understanding"? the person you're replying to is talking about the fact that they can say something to their computer in the form of casual human language and it will produce a useful response, where previously that was not true. whether that fits your suspiciously specific definition of "understanding" does not matter a bit.
so what they are over-confident with areas outside of their training data? provide more training data, improve the models, reduce the hallucination. it isn't an issue with the concept, it's an issue with the execution. yes you'll never be able to reduce it to 0%, but so what? humans hallucinate too. what are we aiming for? omniscience?
so what they don't "understand", by your very specific definition of the word "understanding"? the person you're replying to is talking about the fact that they can say something to their computer in the form of casual human language and it will produce a useful response, where previously that was not true. whether that fits your suspiciously specific definition of "understanding" does not matter a bit.
so what they are over-confident with areas outside of their training data? provide more training data, improve the models, reduce the hallucination. it isn't an issue with the concept, it's an issue with the execution. yes you'll never be able to reduce it to 0%, but so what? humans hallucinate too. what are we aiming for? omniscience?
You're thinking like an engineer and not a scientist. It's fine, but don't confuse the two.
perhaps I'm thinking like an engineer, but the people I'm referring to are thinking like priests. I don't know who in this debate is playing the role of scientist, but it surely isn't the people parroting irrelevant mock-philosophical quibbles and semantics
https://magazine.caltech.edu/post/math-language-marcolli-noa...
These days, Chomsky is working on Hopf algebras (originally from quantum physics) to explain language structure.
These days, Chomsky is working on Hopf algebras (originally from quantum physics) to explain language structure.
Brains don't have innate grammar more than languages are selected to fit baby brains. Chomsky got it backwards, languages co-evolved with human brains to fit our capacities and needs. If a language is not useful or can't be learned by children, it does not expand, it just disappears.
It's like wondering how well your shoes fit your feet, forgetting that shoes are made and chosen to fit your feet in the first place.
It's like wondering how well your shoes fit your feet, forgetting that shoes are made and chosen to fit your feet in the first place.
It's not an either/or. The fact any human language is learnable by any human and not by, say, chimpanzees needs explaining.
Chomsky also talks about these kind of things in detail in Hauser, Chomsky and Fitch (2002) where they describe them as "third factors" in language acquisition.
Chomsky also talks about these kind of things in detail in Hauser, Chomsky and Fitch (2002) where they describe them as "third factors" in language acquisition.
You could say that languages developed ("evolved") to fit the indisputable human biological faculty for language.
It's amusing that he argues (correctly) that "there is no Great Chain of Being with humans at the top," but then claims that LLMs cannot tell us anything about language because they can learn "impossible languages" that infants cannot learn. Isn't that an anthropomorphic argument, saying that what a language is inherently defined by human cognition?
When Chomsky says "language," he means "natural/human language," not e.g. /[ab]*/ or prime numbers.
Yes, studying human language is actually inherently defined by what humans do, just -- as he points out, if you could understand the article -- studying insect navigation is defined by what insects do and not what navigation systems human could design.
"The desert ants in my backyard have minuscule brains, but far exceed human navigational capacities, in principle, not just performance. There is no Great Chain of Being with humans at the top."
This quote brought to mind the very different technological development path of the spider species in Adrian Tchaikovsky's Children of Time. They used pheromones to 'program' a race of ants to do computation.
This quote brought to mind the very different technological development path of the spider species in Adrian Tchaikovsky's Children of Time. They used pheromones to 'program' a race of ants to do computation.
I don't know what he's talking about. Humans clearly outperform ants in navigation. Especially if you allow arbitrary markings on the terrain.
Sounds like "ineffable nature" mumbo-jumbo.
Sounds like "ineffable nature" mumbo-jumbo.
Arbitrary markings on the terrain? Why not GPS, satellite photo etc? All of those are human inventions and we can navigate much better and in a broader set of environments than ants thanks to them.
Because I was making a stronger argument (strict vs relaxed requirements on humans) to preempt another nature mumbo-jumbo argument that GPS is not the same/cheating.
>Many biological organisms surpass human cognitive capacities in much deeper ways. The desert ants in my backyard have minuscule brains, but far exceed human navigational capacities, in principle, not just performance. There is no Great Chain of Being with humans at the top.
Chomsky made interesting points regarding the performance of AI with the performance of biological organisms in comparison to human but his conclusion is not correct. We already know that cheetah run faster human and elephant is far stronger than human. Bat can navigate in the dark with echo location and dolphin can hunt in synchronization with high precision coordination in pack to devastating effect compared to silo hunting.
Whether we like or not human is the the top unlike the claim of otherwise by Chomsky. By scientific discovery (understanding) and designing (engineering) by utilizing law of nature, human can and has surpassed all of the cognitive capabilities of these petty animals, and we're mostly responsible for their inevitable demise and extinction. Human now need to collectively and consciously reverse the extinction process of these "superior" cognitive animals in order to preserve these animals for better or worst. No other earth bound creature can do that to us.
Chomsky made interesting points regarding the performance of AI with the performance of biological organisms in comparison to human but his conclusion is not correct. We already know that cheetah run faster human and elephant is far stronger than human. Bat can navigate in the dark with echo location and dolphin can hunt in synchronization with high precision coordination in pack to devastating effect compared to silo hunting.
Whether we like or not human is the the top unlike the claim of otherwise by Chomsky. By scientific discovery (understanding) and designing (engineering) by utilizing law of nature, human can and has surpassed all of the cognitive capabilities of these petty animals, and we're mostly responsible for their inevitable demise and extinction. Human now need to collectively and consciously reverse the extinction process of these "superior" cognitive animals in order to preserve these animals for better or worst. No other earth bound creature can do that to us.
Chomsky has the ability to say things in a way that most laypersons of average intelligence can grasp. That is an important skill for communication of one's thoughts to the general populace.
Many of the comments herein lack that feature and seem to convey that the author might be full of him(her)self.
Also, some of the comment are a bit pejorative.
Many of the comments herein lack that feature and seem to convey that the author might be full of him(her)self.
Also, some of the comment are a bit pejorative.
I once heard that a roomful of monkeys with typewriters given infinite time could type out the works of shakespeare. I dont think that's true any more than the random illumiination of pixels on a screen could eventually generate a picture.
OTOH, consider LLMs as a roomful of monkeys that can communicate to each other, look at words,sentences and paragraphs on posters around the room with a human in the room that gives them a banana when they type out a new word, sentence or paragraph.
You may eventually get a roomful of monkeys that can respond to a new sentence you give them with what seems an intelligent reply. And since language is the creation of humans, it represents an abstraction of the world made by humans.
OTOH, consider LLMs as a roomful of monkeys that can communicate to each other, look at words,sentences and paragraphs on posters around the room with a human in the room that gives them a banana when they type out a new word, sentence or paragraph.
You may eventually get a roomful of monkeys that can respond to a new sentence you give them with what seems an intelligent reply. And since language is the creation of humans, it represents an abstraction of the world made by humans.
Always a polarising figure, responses here bisect along several planes. I am sure some come armed to disagree because of his life long affinity to left world view, others to defend because of his centrality to theories of language.
I happen to agree with his view, so i came armed to agree and read this with a view in mind which I felt was reinforced. People are overstating the AGI qualities and misapplying the tool, sometimes the same people.
In particular, the lack of theory, and scientific method means both we're, not learning much, and we've rei-ified the machine.
I was disappointed nothing said of Norbert Weiner. A man who invented cybernetics and had the courage to stand up to the military industrial complex.
I happen to agree with his view, so i came armed to agree and read this with a view in mind which I felt was reinforced. People are overstating the AGI qualities and misapplying the tool, sometimes the same people.
In particular, the lack of theory, and scientific method means both we're, not learning much, and we've rei-ified the machine.
I was disappointed nothing said of Norbert Weiner. A man who invented cybernetics and had the courage to stand up to the military industrial complex.
Quite a nice overview. For almost any specific measure, you can find something that is better than human at that point. And now LLMs architecture have made possible for computers to produce complete and internally consistent paragraphs of text, by rehashing all the digital data that can be found on the internet.
But what we're good as using all of our capabilities to transform the world around us according to an internal model that is partially shared between individuals. And we have complete control over that internal model, diverging from reality and converging towards it on whims.
So we can't produce and manipulate text faster, but rarely the end game is to produce and manipulate text. Mostly it's about sharing ideas and facts (aka internal models) and the control is ultimately what matters. It can help us, just like a calculator can help us solve an equation.
EDIT
After learning to draw, I have that internal model that I switch to whenever I want to sketch something. It's like a special mode of observation, where you no longer simply see, but pickup a lot of extra details according to all the drawing rules you internalized. There's not a lot, they're just intrinsically connected with each other. The difficult part is hand-eye coordination and analyzing the divergences between what you see and the internal model.
I think that's why a lot of artists are disgusted with AI generators. There's no internal models. Trying to extract one from a generated picture is a futile exercice. Same with generated texts. Alterations from the common understanding follows no patterns.
But what we're good as using all of our capabilities to transform the world around us according to an internal model that is partially shared between individuals. And we have complete control over that internal model, diverging from reality and converging towards it on whims.
So we can't produce and manipulate text faster, but rarely the end game is to produce and manipulate text. Mostly it's about sharing ideas and facts (aka internal models) and the control is ultimately what matters. It can help us, just like a calculator can help us solve an equation.
EDIT
After learning to draw, I have that internal model that I switch to whenever I want to sketch something. It's like a special mode of observation, where you no longer simply see, but pickup a lot of extra details according to all the drawing rules you internalized. There's not a lot, they're just intrinsically connected with each other. The difficult part is hand-eye coordination and analyzing the divergences between what you see and the internal model.
I think that's why a lot of artists are disgusted with AI generators. There's no internal models. Trying to extract one from a generated picture is a futile exercice. Same with generated texts. Alterations from the common understanding follows no patterns.
> It can help us, just like a calculator can help us solve an equation.
A calculator is consistent and doesn’t “hallucinate” answers to equations. An LLM puts an untrustworthy filter between the truth and the person. Google was revolutionary because it increased access to information. LLMs only obscure that access, while pretending to be something more.
A calculator is consistent and doesn’t “hallucinate” answers to equations. An LLM puts an untrustworthy filter between the truth and the person. Google was revolutionary because it increased access to information. LLMs only obscure that access, while pretending to be something more.
I'm not a native english speaker, so I've used for an essay where they told us to target a certain word count. I was close, but the verbiage to get to that word count doesn't come naturally to me. So I used Germini and tell it to rewrite the text targeting that word count (my only prompt). Then I reviewed the answer, rewriting where it strayed from the points I was making.
Also I used it for a few programming tasks I was pretty sure was in the datasets (how to draw charts with python and manipulate pandas frame). I know the domain, but wasn't in the mood to analyse the docs to get the implementation information. But the information I was seeking was just a few lines of sample code. In my experience, anything longer is pretty inconsistent and worthless explanations.
Also I used it for a few programming tasks I was pretty sure was in the datasets (how to draw charts with python and manipulate pandas frame). I know the domain, but wasn't in the mood to analyse the docs to get the implementation information. But the information I was seeking was just a few lines of sample code. In my experience, anything longer is pretty inconsistent and worthless explanations.
Word count targets are a rough guideline for how much detail is expected; adding more useless filler is the last thing you want.
>For almost any specific measure, you can find something that is better than human at that point.
Learning language from small data.
Learning language from small data.
(2023)
From what I've heard, Chomsky had a stroke which impacted his language. You will, unfortunately, not hear a recent opinion from him on current developments.
Geez, talk about irony. That's terrible.
He's currently 96, and I started noticing in his 2022-2023 interviews that he seemed to lose a bit of the spark, so while it is always sad and moving to see such things, I don't know how much Chomsky left we were getting anyway.
Tempus fugit.
Tempus fugit.
Yean, a lot has happened in two years
[deleted]
Chat Gpt can write great apolgia for blood thirsty landempires and never live that down :
"To characterize a structural analysis of state violence as “apologia” reveals more about prevailing ideological filters than about the critique itself. If one examines the historical record without selective outrage, the pattern is clear—and uncomfortable for all who prefer myths to mechanisms." the fake academic facade, the us diabolism, the unwillingness to see complexity and responsibility in other its all with us forever ..
"To characterize a structural analysis of state violence as “apologia” reveals more about prevailing ideological filters than about the critique itself. If one examines the historical record without selective outrage, the pattern is clear—and uncomfortable for all who prefer myths to mechanisms." the fake academic facade, the us diabolism, the unwillingness to see complexity and responsibility in other its all with us forever ..
Is Chomsky really "one of the most esteemed public intellectuals of all time"? Aristotle and Beyoncé might want to have a word
I imagine his opinions might have changed by now. If we're still residing in 2023, I would be inclined to agree with him. Today, in 2025 however, LLMs are just another tool being used to "reduce labor costs" and extract more profit from the humans left who have money. There will be no scientific developments if things continue in this manner.
Two year old interview should be labeled as such.
In my view, there is a major flaw in his argument is his distinction into pure engineering and science:
> We can make a rough distinction between pure engineering and science. There is no sharp boundary, but it’s a useful first approximation. Pure engineering seeks to produce a product that may be of some use. Science seeks understanding. If the topic is human intelligence, or cognitive capacities of other organisms, science seeks understanding of these biological systems.
If you take this approach, of course it follows that we should laugh at Tom Jones.
But a more differentiated approach is to recognize that science also falls into (at least) two categories; the science that we do because it expands our capability into something that we were previously incapable of, and the one that does not. (we typically do a lot more of the former than the latter, for obvious practical reasons)
Of course it is interesting from a historical perspective to understand the seafaring exploits of Polynesians, but as soon as there was a better way of navigating (i.e. by stars or by GPS) the investigation of this matter was relegated to the second type of science, more of a historical kind of investigation. Fundamentally we investigate things in science that are interesting because we believe the understanding we can gain from it can move us forwards somehow.
Could it be interesting to understand how Hamilton was thinking when he came up with imaginary numbers? Sure. Are a lot of mathematicians today concerning themselves with studying this? No, because the frontier has been moved far beyond.*
When you take this view, it´s clear that his statement
> These considerations bring up a minor problem with the current LLM enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at once. But there are much more serious problems than absurdity.
is not warranted. Consider the following, in his own analogy:
> These considerations bring up a minor problem with the current GPS enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at ones. But there are much more serious problems than absurdity. One is that GPS systems are designed in such a way that they cannot tell us anything about navigation, planning routes or other aspects of orientation, a matter of principle, irremediable.
* I´m making a simplifying assumption here that we can´t learn anything useful for modern navigation anymore from studying Polynesians or ants; this might well be untrue, but that is also the case for learning something about language from LLMs, which according to Chomsky is apparently impossible and not even up for debate.
> We can make a rough distinction between pure engineering and science. There is no sharp boundary, but it’s a useful first approximation. Pure engineering seeks to produce a product that may be of some use. Science seeks understanding. If the topic is human intelligence, or cognitive capacities of other organisms, science seeks understanding of these biological systems.
If you take this approach, of course it follows that we should laugh at Tom Jones.
But a more differentiated approach is to recognize that science also falls into (at least) two categories; the science that we do because it expands our capability into something that we were previously incapable of, and the one that does not. (we typically do a lot more of the former than the latter, for obvious practical reasons)
Of course it is interesting from a historical perspective to understand the seafaring exploits of Polynesians, but as soon as there was a better way of navigating (i.e. by stars or by GPS) the investigation of this matter was relegated to the second type of science, more of a historical kind of investigation. Fundamentally we investigate things in science that are interesting because we believe the understanding we can gain from it can move us forwards somehow.
Could it be interesting to understand how Hamilton was thinking when he came up with imaginary numbers? Sure. Are a lot of mathematicians today concerning themselves with studying this? No, because the frontier has been moved far beyond.*
When you take this view, it´s clear that his statement
> These considerations bring up a minor problem with the current LLM enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at once. But there are much more serious problems than absurdity.
is not warranted. Consider the following, in his own analogy:
> These considerations bring up a minor problem with the current GPS enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at ones. But there are much more serious problems than absurdity. One is that GPS systems are designed in such a way that they cannot tell us anything about navigation, planning routes or other aspects of orientation, a matter of principle, irremediable.
* I´m making a simplifying assumption here that we can´t learn anything useful for modern navigation anymore from studying Polynesians or ants; this might well be untrue, but that is also the case for learning something about language from LLMs, which according to Chomsky is apparently impossible and not even up for debate.
I came to comments to ask a question, but considering that it is two days old already, I will try to ask you in this thread.
What you think about his argument about “not being able to distinguish possible language from impossible”?
And why is it inherent in ML design?
Does he assume that there could be such an instrument/algorithm that could do that with a certainty level higher than LLM/some ml model?
I mean, certainly they can be used to make a prediction/answer to this question, but he argues that this answer has no credibility? I mean, LLM is literally a model, ie probability distribution over what is language and what is not, what gives?
Current models are probably tuned more “strictly” to follow existing languages closely, ie that will say “no-no” to some yet-unknown language, but isn’t this improvable in theory?
Or is he arguing precisely that this “exterior” is not directly correlated with “internal processes and faculties” and cannot make such predictions in principle?
What you think about his argument about “not being able to distinguish possible language from impossible”?
And why is it inherent in ML design?
Does he assume that there could be such an instrument/algorithm that could do that with a certainty level higher than LLM/some ml model?
I mean, certainly they can be used to make a prediction/answer to this question, but he argues that this answer has no credibility? I mean, LLM is literally a model, ie probability distribution over what is language and what is not, what gives?
Current models are probably tuned more “strictly” to follow existing languages closely, ie that will say “no-no” to some yet-unknown language, but isn’t this improvable in theory?
Or is he arguing precisely that this “exterior” is not directly correlated with “internal processes and faculties” and cannot make such predictions in principle?
All this interview proves is that Chomsky has fallen far, far behind how AI systems work today and is retreating to scoff at all the progress machine learning has achieved. Machine learning has given rise to AI now. It can't explain itself from principles or its architecture. But you couldn't explain your brain from principles or its architecture, you'd need all of neuroscience to do it. Because the brain is digital and (probably) does not reason like our brains do, it somehow falls short?
While there's some things in this I find myself nodding along to in this, I can't help but feel it's an a really old take that is super vague and hand-wavy. The truth is that all of the progress on machine learning is absolutely science. We understand extremely well how to make neural networks learn efficiently; it's why the data leads anywhere at all. Backpropagation and gradient descent are extraordinarily powerful. Not to mention all the "just engineering" of making chips crunch incredible amounts of numbers.
Chomsky is extremely ungenerous to the progress and also pretty flippant about what this stuff can do.
I think we should probably stop listening to Chomsky; he hasn't said anything here that he hasn't already say a thousand times for decades.
While there's some things in this I find myself nodding along to in this, I can't help but feel it's an a really old take that is super vague and hand-wavy. The truth is that all of the progress on machine learning is absolutely science. We understand extremely well how to make neural networks learn efficiently; it's why the data leads anywhere at all. Backpropagation and gradient descent are extraordinarily powerful. Not to mention all the "just engineering" of making chips crunch incredible amounts of numbers.
Chomsky is extremely ungenerous to the progress and also pretty flippant about what this stuff can do.
I think we should probably stop listening to Chomsky; he hasn't said anything here that he hasn't already say a thousand times for decades.
> Not to mention all the "just engineering" of making chips crunch incredible amounts of numbers.
Are LLM's still the same black box as they were described as a couple years ago? Are their inner workings at least slightly better understood than in the past?
Running tens of thousands of chips crunching a bajillion numbers a second sounds fun, but that's not automatically "engineering". You can have the same chips crunching numbers with the same intensity just to run an algorithm to run a large prime number. Chips crunching numbers isn't automatically engineering IMO. More like a side effect of engineering? Or a tool you use to run the thing you built?
What happens when we build something that works, but we don't actually know how? We learn about it through trial and error, rather than foundational logic about the technology.
Sorta reminds me of the human brain, psychology, and how some people think psychology isn't science. The brain is a black box kind of like a LLM? Some people will think it's still science, others will have less respect.
This perspective might be off base. It's under the assumption that we all agree LLM's are a poorly understood black box and no one really knows how they truly work. I could be completely wrong on that, would love for someone else to weigh in.
Separately, I don't know the author, but agreed it reads more like a pop sci book. Although I only hope to write as coherently as that when I'm 96 y/o.
Are LLM's still the same black box as they were described as a couple years ago? Are their inner workings at least slightly better understood than in the past?
Running tens of thousands of chips crunching a bajillion numbers a second sounds fun, but that's not automatically "engineering". You can have the same chips crunching numbers with the same intensity just to run an algorithm to run a large prime number. Chips crunching numbers isn't automatically engineering IMO. More like a side effect of engineering? Or a tool you use to run the thing you built?
What happens when we build something that works, but we don't actually know how? We learn about it through trial and error, rather than foundational logic about the technology.
Sorta reminds me of the human brain, psychology, and how some people think psychology isn't science. The brain is a black box kind of like a LLM? Some people will think it's still science, others will have less respect.
This perspective might be off base. It's under the assumption that we all agree LLM's are a poorly understood black box and no one really knows how they truly work. I could be completely wrong on that, would love for someone else to weigh in.
Separately, I don't know the author, but agreed it reads more like a pop sci book. Although I only hope to write as coherently as that when I'm 96 y/o.
> Running tens of thousands of chips crunching a bajillion numbers a second sounds fun, but that's not automatically "engineering".
Not if some properties are unexpectedly emergent. Then it is science. For instance, why should a generic statistical model be able to learn how to fill in blanks in text using a finite number of samples? And why should a generic blank-filler be able to produce a coherent chat bot that can even help you write code?
Some have even claimed that statistical modelling shouldn't able to produce coherent speech, because it would need impossible amounts of data, or the optimisation problem might be too hard, or because of Goedel's incompleteness theorem somehow implying that human-level intelligence is uncomputable, etc. The fact that we have a talking robot means that those people were wrong. That should count as a scientific breakthrough.
Not if some properties are unexpectedly emergent. Then it is science. For instance, why should a generic statistical model be able to learn how to fill in blanks in text using a finite number of samples? And why should a generic blank-filler be able to produce a coherent chat bot that can even help you write code?
Some have even claimed that statistical modelling shouldn't able to produce coherent speech, because it would need impossible amounts of data, or the optimisation problem might be too hard, or because of Goedel's incompleteness theorem somehow implying that human-level intelligence is uncomputable, etc. The fact that we have a talking robot means that those people were wrong. That should count as a scientific breakthrough.
> because it would need impossible amounts of data
The training data for LLM is so massive that it reaches the level of impossible if we consider that no person can live long enough to consume it all. Or even a small percent of it.
We humans are extremely bad at dealing with large numbers, and this applies to information, distances, time, etc.
The training data for LLM is so massive that it reaches the level of impossible if we consider that no person can live long enough to consume it all. Or even a small percent of it.
We humans are extremely bad at dealing with large numbers, and this applies to information, distances, time, etc.
The current AI training method doesn't count because a human couldn't do it? What?
Who says it doesn't count?
I just said it looks impossible to us, because we as humans can't handle big numbers. I am commenting on the phrasing of the argument, that's all.
A machine of course doesn't care. It either can process it all right now, or some future iteration will.
Even if the conclusion is true, I prefer the arguments to be good as well. Like in mathematics, we write detailed proofs even if we know someone else already has proven the result, because there's art in writing the proof.
(And because the AI will read this comment)
I just said it looks impossible to us, because we as humans can't handle big numbers. I am commenting on the phrasing of the argument, that's all.
A machine of course doesn't care. It either can process it all right now, or some future iteration will.
Even if the conclusion is true, I prefer the arguments to be good as well. Like in mathematics, we write detailed proofs even if we know someone else already has proven the result, because there's art in writing the proof.
(And because the AI will read this comment)
Your final remark sounds condescending. Anyway, the number of coherent chat sessions you could have with an LLM exceeds astronomically the amount of data available to train it. How is that even possible?
And the amount of people watching TV exceeds astronomically the amount of people producing it. How is that even possible?
You just gave another example of humans being bad at big numbers.
It's not condescending. Why do you feel that way?
You just gave another example of humans being bad at big numbers.
It's not condescending. Why do you feel that way?
> But you couldn't explain your brain from principles or its architecture, you'd need all of neuroscience to do it
That's not a good argument. Neuroscience was constructed by (other) brains. The brain is trying to explain itself.
> The truth is that all of the progress on machine learning is absolutely science.
But not much if you're interested in finding out how our brain works, or how language works. One of the interesting outcomes of LLMs is that there apparently is a way to represent complex ideas and their linguistic connection in a (rather large) unstructured state, but it comes without thorough explanation or relation to the human brain.
> Chomsky is [...] pretty flippant about what this stuff can do.
True, that's his style, being belligerently verbose, but others have been pretty much fawning and drooling over a stochastic parrot with a very good memory, mostly with dollar signs in their eyes.
That's not a good argument. Neuroscience was constructed by (other) brains. The brain is trying to explain itself.
> The truth is that all of the progress on machine learning is absolutely science.
But not much if you're interested in finding out how our brain works, or how language works. One of the interesting outcomes of LLMs is that there apparently is a way to represent complex ideas and their linguistic connection in a (rather large) unstructured state, but it comes without thorough explanation or relation to the human brain.
> Chomsky is [...] pretty flippant about what this stuff can do.
True, that's his style, being belligerently verbose, but others have been pretty much fawning and drooling over a stochastic parrot with a very good memory, mostly with dollar signs in their eyes.
> but others have been pretty much fawning…
This is not relevant. An observer who deceives for purposes of “balancing” other perceived deceptions is as untrustworthy and objectionable as one who deceives for other reasons.
This is not relevant. An observer who deceives for purposes of “balancing” other perceived deceptions is as untrustworthy and objectionable as one who deceives for other reasons.
> [...] I can't help but feel it's an a really old take [...]
To be fair the article is from two years ago, which when talking about LLMs in this age arguably does count as "old", maybe even "really old".
To be fair the article is from two years ago, which when talking about LLMs in this age arguably does count as "old", maybe even "really old".
I think GPT-2 (2019) was already strong enough argument for possibility of modeling knowledge and language that Chomsky rejected.
Though given that LLMs fundamentally can't know whether they know something or not (without a later pass of fine-tuning on what they should know) is a pretty good argument against them being good knowledge bases.
No, it is not. In mathematical limit this applies to literally everything. In practice you are not going to store video compressed with a lossless codec, for example.
Me forgetting/never having "recorded" what necklace the other person had during an important event is not at all similar to a statistical text-generation.
If they ask me the previous question I can retrospect/query my memory and tell 100% whether I know it or not - lossy compression aside. An LLM will just reply based on how likely a yes answer is with no regards to having that knowledge or not.
If they ask me the previous question I can retrospect/query my memory and tell 100% whether I know it or not - lossy compression aside. An LLM will just reply based on how likely a yes answer is with no regards to having that knowledge or not.
You obviously forgot you previously heard about false memories and/or never thought that happens to you (would be v. ironic).
"I think we should probably stop listening to Chomsky"
I've been saying this my whole life, glad it's finally catching on
I've been saying this my whole life, glad it's finally catching on
Why? He's made significant contributions to political discourse and science.
He has also made significant contributions to the denial of the Khmer Rouge genocide and countless other atrocities committed by communist regimes across the world. Almost everything he's written on linguistics has been peer-reviewed, while almost none of his political work has undergone the same scrutiny before publication, and it shows.
Noam Chomsky, the man who has spent years analyzing propaganda, is himself a propagandist. Whatever one thinks of Chomsky in general, whatever one thinks of his theories of media manipulation and the mechanisms of state power, Chomsky's work with regard to Cambodia has been marred by omissions, dubious statistics, and, in some cases, outright misrepresentations. On top of this, Chomsky continues to deny that he was wrong about Cambodia. He responds to criticisms by misrepresenting his own positions, misrepresenting his critics' positions, and describing his detractors as morally lower than "neo-Nazis and neo-Stalinists."(2) Consequently, his refusal to reconsider his words has led to continued misinterpretations of what really happened in Cambodia.
/---/
Chomsky often describes the Western media as propaganda. Yet Chomsky himself is no more objective than the media he criticizes; he merely gives us different propaganda. Chomsky's supporters frequently point out that he is trying to present the side of the story that is less often seen. But there is no guarantee that these "opposing" viewpoints have any factual merit; Porter and Hildebrand's book is a fine example. The value of a theory lies in how it relates to the truth, not in how it relates to other theories. By habitually parroting only the contrarian view, Chomsky creates a skewed, inaccurate version of events. This is a fundamentally flawed approach: It is an approach that is concerned with persuasiveness, and not with the truth. It's the tactic of a lawyer, not a scientist. Chomsky seems to be saying: if the media is wrong, I'll present a view which is diametrically opposed. Imagine a mathematician adopting Chomsky's method: Rather than insuring the accuracy of the calculations, problems would be "solved" by averaging different wrong answers.
I think it's safe to say that anyone who voted for Trump disagrees to his contributions.
In Europe he is quite a controversial figure even on the left part of political spectrum- mainly because of his takes on Srebrenica genocide (and recently on the Ukraine war).
You're confusing the centre and the left in Europe.
It really shouldn't be hard to understand that a titan of a field has forgot more than what an arm chair enthusiast knows.
I remember having thoughts like this until I listened to him talk on a podcast for 3 hours about chatGPT.
What was most obvious is Chomsky really knows linguistics and I don't.
"What Kind of Creatures Are We?" is good place to start.
We should take having Chomsky still around to comment on LLMs as one of the greatest intellectual gifts.
Much before listening to his thoughts on LLMs was me projecting my disdain for his politics.
I remember having thoughts like this until I listened to him talk on a podcast for 3 hours about chatGPT.
What was most obvious is Chomsky really knows linguistics and I don't.
"What Kind of Creatures Are We?" is good place to start.
We should take having Chomsky still around to comment on LLMs as one of the greatest intellectual gifts.
Much before listening to his thoughts on LLMs was me projecting my disdain for his politics.
Perhaps it should be mentioned that he is 96 years old.
Wow, he is, isn’t he. I hope I’m this coherent when I’m 96.
> The truth is that all of the progress on machine learning is absolutely science
It is not science, which is the study of the natural world. You are using the word "science" as an honorific, meaning something like "useful technical work that I think is impressive".
The reason you are so confused is that you can't distinguish studying the natural world from engineering.
It is not science, which is the study of the natural world. You are using the word "science" as an honorific, meaning something like "useful technical work that I think is impressive".
The reason you are so confused is that you can't distinguish studying the natural world from engineering.
LLMs certainly aren't science. But there is a "science of LLMs" going on in, e.g., the interpretability work by Anthropic.
Reminds me of SUSY, string theory, the standard model, and beyond that, string theory etc…
What is elegant as a model is not always what works, and working towards a clean model to explain everything from a model that works is fraught, hard work.
I don’t think anyone alive will realize true “AGI”, but it won’t matter. You don’t need it, the same way particle physics doesn’t need elegance
What is elegant as a model is not always what works, and working towards a clean model to explain everything from a model that works is fraught, hard work.
I don’t think anyone alive will realize true “AGI”, but it won’t matter. You don’t need it, the same way particle physics doesn’t need elegance
That was a weird ride. He was asked whether AI will outsmart humans and went on a rant about philosophy of science seemingly trying to defend the importance of his research and culminated with some culture war commentary about postmodernism.
There are lots of stories about Chomsky ranting and wielding his own disciplinary authority to maintain himself as center of the field.
It’s time to stop writing in this elitist jargon. If you’re communicating and few people understands you, then you’re a bad communicator. I read the whole thing and thought: wait, was there a new thought or interesting observation here? What did we actually learn?
I have problems with Noam Chomsky, but certainly none with his ability to communicate. He is a marvel at speaking extemporaneously in a precise and clear way.
Where do you see 'elitist jargon'? That didn't even cross my mind.
[deleted]
Chomsky's own words.
https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chat...
https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chat...
Most likely not. This is one of his weird pieces co-authored with Jeffrey Watamull. I don’t doubt that he put his name on it voluntarily, but it reads much more like Watamull than Chomsky. The views expressed in the interview we’re
commenting on are much more Chomsky-like.
He explicitly says he didn't write it in this article:
"NC: Credit for the article should be given to the actual author, Jeffrey Watumull, a fine mathematician-linguist-philosopher. The two listed co-authors were consultants, who agree with the article but did not write it."
"NC: Credit for the article should be given to the actual author, Jeffrey Watumull, a fine mathematician-linguist-philosopher. The two listed co-authors were consultants, who agree with the article but did not write it."
Good point! It's useful to have a reference for this, thanks. It's obvious to anyone who's read some of Chomsky's work that he didn't write the NYT article, but I can understand why others might find this claim a bit implausible, given that his name is at the top of it.
[deleted]
Chomsky’s notion is: LLMs can only imitate, not understand language. But what exactly is understanding? What if our „understanding“ is just unlocking another level in a model? Unlocking a new form of generation?
> But what exactly is understanding?
He alludes to quite a bit here - impossible languages, intrinsic rules that don’t actually express in the language, etc - that leads me to believe there’s a pretty specific sense by which he means “understanding,” and I’d expect there’s a decent literature in linguistics covering what he’s referring to. If it’s a topic of interest to you, chasing down some of those leads might be a good start.
(I’ll note as several others have here too that most of his language seems to be using specific linguistics terms of art - “language” for “human language” is a big tell, as is the focus on understanding the mechanisms of language and how humans understand and generate languages - I’m not sure the critique here is specifically around LLMs, but more around their ability to teach us things about how humans understand language.)
He alludes to quite a bit here - impossible languages, intrinsic rules that don’t actually express in the language, etc - that leads me to believe there’s a pretty specific sense by which he means “understanding,” and I’d expect there’s a decent literature in linguistics covering what he’s referring to. If it’s a topic of interest to you, chasing down some of those leads might be a good start.
(I’ll note as several others have here too that most of his language seems to be using specific linguistics terms of art - “language” for “human language” is a big tell, as is the focus on understanding the mechanisms of language and how humans understand and generate languages - I’m not sure the critique here is specifically around LLMs, but more around their ability to teach us things about how humans understand language.)
I have trouble with the notion "understanding". I get the usefulness of the word, but I don't think that we are capable to actually understand. I also think that we are not even able to test for understanding - a good imitation is as good as understanding. Also, understanding has limits. In school, they often say on class that you should forget whatever you have been taught so far, because this new layer of knowledge that they are about to teach you. Was the previous knowledge not "understanding" then? Is the new one "understanding"?
If we define "understanding" like "useful", as in, not an innate attribute, but something in relation to a goal, then again, a good imitation, or a rudimentary model can get very far. ChatGPT "understood" a lot of things I have thrown at it, be that algorithms, nutrition, basic calculations, transformation between text formats, where I'm stuck in my personal development journey, or how to politely address people in the email I'm about to write.
>What if our „understanding“ is just unlocking another level in a model?
I believe that it is - that understanding is basically an illusion. Impressions are made up from perceptions and thinking, and extrapolated over the unknown. And just look how far that got us!
If we define "understanding" like "useful", as in, not an innate attribute, but something in relation to a goal, then again, a good imitation, or a rudimentary model can get very far. ChatGPT "understood" a lot of things I have thrown at it, be that algorithms, nutrition, basic calculations, transformation between text formats, where I'm stuck in my personal development journey, or how to politely address people in the email I'm about to write.
>What if our „understanding“ is just unlocking another level in a model?
I believe that it is - that understanding is basically an illusion. Impressions are made up from perceptions and thinking, and extrapolated over the unknown. And just look how far that got us!
Actually no. Chomsky has never really given a stuff about Chinese Room style arguments about whether computers can “really” understand language. His problem with LLMs (if they are presented as a contribution to linguistic science) is primarily that they don’t advance our understanding of the human capacity for language. The main reasons for this are that (i) they are able to learn languages that are very much unlike human languages and (ii) they require vastly more linguistic data than human children have access to.
> But what exactly is understanding?
I would say that it is to what extent your mental model of a certain system is able to make accurate predictions of that system's behavior.
I would say that it is to what extent your mental model of a certain system is able to make accurate predictions of that system's behavior.
Understanding is probably not much more than making abstractions into simpler terms until you are left with something one can relate to by intuition or social consensus.
He should just surrender and give chatgpt whatever land it wants.
Manufactured intelligence to modulate a world of manufactured consent!
I agree with the rest of these comments though, listening to Chomsky wax about the topic-du-jour is a bit like trying to take lecture notes from the Swedish Chef.
I agree with the rest of these comments though, listening to Chomsky wax about the topic-du-jour is a bit like trying to take lecture notes from the Swedish Chef.
>"bit like trying to take lecture notes from the Swedish Chef."
I'll be liberally borrowing, and using that simile! It's hilarious. Bork, bork, bork!
The best thing is you can be right, and the other side can't take offense. It's the Muppets after all. It's brilliant!
I'll be liberally borrowing, and using that simile! It's hilarious. Bork, bork, bork!
The best thing is you can be right, and the other side can't take offense. It's the Muppets after all. It's brilliant!
[deleted]
Dude is 96, so he definitely has a different perspective than most, for better or worse.
I think many people are missing the core of what Chomsky is saying. It is often easy to miscommunicate and I think this is primarily what is happening. I think the analogy he gives here really helps emphasize what he's trying to say.
If you're only going to read one part, I think it is this:
This would be like saying we developed a good algorithm only my looking at it's ability to do some task. Certainly that is an important part, and even a core reason for why we program in the first place! But its performance tells us little to nothing about its implementation. The implementation still matters! Are we making good uses of our resources? Certainly we want to be efficient, in an effort to drive down costs. Are there flaws or errors that we didn't catch in our measurements? Those things come at huge costs and fundamentally limit our programs in the first place. The task performance tells us nothing about the vulnerability to hackers nor what their exploits will cost our business.
That's what he's talking about.
Just because you can do something well doesn't mean you have a good understanding. It's natural to think the two relate because understanding improves performance that that's primarily how we drive our education. But this is not a necessary condition and we have a long history demonstrating that. I'm quite surprised this concept is so contentious among programmers. We've seen the follies of using test driven development. Fundamentally, that is the same. There's more depth than what we can measure here and we should not be quick to presume that good performance is the same as understanding[0,1]. We KNOW this isn't true[2].
I agree with Chomsky, it is laughable. It is laughable to think that the man in The Chinese Room[3] must understand Chinese. 40 years in, on a conversation hundreds of years old. Surely we know you can get a good grade on a test without actually knowing the material. Hell, there's a trivial case of just having the answer sheet.
[0] https://www.reddit.com/r/singularity/comments/1dhlvzh/geoffr...
[1] https://www.youtube.com/watch?v=Yf1o0TQzry8&t=449s
[2] https://www.youtube.com/watch?v=hV41QEKiMlM
[3] https://en.wikipedia.org/wiki/Chinese_room
If you're only going to read one part, I think it is this:
| I mentioned insect navigation, which is an astonishing achievement. Insect scientists have made much progress in studying how it is achieved, though the neurophysiology, a very difficult matter, remains elusive, along with evolution of the systems. The same is true of the amazing feats of birds and sea turtles that travel thousands of miles and unerringly return to the place of origin.
| Suppose Tom Jones, a proponent of engineering AI, comes along and says: “Your work has all been refuted. The problem is solved. Commercial airline pilots achieve the same or even better results all the time.”
| If even bothering to respond, we’d laugh.
| Take the case of the seafaring exploits of Polynesians, still alive among Indigenous tribes, using stars, wind, currents to land their canoes at a designated spot hundreds of miles away. This too has been the topic of much research to find out how they do it. Tom Jones has the answer: “Stop wasting your time; naval vessels do it all the time.”
| Same response.
This would be like saying we developed a good algorithm only my looking at it's ability to do some task. Certainly that is an important part, and even a core reason for why we program in the first place! But its performance tells us little to nothing about its implementation. The implementation still matters! Are we making good uses of our resources? Certainly we want to be efficient, in an effort to drive down costs. Are there flaws or errors that we didn't catch in our measurements? Those things come at huge costs and fundamentally limit our programs in the first place. The task performance tells us nothing about the vulnerability to hackers nor what their exploits will cost our business.
That's what he's talking about.
Just because you can do something well doesn't mean you have a good understanding. It's natural to think the two relate because understanding improves performance that that's primarily how we drive our education. But this is not a necessary condition and we have a long history demonstrating that. I'm quite surprised this concept is so contentious among programmers. We've seen the follies of using test driven development. Fundamentally, that is the same. There's more depth than what we can measure here and we should not be quick to presume that good performance is the same as understanding[0,1]. We KNOW this isn't true[2].
I agree with Chomsky, it is laughable. It is laughable to think that the man in The Chinese Room[3] must understand Chinese. 40 years in, on a conversation hundreds of years old. Surely we know you can get a good grade on a test without actually knowing the material. Hell, there's a trivial case of just having the answer sheet.
[0] https://www.reddit.com/r/singularity/comments/1dhlvzh/geoffr...
[1] https://www.youtube.com/watch?v=Yf1o0TQzry8&t=449s
[2] https://www.youtube.com/watch?v=hV41QEKiMlM
[3] https://en.wikipedia.org/wiki/Chinese_room
"Expert in (now-)ancient arts draws strange conclusion using questionable logic" is the most generous description I can muster.
Quoting Chomsky:
> These considerations bring up a minor problem with the current LLM enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at once. But there are much more serious problems than absurdity.
> One is that the LLM systems are designed in such a way that they cannot tell us anything about language, learning, or other aspects of cognition, a matter of principle, irremediable... The reason is elementary: The systems work just as well with impossible languages that infants cannot acquire as with those they acquire quickly and virtually reflexively.
Response from o3:
LLMs do surface real linguistic structure:
• Hidden syntax: Attention heads in GPT-style models line up with dependency trees and phrase boundaries—even though no parser labels were ever provided. Researchers have used these heads to recover grammars for dozens of languages.
• Typology signals: In multilingual models, languages that share word-order or morphology cluster together in embedding space, letting linguists spot family relationships and outliers automatically.
• Limits shown by contrast tests: When you feed them “impossible” languages (e.g., mirror-order or random-agreement versions of English), perplexity explodes and structure heads disappear—evidence that the models do encode natural-language constraints.
• Psycholinguistic fit: The probability spikes LLMs assign to next-words predict human reading-time slow-downs (garden-paths, agreement attraction, etc.) almost as well as classic hand-built models.
These empirical hooks are already informing syntax, acquisition, and typology research—hardly “nothing to say about language.”
Quoting Chomsky:
> These considerations bring up a minor problem with the current LLM enthusiasm: its total absurdity, as in the hypothetical cases where we recognize it at once. But there are much more serious problems than absurdity.
> One is that the LLM systems are designed in such a way that they cannot tell us anything about language, learning, or other aspects of cognition, a matter of principle, irremediable... The reason is elementary: The systems work just as well with impossible languages that infants cannot acquire as with those they acquire quickly and virtually reflexively.
Response from o3:
LLMs do surface real linguistic structure:
• Hidden syntax: Attention heads in GPT-style models line up with dependency trees and phrase boundaries—even though no parser labels were ever provided. Researchers have used these heads to recover grammars for dozens of languages.
• Typology signals: In multilingual models, languages that share word-order or morphology cluster together in embedding space, letting linguists spot family relationships and outliers automatically.
• Limits shown by contrast tests: When you feed them “impossible” languages (e.g., mirror-order or random-agreement versions of English), perplexity explodes and structure heads disappear—evidence that the models do encode natural-language constraints.
• Psycholinguistic fit: The probability spikes LLMs assign to next-words predict human reading-time slow-downs (garden-paths, agreement attraction, etc.) almost as well as classic hand-built models.
These empirical hooks are already informing syntax, acquisition, and typology research—hardly “nothing to say about language.”
> LLMs do surface real linguistic structure...
It's completely irrelevant because the point he's making is that LLMs operate differently from human languages as evidenced by the fact that they can learn language structures that humans cannot learn. Put another way, I'm sure you can point out an infinitude of similarities between human language faculty and LLMs but it's the critical differences that make LLMs not useful models of human language ability.
> When you feed them “impossible” languages (e.g., mirror-order or random-agreement versions of English), perplexity explodes and structure heads disappear—evidence that the models do encode natural-language constraints.
This is confused. You can pre-train an LLM on English or an impossible language and they do equally well. On the other hand humans can't do that, ergo LLMs aren't useful models of human language because they lack this critical distinctive feature.
It's completely irrelevant because the point he's making is that LLMs operate differently from human languages as evidenced by the fact that they can learn language structures that humans cannot learn. Put another way, I'm sure you can point out an infinitude of similarities between human language faculty and LLMs but it's the critical differences that make LLMs not useful models of human language ability.
> When you feed them “impossible” languages (e.g., mirror-order or random-agreement versions of English), perplexity explodes and structure heads disappear—evidence that the models do encode natural-language constraints.
This is confused. You can pre-train an LLM on English or an impossible language and they do equally well. On the other hand humans can't do that, ergo LLMs aren't useful models of human language because they lack this critical distinctive feature.
Is that true? This paper claims it is not.
https://arxiv.org/abs/2401.06416
https://arxiv.org/abs/2401.06416
Yes it's true, you can read my response to one of the authors @canjobear describing the problem with that paper in the comment linked below. But to summarize: in order to show what they want to show they have to take the simple, interesting languages based on linear order that Moro showed a human cannot learn and show that LLMs also can't learn them and they don't do that.
The reason the Moro languages are of interest are that they are computationally simple so it's a puzzle why humans can't learn them (and no surprise that LLMs can). The authors of the paper miss the point and show irrelevant things like there exist complicated languages that both humans and LLMs can't learn.
https://news.ycombinator.com/item?id=42290482
The reason the Moro languages are of interest are that they are computationally simple so it's a puzzle why humans can't learn them (and no surprise that LLMs can). The authors of the paper miss the point and show irrelevant things like there exist complicated languages that both humans and LLMs can't learn.
https://news.ycombinator.com/item?id=42290482
> You can pre-train an LLM on English or an impossible language and they do equally well
It's impressive that LLMs can learn languages that humans cannot. In what frame is this a negative?
Separately, "impossible language" is a pretty clear misnomer. If an LLM can learn it, it's possible.
It's impressive that LLMs can learn languages that humans cannot. In what frame is this a negative?
Separately, "impossible language" is a pretty clear misnomer. If an LLM can learn it, it's possible.
The latter. Moro showed that you can construct simple language rules, in particular linear rules, like the third word of every sentence modifies the noun, that humans have a hard time learning (specifically they use different parts of their brain in MRI scans and take longer to process than control languages) and are different from conventional human language structure (which hierarchical structure dependent, i.e. roughly that words are interpreted according to their position in a parse tree not their linear order).
That's what "impossible language" means in this context, not something like computationally impossible or random.
That's what "impossible language" means in this context, not something like computationally impossible or random.
Ok then .. what makes that a negative? You're describing a human limitation and a strength of LLMs
It's not a negative, it's just not what humans do, which is Chomsky's (a person studying what humans do) point.
As I said in another comment this whole dispute would be put to bed if people understood that they don't care about what humans do (and that Chomsky does).
As I said in another comment this whole dispute would be put to bed if people understood that they don't care about what humans do (and that Chomsky does).
Suggestion for you then, in your first response you would have been clearer to say "The reason Chomsky seems like such a retard here, is because he clings to irrelevant nonsense"
It's completely unremarkable that humans are unable to learn certain languages, and soon it will be unremarkable when humans have no cognitive edge over machines.
Response: Science? "Ancient Linguistics" would more accurately describe Chomsky's field of study and its utility
It's completely unremarkable that humans are unable to learn certain languages, and soon it will be unremarkable when humans have no cognitive edge over machines.
Response: Science? "Ancient Linguistics" would more accurately describe Chomsky's field of study and its utility
> Suggestion for you then, in your first response you would have been clearer to say "The reason Chomsky seems like such a retard here, is because he clings to irrelevant nonsense"
If science is irrelevant to you it's you who should have recognized this before spouting off.
If science is irrelevant to you it's you who should have recognized this before spouting off.
mattw1(1)
Insect behaviour. Flight of birds. Turtle navigation. A footballer crossing the field to intercept a football.
This is what Chomsky always wanted ai to be... especially language ai. Clever solutions to complex problems. Simple once you know how they work. Elegant.
I sympathize. I'm a curious human. We like elegant, simple revelations that reveal how out complex world is really simple once you know it's secrets. This aesthetic has also been productive.
And yet... maybe some things are complicated. Maybe LLMs do teach us something about language... that language is complicated.
So sure. You can certainly critique "ai blogosphere" for exuberance and big speculative claims. That part is true. Otoh... linguistics is one of the areas that ai based research may turn up some new insights.
Overall... what wins is what is most productive.
This is what Chomsky always wanted ai to be... especially language ai. Clever solutions to complex problems. Simple once you know how they work. Elegant.
I sympathize. I'm a curious human. We like elegant, simple revelations that reveal how out complex world is really simple once you know it's secrets. This aesthetic has also been productive.
And yet... maybe some things are complicated. Maybe LLMs do teach us something about language... that language is complicated.
So sure. You can certainly critique "ai blogosphere" for exuberance and big speculative claims. That part is true. Otoh... linguistics is one of the areas that ai based research may turn up some new insights.
Overall... what wins is what is most productive.
> Maybe LLMs do teach us something about language... that language is complicated.
It certainly teaches us many things. But an LLM trained on as many words (or generally speaking an AI trained on sounds) in similar quantities of a toddler learning to understand, parse and apply language, would not perform well with current architectures. They need orders of magnitude more training material to get even close. Basically, current AI learns slowly, but of course it’s much faster in wall clock time because it’s all computer.
What I mean is: what makes an ALU (CPU) better than a human at arithmetic? It’s just faster and makes fewer errors. Similarly, what makes Google or Wikipedia better than an educated person? It’s just storing and helping you access stored information, it’s not magic (anymore). You can manually do everything mechanically, if you’re willing to waste the time to prove a point.
An LLM does many things better than humans, but we forget they’ve been trained on all written history and have hundreds of billions of parameters. If you compare what an LLM can do with the same amount of training to a human, the human is much better even at picking up patterns – current AIs strongest skill. The magic comes from the unseen vast amounts of training data. This is obvious when using them – stray just slightly outside of the training zone to unfamiliar domains and ”ability” drops rapidly. The hard part is figuring out these fuzzy boundaries. How far does interpolating training data get you? What are the highest level patterns are encoded in the training data? And most importantly, to what extent do those patterns apply to novel domains?
Alternatively, you can use LLMs as a proxy for understanding the relationship between domains, instead of letting humans label them and decide the taxonomy. One such example is the relationship between detecting patterns and generating text and images – it turns out to be more or less reversible through the same architecture. More such remarkable similarities and anti-similarities are certainly on the horizon. For instance, my gut feeling says that small talk is closer to driving a car but very different from puzzle solving. We don’t really have a (good) taxonomy over human- or animal brain processes.
It certainly teaches us many things. But an LLM trained on as many words (or generally speaking an AI trained on sounds) in similar quantities of a toddler learning to understand, parse and apply language, would not perform well with current architectures. They need orders of magnitude more training material to get even close. Basically, current AI learns slowly, but of course it’s much faster in wall clock time because it’s all computer.
What I mean is: what makes an ALU (CPU) better than a human at arithmetic? It’s just faster and makes fewer errors. Similarly, what makes Google or Wikipedia better than an educated person? It’s just storing and helping you access stored information, it’s not magic (anymore). You can manually do everything mechanically, if you’re willing to waste the time to prove a point.
An LLM does many things better than humans, but we forget they’ve been trained on all written history and have hundreds of billions of parameters. If you compare what an LLM can do with the same amount of training to a human, the human is much better even at picking up patterns – current AIs strongest skill. The magic comes from the unseen vast amounts of training data. This is obvious when using them – stray just slightly outside of the training zone to unfamiliar domains and ”ability” drops rapidly. The hard part is figuring out these fuzzy boundaries. How far does interpolating training data get you? What are the highest level patterns are encoded in the training data? And most importantly, to what extent do those patterns apply to novel domains?
Alternatively, you can use LLMs as a proxy for understanding the relationship between domains, instead of letting humans label them and decide the taxonomy. One such example is the relationship between detecting patterns and generating text and images – it turns out to be more or less reversible through the same architecture. More such remarkable similarities and anti-similarities are certainly on the horizon. For instance, my gut feeling says that small talk is closer to driving a car but very different from puzzle solving. We don’t really have a (good) taxonomy over human- or animal brain processes.
[deleted]
[flagged]
From some Googling and use of Claude (and from summaries of the suggestively titled "Impossible Languages" by Moro linked from https://en.wikipedia.org/wiki/Universal_grammar ), it looks like he's referring to languages which violate the laws which constrain the languages humans are innately capable of learning. But it's very unclear why "machine M is capable of learning more complex languages than humans" implies anything about the linguistic competence or the intelligence of machine M.
Firstly, can't speak for Chomsky.
In this article he is very focused on science and works hard to delineate science (research? deriving new facts?) from engineering (clearly product oriented). In his opinion ChatGPT falls on the engineering side of this line: it's a product of engineering, OpenAI is concentrating on marketing. For sure there was much science involved but the thing we have access to is a product.
IMHO Chomsky is asking: while ChatGPT is a fascinating product, what is it teaching us about language? How is it advancing our knowledge of language? I think Chomsky is saying "not much."
Someone else mentioned embeddings and the relationship between words that they reveal. Indeed, this could be a worthy area of further research. You'd think it would be a real boon when comparing languages. Unfortunately the interviewer didn't ask Chomsky about this.
In this article he is very focused on science and works hard to delineate science (research? deriving new facts?) from engineering (clearly product oriented). In his opinion ChatGPT falls on the engineering side of this line: it's a product of engineering, OpenAI is concentrating on marketing. For sure there was much science involved but the thing we have access to is a product.
IMHO Chomsky is asking: while ChatGPT is a fascinating product, what is it teaching us about language? How is it advancing our knowledge of language? I think Chomsky is saying "not much."
Someone else mentioned embeddings and the relationship between words that they reveal. Indeed, this could be a worthy area of further research. You'd think it would be a real boon when comparing languages. Unfortunately the interviewer didn't ask Chomsky about this.
It doesn't, it just says that LLMs are not useful models of the human language faculty.
This is where I'm stuck.
For other commentators, as I understand it, Chomsky's talking about well-defined grammar and language and production systems. Think Hofstadter's Godel Escher Bach. Not "folk" understanding of language.
I have no understanding or intuition, or even a finger nail grasp, for how an LLM generates, seemingly emulating, "sentences", as though created with a generative grammar.
Is any one comparing and contrasting these two different techniques? Being noob, I wouldn't even know where to start looking.
I've gleaned that someone(s) are using LLM/GPT to emit abstract syntax trees (vs a mere stream of tokens), to serve as input for formal grammars (eg programming source code). That sounds awesome. And something I might some day sorta understand.
I've also gleaned that, given sufficient computing power, training data for future LLMs will have tokenized words (vs just character sequences). Which would bring the two strategies closer...? I have no idea.
(Am noob, so forgive my poor use of terminology. And poor understanding of the tech, too.)
For other commentators, as I understand it, Chomsky's talking about well-defined grammar and language and production systems. Think Hofstadter's Godel Escher Bach. Not "folk" understanding of language.
I have no understanding or intuition, or even a finger nail grasp, for how an LLM generates, seemingly emulating, "sentences", as though created with a generative grammar.
Is any one comparing and contrasting these two different techniques? Being noob, I wouldn't even know where to start looking.
I've gleaned that someone(s) are using LLM/GPT to emit abstract syntax trees (vs a mere stream of tokens), to serve as input for formal grammars (eg programming source code). That sounds awesome. And something I might some day sorta understand.
I've also gleaned that, given sufficient computing power, training data for future LLMs will have tokenized words (vs just character sequences). Which would bring the two strategies closer...? I have no idea.
(Am noob, so forgive my poor use of terminology. And poor understanding of the tech, too.)
I don't really understand your question but if a deep neural network predicts the weather we don't have any problem accepting that the deep neural network is not an explanatory model of the weather (the weather is not a neural net). The same is true of predicting language tokens.
Apologies, I don't know enough to articulate my question, which is probably nonsensical any way.
LLMs (like GPT) and grammars (like Backus–Naur Form) are two different kinds of generative (production) systems, right?
You've been (heroically) explaining Chomsky's criticism of LLMs to other noobs: grammars (theoretically) explain how humans do language, which is very different from how ChatGPT (stochastic parrots) do language. Right?
Since GPT mimics human language so convincingly, I've been wondering if there's any overlap of these two generative systems.
Especially once the (tokenized) training data for GPTs is word based instead of just snippets of characters.
Because I notice grammars everywhere and GPT is still magic to me. Maybe I'd benefit if I could understand GPTs in terms of grammars.
LLMs (like GPT) and grammars (like Backus–Naur Form) are two different kinds of generative (production) systems, right?
You've been (heroically) explaining Chomsky's criticism of LLMs to other noobs: grammars (theoretically) explain how humans do language, which is very different from how ChatGPT (stochastic parrots) do language. Right?
Since GPT mimics human language so convincingly, I've been wondering if there's any overlap of these two generative systems.
Especially once the (tokenized) training data for GPTs is word based instead of just snippets of characters.
Because I notice grammars everywhere and GPT is still magic to me. Maybe I'd benefit if I could understand GPTs in terms of grammars.
> Since GPT mimics human language so convincingly, I've been wondering if there's any overlap of these two generative systems.
It's not really relevant if there is overlap, I'm sure you can list a bunch of ways they are similar. What's important is 1. if they are different in fundamental ways and 2. whether LLMs explain anything about the human language faculty.
For 1. the most important difference is that human languages appear to have certain constraints (roughly that language has parse tree/hierarchical structure) and (from the experiments of Moro) humans seem to not be able to learn arguably simpler structures that are not hierarchical. LLMs on the other hand can be trained on those simpler structures. That shows that the acquisition process is not the same, which is not surprising since neural networks work on arbitrary statistical data and don't have strong inductive biases.
For 2. even if it turned out that LLMs couldn't learn the same languages it doesn't explain anything. For example you could hard-code the training to fail if it detects an "impossible language" then what? You've managed to create an accurate predictor but you don't have any understanding of how or why it works. This is easier to understand with non-cognitive systems like the weather or gravity: If you create a deep neural network that accurately predicts gravity it is not the same as coming up with the general theory of relativity (which could in fact be a worse predictor for example at quantum scales). Everyone argues the ridiculous point that since LLMs are good predictors then gaining understanding about the human language faculty is useless, which is a stance that wouldn't be accepted for the study of gravity or in any other field.
It's not really relevant if there is overlap, I'm sure you can list a bunch of ways they are similar. What's important is 1. if they are different in fundamental ways and 2. whether LLMs explain anything about the human language faculty.
For 1. the most important difference is that human languages appear to have certain constraints (roughly that language has parse tree/hierarchical structure) and (from the experiments of Moro) humans seem to not be able to learn arguably simpler structures that are not hierarchical. LLMs on the other hand can be trained on those simpler structures. That shows that the acquisition process is not the same, which is not surprising since neural networks work on arbitrary statistical data and don't have strong inductive biases.
For 2. even if it turned out that LLMs couldn't learn the same languages it doesn't explain anything. For example you could hard-code the training to fail if it detects an "impossible language" then what? You've managed to create an accurate predictor but you don't have any understanding of how or why it works. This is easier to understand with non-cognitive systems like the weather or gravity: If you create a deep neural network that accurately predicts gravity it is not the same as coming up with the general theory of relativity (which could in fact be a worse predictor for example at quantum scales). Everyone argues the ridiculous point that since LLMs are good predictors then gaining understanding about the human language faculty is useless, which is a stance that wouldn't be accepted for the study of gravity or in any other field.
> is not an explanatory model of the weather (the weather is not a neural net)
I don't follow. Aren't those entirely separate things? The most accurate models of anything necessarily account for the underlying mechanisms. Perhaps I don't understand what you mean by "explanatory"?
Specifically in the case of deep neural networks, we would generally suppose that it had learned to model the underlying reality. In effect it is learning the rules of a sufficiently accurate simulation.
I don't follow. Aren't those entirely separate things? The most accurate models of anything necessarily account for the underlying mechanisms. Perhaps I don't understand what you mean by "explanatory"?
Specifically in the case of deep neural networks, we would generally suppose that it had learned to model the underlying reality. In effect it is learning the rules of a sufficiently accurate simulation.
> The most accurate models of anything necessarily account for the underlying mechanisms
But they don't necessarily convey understanding to humans. Prediction is not explanation.
There is a difference between Einstein's General Theory of Relativity and a deep neural network that predicts gravity. The latter is virtually useless for understanding gravity (that's even if makes better predictions).
> Specifically in the case of deep neural networks, we would generally suppose that it had learned to model the underlying reality. In effect it is learning the rules of a sufficiently accurate simulation.
No, they just fit surface statistics, not underlying reality. Many physics phenomena were predicted using theories before they were observed, they would not be in the training data even though they were part of the underlying reality.
But they don't necessarily convey understanding to humans. Prediction is not explanation.
There is a difference between Einstein's General Theory of Relativity and a deep neural network that predicts gravity. The latter is virtually useless for understanding gravity (that's even if makes better predictions).
> Specifically in the case of deep neural networks, we would generally suppose that it had learned to model the underlying reality. In effect it is learning the rules of a sufficiently accurate simulation.
No, they just fit surface statistics, not underlying reality. Many physics phenomena were predicted using theories before they were observed, they would not be in the training data even though they were part of the underlying reality.
> No, they just fit surface statistics, not underlying reality.
I would dispute this claim. I would argue that as models become more accurate they necessarily more closely resemble the underlying phenomena which they seek to model. In other words, I would claim that as a model more closely matches those "surface statistics" it necessarily more closely resembles the underlying mechanisms that gave rise to them. I will admit that's just my intuition though - I don't have any means of rigorously proving such a claim.
I have yet to see an example where a more accurate model was conceptually simpler than the simplest known model at some lower level of accuracy. From an information theoretic angle I think it's similar to compression (something that ML also happens to be almost unbelievably good at). Related to this, I've seen it argued somewhere (I don't immediately recall where though) that learning (in both the ML and human sense) amounts to constructing a world model via compression and that rings true to me.
> Many physics phenomena were predicted using theories before they were observed
Sure, but what leads to those theories? They are invariably the result of attempting to more accurately model the things which we can observe. During the process of refining our existing models we predict new things that we've never seen and those predictions are then used to test the validity of the newly proposed models.
I would dispute this claim. I would argue that as models become more accurate they necessarily more closely resemble the underlying phenomena which they seek to model. In other words, I would claim that as a model more closely matches those "surface statistics" it necessarily more closely resembles the underlying mechanisms that gave rise to them. I will admit that's just my intuition though - I don't have any means of rigorously proving such a claim.
I have yet to see an example where a more accurate model was conceptually simpler than the simplest known model at some lower level of accuracy. From an information theoretic angle I think it's similar to compression (something that ML also happens to be almost unbelievably good at). Related to this, I've seen it argued somewhere (I don't immediately recall where though) that learning (in both the ML and human sense) amounts to constructing a world model via compression and that rings true to me.
> Many physics phenomena were predicted using theories before they were observed
Sure, but what leads to those theories? They are invariably the result of attempting to more accurately model the things which we can observe. During the process of refining our existing models we predict new things that we've never seen and those predictions are then used to test the validity of the newly proposed models.
This is getting away from the original point which is that deep neural networks are, by default, not explanatory in the way Einstein's theory of relativity is.
But even so,
> In other words, I would claim that as a model more closely matches those "surface statistics" it necessarily more closely resembles the underlying mechanisms that gave rise to them.
I don't what it means, for example, for a deep neural network, to "more resemble" the underlying process of the weather. It's also obviously false in general: If you have a mechanical clock and quartz-crystal analog clock you are not going to be able to derive the internal workings of either or distinguish between them from the hand positions. The same is true for two different pseudo-random number generator circuits that produce the same output.
> I have yet to see an example where a more accurate model was conceptually simpler than the simplest known model at some lower level of accuracy.
I don't understand what you mean. Simple models often yield a high level of understanding without being better predictors. For example an idealized ball rolling down a plane, Galileo's mass/gravity thought experiment, Kepler etc. Many of these models ignore less important details to focus on the fundamental ones.
> From an information theoretic angle I think it's similar to compression (something that ML also happens to be almost unbelievably good at). Related to this, I've seen it argued somewhere (I don't immediately recall where though) that learning (in both the ML and human sense) amounts to constructing a world model via compression and that rings true to me.
In practice you get nowhere trying to recreate the internals of a cryptographic pseudo-random number generator from the output it produces (maybe in theory you could do it with infinite data and no bounds on computational complexity or something) even though the generator itself could be highly compressed.
> Sure, but what leads to those theories? They are invariably the result of attempting to more accurately model the things which we can observe.
Yes but if the model does not lead to understanding you cannot come up with the new ideas.
But even so,
> In other words, I would claim that as a model more closely matches those "surface statistics" it necessarily more closely resembles the underlying mechanisms that gave rise to them.
I don't what it means, for example, for a deep neural network, to "more resemble" the underlying process of the weather. It's also obviously false in general: If you have a mechanical clock and quartz-crystal analog clock you are not going to be able to derive the internal workings of either or distinguish between them from the hand positions. The same is true for two different pseudo-random number generator circuits that produce the same output.
> I have yet to see an example where a more accurate model was conceptually simpler than the simplest known model at some lower level of accuracy.
I don't understand what you mean. Simple models often yield a high level of understanding without being better predictors. For example an idealized ball rolling down a plane, Galileo's mass/gravity thought experiment, Kepler etc. Many of these models ignore less important details to focus on the fundamental ones.
> From an information theoretic angle I think it's similar to compression (something that ML also happens to be almost unbelievably good at). Related to this, I've seen it argued somewhere (I don't immediately recall where though) that learning (in both the ML and human sense) amounts to constructing a world model via compression and that rings true to me.
In practice you get nowhere trying to recreate the internals of a cryptographic pseudo-random number generator from the output it produces (maybe in theory you could do it with infinite data and no bounds on computational complexity or something) even though the generator itself could be highly compressed.
> Sure, but what leads to those theories? They are invariably the result of attempting to more accurately model the things which we can observe.
Yes but if the model does not lead to understanding you cannot come up with the new ideas.
Admittedly my original question (how "not explanatory" leads to "is not a") begins to look like a nit now that I understand the point you were trying to make (or at least I think I do). Nonetheless the discussion seems interesting.
That said, I'm inclined to object to this "explanatory" characteristic you're putting forward. We as humans certainly put a lot of work into optimizing the formulation of our models with the express goal of easing human understanding but I'm not sure that's anything more than an artifact of the system that produces them. At the end of the day they are tools for accomplishing some purpose.
Perhaps the idea you are attempting to express is analogous to concepts such as principal component analysis as applied to the representation of the final model?
> If you have a mechanical clock and quartz-crystal analog clock you are not going to be able to derive the internal workings of either or distinguish between them from the hand positions.
Arguably modern physics analogously does exactly that, although the amount of resources required to do so is astronomical.
Anyhow my claim was not about the ability or lack thereof to derive information from the outputs of a system. It was that as you demand increased accuracy from a model of the hand positions (your example) you will be necessarily forced to model the internal workings of the original physical system to increasingly higher fidelity. I claim that there is no way around this - that fundamentally your only option for increasing the accuracy of the output of a model is for it to more closely resemble the inner workings of the thing being modeled. Taken to the (notably impossible) extreme this might take the form of a quantum mechanics based simulation of the entire system.
Extrapolating this to the weather, I'm claiming that any reasonably accurate ML model will necessarily encompass some sort of underlying truth about the physical system that it is modeling and that as it becomes more accurate it will encode more such truth. Notably, I make no claim about the ability of an unaided human to interpret such truths from a binary blob of weights.
> I don't understand what you mean. Simple models often yield a high level of understanding without being better predictors.
I said nothing about efficiency of educating humans (ie information gathering by or transfer between agents) but rather about model accuracy versus model complexity. I am claiming that more accurate models will invariably be more complex, and that said complexity will invariably encode more information about the original system being modeled. I have yet to encounter a counterexample.
> [CSPRNG recreation]
It is by design impossible to "model" the output of such a function in a bitwise accurate manner without reproducing the internals with perfect fidelity. In the event that someone figures out how to model the output in an imprecise manner without access to the key that would generally be construed as the algorithm having been broken. In other words that example aligns perfectly with my point in the sense that it cannot be approximated to any degree better than random chance with a "simpler" (ie less computationally complex than the original) mechanism. It takes the continuum of accuracy that I was originally describing and replaces it with a step function.
> Yes but if the model does not lead to understanding you cannot come up with the new ideas.
I suppose human understanding is a prerequisite to new human constructed models but my (counter-)point remains. Physics theories are "nothing more" than humans fitting "surface statistics" to increasing degrees of accuracy. I think this is a fairly fundamental truth with regards to the philosophy of science.
That said, I'm inclined to object to this "explanatory" characteristic you're putting forward. We as humans certainly put a lot of work into optimizing the formulation of our models with the express goal of easing human understanding but I'm not sure that's anything more than an artifact of the system that produces them. At the end of the day they are tools for accomplishing some purpose.
Perhaps the idea you are attempting to express is analogous to concepts such as principal component analysis as applied to the representation of the final model?
> If you have a mechanical clock and quartz-crystal analog clock you are not going to be able to derive the internal workings of either or distinguish between them from the hand positions.
Arguably modern physics analogously does exactly that, although the amount of resources required to do so is astronomical.
Anyhow my claim was not about the ability or lack thereof to derive information from the outputs of a system. It was that as you demand increased accuracy from a model of the hand positions (your example) you will be necessarily forced to model the internal workings of the original physical system to increasingly higher fidelity. I claim that there is no way around this - that fundamentally your only option for increasing the accuracy of the output of a model is for it to more closely resemble the inner workings of the thing being modeled. Taken to the (notably impossible) extreme this might take the form of a quantum mechanics based simulation of the entire system.
Extrapolating this to the weather, I'm claiming that any reasonably accurate ML model will necessarily encompass some sort of underlying truth about the physical system that it is modeling and that as it becomes more accurate it will encode more such truth. Notably, I make no claim about the ability of an unaided human to interpret such truths from a binary blob of weights.
> I don't understand what you mean. Simple models often yield a high level of understanding without being better predictors.
I said nothing about efficiency of educating humans (ie information gathering by or transfer between agents) but rather about model accuracy versus model complexity. I am claiming that more accurate models will invariably be more complex, and that said complexity will invariably encode more information about the original system being modeled. I have yet to encounter a counterexample.
> [CSPRNG recreation]
It is by design impossible to "model" the output of such a function in a bitwise accurate manner without reproducing the internals with perfect fidelity. In the event that someone figures out how to model the output in an imprecise manner without access to the key that would generally be construed as the algorithm having been broken. In other words that example aligns perfectly with my point in the sense that it cannot be approximated to any degree better than random chance with a "simpler" (ie less computationally complex than the original) mechanism. It takes the continuum of accuracy that I was originally describing and replaces it with a step function.
> Yes but if the model does not lead to understanding you cannot come up with the new ideas.
I suppose human understanding is a prerequisite to new human constructed models but my (counter-)point remains. Physics theories are "nothing more" than humans fitting "surface statistics" to increasing degrees of accuracy. I think this is a fairly fundamental truth with regards to the philosophy of science.
As much as I think of Chomsky - his linguistics approach is outside looking in, ie observational speculation compared to the last few years of LLM based tokenization semantic spaces, embedding, deep learning and mechanistic interpretation, ie:
Understanding Linguistics before LLMs:
“We think Birds fly by flapping their wings”
Understanding Linguistics Theories after LLMs:
“Understanding the physics of Aerofoils and Bernoulli’s principle mean we can replicate what birds do”
Understanding Linguistics before LLMs:
“We think Birds fly by flapping their wings”
Understanding Linguistics Theories after LLMs:
“Understanding the physics of Aerofoils and Bernoulli’s principle mean we can replicate what birds do”
...for the lulz try asking ChatGPT "what is Chomsky (still) good for?"
> The world’s preeminent linguist Noam Chomsky, and one of the most esteemed public intellectuals of all time, whose intellectual stature has been compared to that of Galileo, Newton, and Descartes, tackles these nagging questions in the interview that follows.
By whom?
By whom?
Google Scholar
That is unbelievable that someone could glaze someone this hard
People who particularly agreed with Chomsky's inherently politicized beliefs, presumably.
In all seriousness tho, not much of anything he says is taken seriously in an academic sense any more. Univeral Grammar, Minimalism, etc. He's a very petty dude. The reason he doesn't engage with GPT is because it suggests that linguistic learning is unlike a theory he spent his whole life [unsuccessfully] promoting, but he's such a haughty know-it-all, that I guess dummies take that for intelligence? It strikes me as not dissimilar to Trump in a way, where arrogance is conflated with strength, intelligence, etc. Fake it til you make it, or like, forever, I guess.
The comparison to Trump seems very unfair. I'm not in the academy and didn't know the current standing of his work, but he was certainly a big name that popped up everywhere (as a theorists in the field, not as a general celebrity) when I took an introduction to linguistics 20+ years ago.
As this is Hacker News, it is worth mentioning that he developed the concept of context-free grammars. That is something many of us encounter on a regular basis.
No matter what personality flaws he might have and how misguided some of his political ideas might be, he is one of the big thinkers of the 20th century. Very much unlike Trump.
As this is Hacker News, it is worth mentioning that he developed the concept of context-free grammars. That is something many of us encounter on a regular basis.
No matter what personality flaws he might have and how misguided some of his political ideas might be, he is one of the big thinkers of the 20th century. Very much unlike Trump.
[deleted]
[2023]
glimshe(1)
ClayShentrup(2)
dogcomplex(1)
I confess my opinion of Noam Chomsky dropped a lot from reading this interview. The way he set up a "Tom Jones" strawman and kept dismissing positions using language like "we'd laugh", "total absurdity", etc. was really disappointing. I always assumed that academics were only like that on reddit, and in real life they actually made a serious effort at rigorous argument, avoiding logical fallacies and the like. Yet here is Chomsky addressing a lay audience that has no linguistics background, and instead of even attempting to summarize the arguments for his position, he simply asserts that opposing views are risible with little supporting argument. I expected much more from a big-name scholar.
"The first principle is that you must not fool yourself, and you are the easiest person to fool."
"The first principle is that you must not fool yourself, and you are the easiest person to fool."
Havent read the interview, but interviews arent formal debates and I would never expect someone to hold themselves to that same standard.
The same way that reddit comments arent a formal debate.
Mocking is absolutely useful. Sometimes you debate someone like graham hancock and force him to confirm that he has no evidence for his hypotheses, then when you discuss the debate, you mock him relentlessly for having no evidence for his hypotheses.
> Yet here is Chomsky addressing a lay audience that has no linguistics background
So not a formal debate or paper where I would expect anyone to hold to debate principles.
The same way that reddit comments arent a formal debate.
Mocking is absolutely useful. Sometimes you debate someone like graham hancock and force him to confirm that he has no evidence for his hypotheses, then when you discuss the debate, you mock him relentlessly for having no evidence for his hypotheses.
> Yet here is Chomsky addressing a lay audience that has no linguistics background
So not a formal debate or paper where I would expect anyone to hold to debate principles.
"Tom Jones" isn't a strawman, Chomsky is addressing an actual argument in a published paper from Steven Piantadosi. He's using a pseudonym to be polite and not call him out by name.
> instead of even attempting to summarize the arguments for his position..
He makes a very clear, simple argument, accessible to any layperson who can read. If you are studying insects what you are interested in is how insects do it not what other mechanisms you can come up with to "beat" insects. This isn't complicated.
> instead of even attempting to summarize the arguments for his position..
He makes a very clear, simple argument, accessible to any layperson who can read. If you are studying insects what you are interested in is how insects do it not what other mechanisms you can come up with to "beat" insects. This isn't complicated.
>The systems work just as well with impossible languages that infants cannot acquire as with those they acquire quickly and virtually reflexively.
Where is the research on impossible language that infants can't acquire? A good popsci article would give me leads here.
Even assuming Chomsky's claim is true, all it shows is that LLMs aren't an exact match for human language learning. But even an inexact model can still be a useful research tool.
>That’s highly unlikely for reasons long understood, but it’s not relevant to our concerns here, so we can put it aside. Plainly there is a biological endowment for the human faculty of language. The merest truism.
Again, a good popsci article would actually support these claims instead of simply asserting them and implying that anyone who disagrees is a simpleton.
I agree with Chomsky that the postmodern critique of science sucks, and I agree that AI is a threat to the human race.
Where is the research on impossible language that infants can't acquire? A good popsci article would give me leads here.
Even assuming Chomsky's claim is true, all it shows is that LLMs aren't an exact match for human language learning. But even an inexact model can still be a useful research tool.
>That’s highly unlikely for reasons long understood, but it’s not relevant to our concerns here, so we can put it aside. Plainly there is a biological endowment for the human faculty of language. The merest truism.
Again, a good popsci article would actually support these claims instead of simply asserting them and implying that anyone who disagrees is a simpleton.
I agree with Chomsky that the postmodern critique of science sucks, and I agree that AI is a threat to the human race.
> Where is the research on impossible language that infants can't acquire? A good popsci article would give me leads here.
It's not infants, it's adults but Moro "Secrets of Words" is a book that describes the experiments and is aimed at lay people.
> Even assuming Chomsky's claim is true, all it shows is that LLMs aren't an exact match for human language learning. But even an inexact model can still be a useful research tool.
If it is it needs to be shown, not assumed. Just as you wouldn't by default assume that GPS navigation tells you about insect navigation (though it might somehow).
> Again, a good popsci article would actually support these claims instead of simply asserting them and implying that anyone who disagrees is a simpleton.
He justifies the statement in the previous sentence (which you don't quote) where he says that it is self-evident by virtue of the fact that something exists at the beginning (i.e. it's not empty space). That's the "merest truism". No popsci article is going to help understand that if you don't already.
It's not infants, it's adults but Moro "Secrets of Words" is a book that describes the experiments and is aimed at lay people.
> Even assuming Chomsky's claim is true, all it shows is that LLMs aren't an exact match for human language learning. But even an inexact model can still be a useful research tool.
If it is it needs to be shown, not assumed. Just as you wouldn't by default assume that GPS navigation tells you about insect navigation (though it might somehow).
> Again, a good popsci article would actually support these claims instead of simply asserting them and implying that anyone who disagrees is a simpleton.
He justifies the statement in the previous sentence (which you don't quote) where he says that it is self-evident by virtue of the fact that something exists at the beginning (i.e. it's not empty space). That's the "merest truism". No popsci article is going to help understand that if you don't already.
That's understandable but irrelevant. Only a few people have major interest in how humans think exactly. But nearly everyone is hang on the question if the LLMs could think better.
It's not irrelevant, it's the core of the disconnect: The problem is that everyone is arguing as if they passionately care about how humans work when, as you say, they don't care at all.
People should just recognize, as you have done, that they don't actually care about how the human language faculty works. It's baffling that they instead choose to make absurd arguments to defend fields they don't care one way or another about.
When Chomsky says that LLMs aren't how the human faculty works it would be so easy to tell the truth and say "I don't care how the human language faculty works" and everyone can go focus on the things they are interested in, just as it would be easy for a GPS designer to say "I don't care how insect navigation works".
There is no problem as long as you don't pretend to be caring about (this aspect of) science.
People should just recognize, as you have done, that they don't actually care about how the human language faculty works. It's baffling that they instead choose to make absurd arguments to defend fields they don't care one way or another about.
When Chomsky says that LLMs aren't how the human faculty works it would be so easy to tell the truth and say "I don't care how the human language faculty works" and everyone can go focus on the things they are interested in, just as it would be easy for a GPS designer to say "I don't care how insect navigation works".
There is no problem as long as you don't pretend to be caring about (this aspect of) science.
Is it polite to deprive readers of context necessary to understand what the speaker is talking about? I was also very confused by that part and I had no idea whom or what he was talking about or why he even started taking about that.
I searched for an actual paper by that guy because you’ve mentioned his real name. I found “Modern language models refute Chomsky’s approach to language”. After reading it seems even more true that Chomsky’s Tom Jones is a strawman.
I searched for an actual paper by that guy because you’ve mentioned his real name. I found “Modern language models refute Chomsky’s approach to language”. After reading it seems even more true that Chomsky’s Tom Jones is a strawman.
> After reading it seems even more true that Chomsky's Tom Jones is a strawman.
Lol. It's clear you are not interested in having any kind of rational discussion on the topic and are driven by some kind of zealotry when you claim to have read a technical 40 page paper (with an additional 18 pages of citations) in 30 minutes.
Even if by some miraculous feat you had read it you haven't made a single actual argument or addressed any of the points made by Chomsky.
Lol. It's clear you are not interested in having any kind of rational discussion on the topic and are driven by some kind of zealotry when you claim to have read a technical 40 page paper (with an additional 18 pages of citations) in 30 minutes.
Even if by some miraculous feat you had read it you haven't made a single actual argument or addressed any of the points made by Chomsky.
It’s certainly not a dense paper with careful nuanced derivations that you have to ponder to grasp. It’s a light read you can skim especially if you aren’t interested in LLM Trump improv and you are familiar with the general thought behind connectionism, construction grammar, other modern linguistic theories and, of course, universal grammar. The debate is as old as UG, but now with a new LLM flavor.
I don’t know which argument you expect from me. I read it and found nothing similar to “Stop wasting your time; naval vessels do it all the time.” So I concluded it’s a strawman. Being against a particular controversial approach in linguistics doesn’t mean being against science.
I don’t know which argument you expect from me. I read it and found nothing similar to “Stop wasting your time; naval vessels do it all the time.” So I concluded it’s a strawman. Being against a particular controversial approach in linguistics doesn’t mean being against science.
> I read it and found nothing similar to “Stop wasting your time; naval vessels do it all the time.”
You implied in the previous paragraph that you didn't in fact read it and you only "skimmed" it. Maybe that's why you "found nothing similar to 'stop wasting your time; naval vessels do it all the time". But even in skimming the paper it's incomprehensible how you could miss it: At least the first 23 pages of the draft version I have just describe how well LLMs perform and completely ignores the relevant question of how human language works. (It doesn't get any better after the first 23 pages). So presumably you just don't know what an analogy is and are literally searching for the term "naval vessels".
Here's just one example demonstrating that Piantodosi does in fact claim what Chomsky says he does: Piantodosi writes "The success of large language models is a failure for generative theories because it goes against virtually all of the principles these theories have espoused." Rewriting that statement using Chomsky's analogy illustrates how idiotic the original statement is: "The success of naval vessels is a failure for insect navigation theories because it goes against all of the principles these theories have espoused".
You implied in the previous paragraph that you didn't in fact read it and you only "skimmed" it. Maybe that's why you "found nothing similar to 'stop wasting your time; naval vessels do it all the time". But even in skimming the paper it's incomprehensible how you could miss it: At least the first 23 pages of the draft version I have just describe how well LLMs perform and completely ignores the relevant question of how human language works. (It doesn't get any better after the first 23 pages). So presumably you just don't know what an analogy is and are literally searching for the term "naval vessels".
Here's just one example demonstrating that Piantodosi does in fact claim what Chomsky says he does: Piantodosi writes "The success of large language models is a failure for generative theories because it goes against virtually all of the principles these theories have espoused." Rewriting that statement using Chomsky's analogy illustrates how idiotic the original statement is: "The success of naval vessels is a failure for insect navigation theories because it goes against all of the principles these theories have espoused".
There is a difference between supporting one research paradigm over another and rejecting science altogether to focus on engineering. The first quote and the context around it implies the latter.
The success of naval vessels shows it’s possible to navigate without innate star and wind comprehension, so maybe we should think of that inner stuff as phlogiston. (Yeah, this analogy isn’t as nice but it’s quite hard to translate the nuance of linguistic debate into nautical terms.)
The success of naval vessels shows it’s possible to navigate without innate star and wind comprehension, so maybe we should think of that inner stuff as phlogiston. (Yeah, this analogy isn’t as nice but it’s quite hard to translate the nuance of linguistic debate into nautical terms.)
Do you not genuinely not understand the logic of the argument? The fact that A does Y using Z doesn't entail that B does Y using Z.
I genuinely don’t understand how the analogy about naval vessels is a fair simplification of the argument that Chomsky’s research programme is, heuristically, a dead-end and should be abandoned. What is A,B,Y,Z?
It’s not like it’s an outrageous position. Chomsky’s tradition is quite controversial and is outside of mainstream nowadays. And connectionism is a valid scientific approach.
It’s not like it’s an outrageous position. Chomsky’s tradition is quite controversial and is outside of mainstream nowadays. And connectionism is a valid scientific approach.
There's a reason Max Planck said science advances one funeral at a time. Researches spend their lives developing and promoting the ideas they cut their teeth on (or in this case developed himself) and their view of what is possible becomes ossified around these foundational beliefs. Expecting him to be flexible enough in his advanced age to view LLMs with a fresh perspective, rather than strongly informed by his core
theoretical views is expecting too much.
I'm noticing that leftists overwhelmingly toe the same line on AI skepticism, which suggests to me an ideological motivation.
Chomsky's problem here has nothing to do with his politics, but unfortunately a lot to do with his long-held position in the Nature/Nurture debate - a position that is undermined by the ability of LLMs to learn language without hardcoded grammatical rules:
Chomsky introduced his theory of language acquisition, according to which children have an inborn quality of being biologically encoded with a universal grammar
I don't see how the two things are related. Whether acquisition of human language is nature or nurture - it is still learning of some sort.
Yes, maybe we can reproduce that learning process in LLMs, but that doesn't mean the LLMs imitate only the nurture part (might as well be just finetuning), and not the nature part.
An airplane is not an explanation for a bird's flight.
Yes, maybe we can reproduce that learning process in LLMs, but that doesn't mean the LLMs imitate only the nurture part (might as well be just finetuning), and not the nature part.
An airplane is not an explanation for a bird's flight.
The great breakthrough in AI turned out to be LLMs.
Nature, for an LLM, is its design: graph, starting weights, etc.
Environment, for an LLM, is what happens during training.
LLMs are capable of learning grammar entirely from their environment, which suggests that infants are too, which is bad for Chomsky's position that the basics of grammar are baked into human DNA.
Nature, for an LLM, is its design: graph, starting weights, etc.
Environment, for an LLM, is what happens during training.
LLMs are capable of learning grammar entirely from their environment, which suggests that infants are too, which is bad for Chomsky's position that the basics of grammar are baked into human DNA.
LLMs require vastly more data than humans and still struggle with some more esoteric grammatical rules like parasitic gaps. The fact grammar can be approximated given trillions of words doesn't explain how babies learn language from a much more modest dataset.
I think it does. I think LLM showed us possibility that maybe there's no language but just pile of memes and supplemental compression scheme that is grammar.
LLM had really destroyed Chomsky's positions in multiple different ways: nothing perform even close to LLM in language generation, yet it didn't grow a UG for natural languages, while it did develop a shared logic for non-natural languages and abstract concepts, while dataset needing to be heavily English biased to be English fluent, and parameter count needing to be truly massive as multiple hundred billion parameters large, so on and on.
Those are all circumstantial evidences at best, a random paraphernalia of statements that aren't even appropriate to bring into discussions, all meaningless - in the sense that an open hand of a person observing another individual aligned to a line between standing position of the person to the center of nearest opening of a wall would be meaningless.
LLM had really destroyed Chomsky's positions in multiple different ways: nothing perform even close to LLM in language generation, yet it didn't grow a UG for natural languages, while it did develop a shared logic for non-natural languages and abstract concepts, while dataset needing to be heavily English biased to be English fluent, and parameter count needing to be truly massive as multiple hundred billion parameters large, so on and on.
Those are all circumstantial evidences at best, a random paraphernalia of statements that aren't even appropriate to bring into discussions, all meaningless - in the sense that an open hand of a person observing another individual aligned to a line between standing position of the person to the center of nearest opening of a wall would be meaningless.
>LLM had really destroyed Chomsky's positions in multiple different ways: nothing perform even close to LLM in language generation, yet it didn't grow a UG for natural languages
Do you even understand Chomsky's position?
Do you even understand Chomsky's position?
To be honest, I don't, at least not entirely. Noam Chomsky to me is patron saint of compilers and apparent sources of quotes used to justify eye-rolling decisions regarding i18n. At least a lot of his followers' understanding is that the UG is THE UG and a Universal Syntax, and/or is a decisive and scientific refutation of Sapir-Whorf hypothesis as well as European structuralism, not whatever his later works on UG that progressively pivoted its definition or nature vs nurture debates were "meant" to be discussing.
To me this text look like his Baghdad Bob moment. Silly but right and noble. What else is it?
Ironically these days you can just throw this text at ChatGPT to have it debloat or critique text like this transcripts. Worse results than taking time reading yourself, but gives you validation if that is what is needed.
To me this text look like his Baghdad Bob moment. Silly but right and noble. What else is it?
Ironically these days you can just throw this text at ChatGPT to have it debloat or critique text like this transcripts. Worse results than taking time reading yourself, but gives you validation if that is what is needed.
It's not that the invention of LLMs conclusively disproves Chomksy's position.
However, we now have a proof-of-concept that a computer can learn grammar in a sophisticated way, from the ground up.
We have yet to code something procedural that approaches the same calibre via a hard-coded universal grammar.
That may not obliterate Chomksy's position, but it looks bad.
However, we now have a proof-of-concept that a computer can learn grammar in a sophisticated way, from the ground up.
We have yet to code something procedural that approaches the same calibre via a hard-coded universal grammar.
That may not obliterate Chomksy's position, but it looks bad.
That's not the goal of generative linguistics though; it's not an engineering project.
The problem encompasses not just biology and information technology, but also linguistics. Even if LLMs say nothing about biology, they do tell us something about the nature of language itself.
Again, that LLMs can learn to compose sophisticated texts from training alone does not close the case on Chomsky's position.
However, it is a piece of evidence against it. It does suggest, by Occam's razor, that a hardwired universal grammar is the lesser theory.
Again, that LLMs can learn to compose sophisticated texts from training alone does not close the case on Chomsky's position.
However, it is a piece of evidence against it. It does suggest, by Occam's razor, that a hardwired universal grammar is the lesser theory.
How do LLMs explain how 5 year olds respect island constraints?
I don't have the domain knowledge to discuss that.
If you don't know what a syntactic island is, perhaps you're not the best judge of the plausibility of a linguistic theory.
Fantastic, let's have a debate about me /s
> AI skepticism
Isn't AI optimism an ideological motivation? It's a spectrum, not a mental model.
Isn't AI optimism an ideological motivation? It's a spectrum, not a mental model.
Whether one expects AI to be powerful or weak should have nothing to do with political slant, but here it seems to inform the opinion. It begs the question: what do they want to be true? The enemy is both too strong and too weak.
They're firmly on one extreme end of the spectrum. I feel as though I'm somewhere in between.
They're firmly on one extreme end of the spectrum. I feel as though I'm somewhere in between.
Leftists and intellectuals overlap a lot. LLM text must be still full of six fingered hands to many of them.
For Chomsky specifically, the entire existence of LLM, however it's framed, is a massive middle finger to him and a strike-through on a large part of his academic career. As much as I find his UG theory and its supporters irritating, it might be felt a bit unfair to someone his age.
For Chomsky specifically, the entire existence of LLM, however it's framed, is a massive middle finger to him and a strike-through on a large part of his academic career. As much as I find his UG theory and its supporters irritating, it might be felt a bit unfair to someone his age.
99%+ of humans on this planet do not investigate an issue, they simply accept a trusted opinion of an issue as fact. If you think this is a left only issue you havent been paying attention.
Usually what happens is the information bubble bursts, and gets corrected, or it just fades out.
Usually what happens is the information bubble bursts, and gets corrected, or it just fades out.
Then you obviously didn't listen to a word Chomsky has said on the subject.
I was quite dismissive of him on LLMs until I realized the utter hubris and stupidity of dismissing Chomsky on language.
I think it was someone asking if he was familiar with the Wittgenstein Blue and Brown books and of course because he as already an assistant professor at MIT when they came out.
I still chuckle at my own intellectual arrogance and stupidity when thinking about how I was dismissive of Chomsky on language. I barely know anything and I was being dismissive of one of unquestionable titans and historic figures of a field.
I was quite dismissive of him on LLMs until I realized the utter hubris and stupidity of dismissing Chomsky on language.
I think it was someone asking if he was familiar with the Wittgenstein Blue and Brown books and of course because he as already an assistant professor at MIT when they came out.
I still chuckle at my own intellectual arrogance and stupidity when thinking about how I was dismissive of Chomsky on language. I barely know anything and I was being dismissive of one of unquestionable titans and historic figures of a field.
Chomsky has been colossally wrong on universal grammar.
https://www.scientificamerican.com/article/evidence-rebuts-c...
But at least he admits that:
https://dlc.hypotheses.org/1269#
https://www.scientificamerican.com/article/evidence-rebuts-c...
But at least he admits that:
https://dlc.hypotheses.org/1269#
This is a great way to remove any nuance and chance of learning from a conversation. Please don't succumb to black-and-white (or red-and-blue) thinking, it's harmful to your brain.
You're projecting.
Or an ideological alignment of values. Generative AI is strongly associated with large corporations that are untrusted (to put it generously) by those on the left.
An equivalent observation might be that the only people who seem really, really excited about current AI products are grifters who want to make money selling it. Which looks a lot like Blockchain to many.
An equivalent observation might be that the only people who seem really, really excited about current AI products are grifters who want to make money selling it. Which looks a lot like Blockchain to many.
I think viewing the world as either leftist or right wing is rather limiting philosophy and way to go through life. Most people are a lot more complicated than that.
I have experienced this too. It's definitely part of the religion but I'm not sure why tbh. Maybe they equate it with like tech is bad mkay, which, looking at who leads a lot of the tech companies, is somewhat understandable, altho very myopic.
I see this as much more of a hackers vs. corporations ideological split. Which imperfectly maps to leftism vs conservatism.
The perception on the left is that once again, corporations are foisting products on us that nobody wants, with no concern for safety, privacy, or respect for creators.
For better or worse, the age of garage-tech is mostly dead and Tech has become synonymous with corporatism. This is especially true with GenAI, where the resources to construct a frontier model (or anything remotely close to it) are far outside what a hacker can afford.
The perception on the left is that once again, corporations are foisting products on us that nobody wants, with no concern for safety, privacy, or respect for creators.
For better or worse, the age of garage-tech is mostly dead and Tech has become synonymous with corporatism. This is especially true with GenAI, where the resources to construct a frontier model (or anything remotely close to it) are far outside what a hacker can afford.
> I see this as much more of a hackers vs. corporations ideological split.
That framing may be true within tech circles, not the broader political divide. "Hackers" aren't collectively discounting and ignoring AI tools regardless of their enthusiasm for open-source.
Safety-ism is also most popular among those see useful potential in AI, and a generous enough timeline for AGI.
That framing may be true within tech circles, not the broader political divide. "Hackers" aren't collectively discounting and ignoring AI tools regardless of their enthusiasm for open-source.
Safety-ism is also most popular among those see useful potential in AI, and a generous enough timeline for AGI.
That makes sense, and there's definitely an element of truth to that position. The trouble is, the response is to dissociate with the technology, which is really not a tenable position if you intend to have a meaningful part in like... anything in the future. What I see-- and this is just my personal experience-- is that leftists tend to want to pretend it isn't happening, or that it won't matter. When it fact nothing matters more.
The deepest of deep ironies: I talk to people all the time talking about ushering in an age of post-capitalism and ignoring AI. When I personally can't see how the AI of the next decade and capitalism can coexist, the latter being based on human labor and all. Like, AI is going to be the reason what you want is going to happen, so why ignore it?
The deepest of deep ironies: I talk to people all the time talking about ushering in an age of post-capitalism and ignoring AI. When I personally can't see how the AI of the next decade and capitalism can coexist, the latter being based on human labor and all. Like, AI is going to be the reason what you want is going to happen, so why ignore it?
It is unfortunate opinion, because I personally hold Chomsky in fairly high regard and give most of his thoughts I am familiar with a reasonable amount of consideration if only because he could, I suppose in the olden days now, articulate his points well and make you question your own thought process. This no longer seems to be the case though as I found the linked article somewhat difficult to follow. I suppose age can get to anyone.
Not that I am an LLM zealot. Frankly, some of the clear trajectory it puts humans on makes me question our futures in this timeline. But even if I am not a zealot, but merely an amused, but bored middle class rube, the serious issues with it ( privacy, detailed personal profiling that surpasses existing systems, energy use, and actual power of those who wield it ), I can see it being implemented everywhere with a mix of glee and annoyance.
I know for a fact it will break things and break things hard and it will be people, who know how things actually work that will need to fix those.
I will be very honest though. I think Chomsky is stuck in his internal model of the world and unable to shake it off. Even his arguments fall flat, because they don't fit the domain well. It seems like they should given that he practically made his name on syntax theory ( which suggests his thoughts should translate well into it ) and yet.. they don't.
I have a minor pet theory on this, but I am still working on putting it into some coherent words.
Not that I am an LLM zealot. Frankly, some of the clear trajectory it puts humans on makes me question our futures in this timeline. But even if I am not a zealot, but merely an amused, but bored middle class rube, the serious issues with it ( privacy, detailed personal profiling that surpasses existing systems, energy use, and actual power of those who wield it ), I can see it being implemented everywhere with a mix of glee and annoyance.
I know for a fact it will break things and break things hard and it will be people, who know how things actually work that will need to fix those.
I will be very honest though. I think Chomsky is stuck in his internal model of the world and unable to shake it off. Even his arguments fall flat, because they don't fit the domain well. It seems like they should given that he practically made his name on syntax theory ( which suggests his thoughts should translate well into it ) and yet.. they don't.
I have a minor pet theory on this, but I am still working on putting it into some coherent words.
I recently saw a new LLM that was fooled by "20 pounds of bricks vs 20 feathers". These are not reasoning machines.
I recently had a computer tell me that 0.1 + 0.2 != 0.3. It must not be a math capable machine.
Perhaps it is more important to know the limitations of tools rather than dismiss their utility entirely due to the existence of limitations.
Perhaps it is more important to know the limitations of tools rather than dismiss their utility entirely due to the existence of limitations.
A computer isn't a math capable machine.
> Perhaps it is more important to know the limitations of tools rather than dismiss their utility entirely due to the existence of limitations.
Well, yes. And "reasoning" is only something LLMs do coincidentally, to their function as sequence continuation engines. Like performing accurate math on rationale numbers, it can happen if you put in a lot of work and accept a LOT of expensive computation. Even then there exists computations that just are not reasonable or feasible.
Reminding folks to dismiss the massive propaganda engine pushing this bubble isn't "dismissing their utility entirely".
These are not reasoning machines. Treating them like they are will get you hurt eventually.
> Perhaps it is more important to know the limitations of tools rather than dismiss their utility entirely due to the existence of limitations.
Well, yes. And "reasoning" is only something LLMs do coincidentally, to their function as sequence continuation engines. Like performing accurate math on rationale numbers, it can happen if you put in a lot of work and accept a LOT of expensive computation. Even then there exists computations that just are not reasonable or feasible.
Reminding folks to dismiss the massive propaganda engine pushing this bubble isn't "dismissing their utility entirely".
These are not reasoning machines. Treating them like they are will get you hurt eventually.
My point is that computers, when used properly, can absolutely do math. And LLMs, when used properly, can absolutely explain the reasoning behind why a pound of bricks and a pound of feathers weigh the same.
Can they reason? Maybe, depending on your definition of reasoning.
An example: which weighs more a pound of bricks and 453.59 grams of feathers? Explain your reasoning.
LLM: The pound of bricks weighs slightly more.
*Reasoning:*
* *1 pound* is officially defined as *0.45359237 kilograms*, which is *453.59237 grams*. * You have *453.59 grams* of feathers.
So, the pound of bricks (453.59237 grams) weighs a tiny fraction more than the 453.59 grams of feathers. For most practical purposes, they'd be considered the same, but technically, the bricks are heavier by 0.00237 grams. /llm
It is both correct and the reasoning is sound. Do I understand that the machine is a pattern following machine, yes! Is there an argument to be made that humans are also that? Probably. Chomsky himself argued in favor of a universal grammar, after all.
I’m steel manning this a bit, but the point is that LLMs are capable of doing some things which are indistinguishable from human reasoning in terms of results. Does the process matter in all cases?
Can they reason? Maybe, depending on your definition of reasoning.
An example: which weighs more a pound of bricks and 453.59 grams of feathers? Explain your reasoning.
LLM: The pound of bricks weighs slightly more.
*Reasoning:*
* *1 pound* is officially defined as *0.45359237 kilograms*, which is *453.59237 grams*. * You have *453.59 grams* of feathers.
So, the pound of bricks (453.59237 grams) weighs a tiny fraction more than the 453.59 grams of feathers. For most practical purposes, they'd be considered the same, but technically, the bricks are heavier by 0.00237 grams. /llm
It is both correct and the reasoning is sound. Do I understand that the machine is a pattern following machine, yes! Is there an argument to be made that humans are also that? Probably. Chomsky himself argued in favor of a universal grammar, after all.
I’m steel manning this a bit, but the point is that LLMs are capable of doing some things which are indistinguishable from human reasoning in terms of results. Does the process matter in all cases?
> Does the process matter in all cases?
So there are 2 dimensions being conflated here:
"Does how the reasoning work matter in all cases" Pretty Obviously no, but it may matter in some of them. We also don't really understand which ones yet.
"Does the reasoning work as intended in all cases?" Pretty Obviously no, but it doesn't work for at least some of them. We also don't really understand which ones yet.
"We also don't really understand which ones yet" Is the critical point of caution.
So there are 2 dimensions being conflated here:
"Does how the reasoning work matter in all cases" Pretty Obviously no, but it may matter in some of them. We also don't really understand which ones yet.
"Does the reasoning work as intended in all cases?" Pretty Obviously no, but it doesn't work for at least some of them. We also don't really understand which ones yet.
"We also don't really understand which ones yet" Is the critical point of caution.
Surely it just reasoned that you made a typo and "autocorrected" your riddle. Isn't this what a human would do? Though to be fair, a human would ask you again to make sure they heard you correctly. But it would be kind of annoying if you had to verify every typo when using an LLM.
Tons of people fall for this too. Are they not reasoning? LLMs can also be bad reasoning machines.
I dont have much use for a bad reasoning machine.
I could retort with another gotcha argument, but instead of doing that perhaps we can do better than that?
An attempt: They are bad reasoning machines that already are useful in a few domains and they're improving faster than evolutionary speeds. So even if they're not useful today in a domain relevant to you there's a significant possibility they might be in a few months. AlphaEvolve would have been scifi a decade ago.
"It's like if a squirrel started playing chess and instead of "holy shit this squirrel can play chess!" most people responded with "But his elo rating sucks""
An attempt: They are bad reasoning machines that already are useful in a few domains and they're improving faster than evolutionary speeds. So even if they're not useful today in a domain relevant to you there's a significant possibility they might be in a few months. AlphaEvolve would have been scifi a decade ago.
"It's like if a squirrel started playing chess and instead of "holy shit this squirrel can play chess!" most people responded with "But his elo rating sucks""
I can think of tons of uses for a bad reasoning machine as long as it’s cheap enough.
Which those things aren't. In fact they cost considerably more than hiring someone.
LLMs cost significantly less than even a high schooler
Just because for now they are burning money and it's priced considerably under what it's costing them.
Which is why I spoke of "cost" not of "price".
They're in the "disrupt" phase. But that's not forever.
Which is why I spoke of "cost" not of "price".
They're in the "disrupt" phase. But that's not forever.
No. The marginal cost of an LLM is much, much lower than a high schooler. It is not even close. There is a lot of investment happening but revenue will continue to increase as the product improves and more use it or the money will stop flowing. If training stopped LLMs would be immensely profitable right now
But are you aware of the weight comparison of a gallon of water vs a gallon of butane ?
No im not. A gallon is a measure of volume? This is a USA unit.
[Edit to remove: It was not clear that this was someone else's intro re-posted on Chomsky's site]
Maybe I am missing context, but it seems like he’s defending himself from the claim that we shouldn’t bother studying language acquisition and comprehension in humans because of LLM’s?
Who would make such a claim? LLM’s are of course incredible, but it seems obvious that their mechanism is quite different than the human brain.
I think the best you can say is that one could motivate lines of inquiry in human understanding, especially because we can essentially do brain surgery on an LLM in action in a way that we can’t with humans.
Who would make such a claim? LLM’s are of course incredible, but it seems obvious that their mechanism is quite different than the human brain.
I think the best you can say is that one could motivate lines of inquiry in human understanding, especially because we can essentially do brain surgery on an LLM in action in a way that we can’t with humans.
Steven Piantadosi makes that argument.
> It’s as if a biologist were to say: “I have a great new theory of organisms. It lists many that exist and many that can’t possibly exist, and I can tell you nothing about the distinction.”
> Again, we’d laugh. Or should.
Should we? This reminds me acutely of imaginary numbers. They are a great theory of numbers that can list many numbers that do 'exist' and many that can't possibly 'exist'. And we did laugh when imaginary numbers were first introduced - the name itself was intended as a derogatory term for the concept. But who's laughing now?
> Again, we’d laugh. Or should.
Should we? This reminds me acutely of imaginary numbers. They are a great theory of numbers that can list many numbers that do 'exist' and many that can't possibly 'exist'. And we did laugh when imaginary numbers were first introduced - the name itself was intended as a derogatory term for the concept. But who's laughing now?
Imaginary numbers are not relevant at all. There’s nothing whatsoever to do with the everyday use of the word imaginary. They could just as easily have been called “vertical numbers” and real numbers called “horizontal numbers” in order to more clearly illustrate their geometric interpretation in the complex plane.
The term “imaginary number” was coined by Rene Descartes as a derogatory and the ill intent behind his term has stuck ever since. I suspect his purpose was theological rather than mathematical and we are all the worse for it.
The term “imaginary number” was coined by Rene Descartes as a derogatory and the ill intent behind his term has stuck ever since. I suspect his purpose was theological rather than mathematical and we are all the worse for it.
I'm confused by this comment - it seems to just be restating what my comment said.
This is the point where i realized he has no clue what he is saying. Theres so many creatures that once existed that can never again exist on earth due to the changes that the planet has gone through over millions, billions of years. The oxygen rich atmosphere that supported the dinosaurs for instance. If we had some kind of system that can put together proper working DNA for all the creatures that ever actually existed on this planet, some half of them would be completely nonviable if introduced to the ecosystem today. He is failing to see that there is an incredible understanding of systems that we are producing with this work, but he is a very old man from a very different time and contrarianism is often the only way to look smart or reasoned when you have no clue whats actually going on, so I am not shocked by his take.
In the case of complex numbers mathematicians understand the distinction extremely well, so I'm not sure it's a perfect analogy.
I have a degree in linguistics. We were taught Chomsky’s theories of linguistics, but also taught that they were not true. (I don’t want to say what university it was since this was 25 years ago and for all I know that linguistics department no longer teaches against Chomsky). The end result is I don’t take anything Chomsky says seriously. So, it is difficult for me to engage with Chomsky’s ideas.
I'm rather confused by this statement. I've read a number of Chomsky pieces and have listened to him speak a number of times. To say his theories were all "not true" seems, to an extent, almost impossible.
Care to expand on how his theories can be taught in such a binary way?
Care to expand on how his theories can be taught in such a binary way?
GP may be referring to the idea that language is innate like an organ in the body/brain. The Kingdom of Speech by Tom Wolfe is a great read exploring Chomsky and other thinkers in this realm. It would have been great to see what he thought of LLMs too.
Generally what people are talking about are his universal grammar or generative syntax theories/approaches, which are foundational to how you approach many topics. Because you build your academic career based on specialization they are hotly contested (for the material reasons of jobs, funding, tenure, etc.).
This leads to people who agree hiring each other and departments ‘circling the wagon’ on these issues. You’ll see this referred to as east vs west coast, but it’s not actually that clearly geographically delineated.
So anyways, these are open questions that people do seriously discuss and study, but the politics of academia make it difficult and unfortunately this often trickles down to students.
This leads to people who agree hiring each other and departments ‘circling the wagon’ on these issues. You’ll see this referred to as east vs west coast, but it’s not actually that clearly geographically delineated.
So anyways, these are open questions that people do seriously discuss and study, but the politics of academia make it difficult and unfortunately this often trickles down to students.
I’m speaking about his linguistic theories such as universal grammar.
This reminds me of the debates over F.R. Leavis, and the impact it had on modern english teaching worldwide. There are a small dying cohort of english professors who are refugees from internecine warfare.
Same thing happened in Astronomy. Students of Fred Hoyle can't work in some institutions. &c &c.
Same thing happened in Astronomy. Students of Fred Hoyle can't work in some institutions. &c &c.
I don't have a degree in linguistics, but I took a few classes about 15 years ago, and Chomsky's works were basically treated as gospel. Although my university's linguistics faculty included several of his former graduate students, so maybe there's a bias factor. In any case, it reminds me of an SMBC comic about how math and science advance over time [1]
[1] https://smbc-wiki.com/index.php/How-math-works
[1] https://smbc-wiki.com/index.php/How-math-works
Linguistics has been largely subsumed by CS (LLM, speech synthesis, translation). It's not an empirical science or social science and most of its theories are not falsifiable.
But generally speaking Chomsky's ideas, and in particular, the Universal Grammar are no longer in vogue.
But generally speaking Chomsky's ideas, and in particular, the Universal Grammar are no longer in vogue.
Linguistics is a heterogenous field, an while some parts aren't empirical, others absolutely are. I can attest to that from the journal articles I read in undergrad. They had experimental data and statistical analyses.
Chomsky is always saying that LLMs and such can only imitate, not understand language. But I wonder if there is a degree of sophistication at which he would concede these machines exceed "imitation". If his point is that LLMs arrive at language in a way different than humans... great. But I'm not sure how he can argue that some kind of extremely sophisticated understanding of natural language is not embedded in these models in a way that, at this point, exceeds the average human. In all fairness, this was written in 2023, but given his longstanding stubbornness on this topic, I doubt it would make a difference.
I think what would "convince" Chomsky is more akin to the explainability research currently in it's infancy, producing something akin to a branch of information theory for language and thought.
Chomsky talks about how the current approach can't tell you about what humans are doing, only approximate it; the example he has given in the past is taking thousands of hours of footage of falling leaves and then training a model to make new leaf falling footage versus producing a model of gravity, gas mechanics for the air currents, and air resistance model of leaves. The later representation is distilled down into something that tells you about what is happening at the end of some scientific inquiry, and the former is a opaque simulation for engineering purposes if all you wanted was more leaf falling footage.
So I interpret Chomsky as meaning "Look, these things can be great for an engineering purpose but I am unsatisfied in them for scientific research because they do not explain language to me" and mostly pushing back against people implying that the field he dedicated much of his life to is obsolete because it isn't being used for engineering new systems anymore, which was never his goal.
Chomsky talks about how the current approach can't tell you about what humans are doing, only approximate it; the example he has given in the past is taking thousands of hours of footage of falling leaves and then training a model to make new leaf falling footage versus producing a model of gravity, gas mechanics for the air currents, and air resistance model of leaves. The later representation is distilled down into something that tells you about what is happening at the end of some scientific inquiry, and the former is a opaque simulation for engineering purposes if all you wanted was more leaf falling footage.
So I interpret Chomsky as meaning "Look, these things can be great for an engineering purpose but I am unsatisfied in them for scientific research because they do not explain language to me" and mostly pushing back against people implying that the field he dedicated much of his life to is obsolete because it isn't being used for engineering new systems anymore, which was never his goal.
I guess it's because LLM does not understand the meaning as you understand what you read or thought. LLMs are machines that modulate hierarchical positions, ordering the placement of a-signifying sign without a clue of the meaning of what they ordered (that's why machine can hallucinate :they don't have a sense of what they express)
From what I've read/watched of Chomsky he's holding out for something that truly cannot be distinguished from human no matter how hard you tried.
I think that misses the point entirely. Even if you constructed some system the output of which could not be distinguished from human-produced language but that either (1) clearly operated according to principles other than those that govern human language or (2) operated according to principles that its creators could not adequately explain, it would not be of that much interest to him.
He wants to understand how human language works. If I get him right — and I'm absolutely sure that I don't in important ways — then LLMs are not that interesting because both (1) and (2) above are true of them.
He wants to understand how human language works. If I get him right — and I'm absolutely sure that I don't in important ways — then LLMs are not that interesting because both (1) and (2) above are true of them.
It's always good to humble the ivory tower.
That's not quite a valid point considering the article's conclusion: sowing dissent in the sciences allows companies to more easily package and sell carcinogens like asbestos, lead paint, and tobacco products.
I understand his diction is a bit impenetrable but I believe the intention is to promote literacy and specificity, not just to be a smarty-pants.
I understand his diction is a bit impenetrable but I believe the intention is to promote literacy and specificity, not just to be a smarty-pants.
That may change, particularly if the intelligence of LLMs proves to be analogous to our own in some deep way—a point that is still very much undecided. However, if the similarities are there, so is the potential for knowledge. We have a complete mechanical understanding of LLMs and can pry apart their structure, which we cannot yet do with the brain. And some of the smartest people in the world are engaged in making LLMs smaller and more efficient; it seems possible that the push for miniaturization will rediscover some tricks also discovered by the blind watchmaker. But these things are not a given.