Do large language models understand us?(medium.com)
medium.com
Do large language models understand us?
https://medium.com/@blaisea/do-large-language-models-understand-us-6f881d6d8e75
73 comments
> Animals with very limited language capabilities have the ability to run their own lives, and to some extent manipulate their environment.
This makes me think of the ongoing effort by some people to train their pets to use talking buttons to communicate. These include the animals occasionally trying to convey relatively complicated ideas with simple sets of words, like "play help" ("I'm bored, come entertain me") [1] and "dog settle" ("make that dog outside stop being annoying") [2].
[1]: https://www.youtube.com/watch?v=zAQKA9WFlq4
[2]: https://www.youtube.com/watch?v=5sY3K83Zao8
This makes me think of the ongoing effort by some people to train their pets to use talking buttons to communicate. These include the animals occasionally trying to convey relatively complicated ideas with simple sets of words, like "play help" ("I'm bored, come entertain me") [1] and "dog settle" ("make that dog outside stop being annoying") [2].
[1]: https://www.youtube.com/watch?v=zAQKA9WFlq4
[2]: https://www.youtube.com/watch?v=5sY3K83Zao8
The contrary view is that language is just part of the I/O system, not where the hard work gets done.
The meaning of "where" in your claim is very slippery. AR Luria in The Working Brain points out that damage to virtually every part of the brain can result in language dysfunction. To say that a whole lot of things happen apparently without language doesn't demonstrate that language is an extra. The brain generally isn't very modular and my alternative take would view language as accompaniment that sometimes is "just words" and sometimes touches very specific details (edit: plus allow recursive access these other brain-functionalities, as others point out). We can see how GPT-3's output is a stream of plausible things punctuated by some statement that shows "it doesn't anything" (IE, gets the implicit context wildly wrong).
The meaning of "where" in your claim is very slippery. AR Luria in The Working Brain points out that damage to virtually every part of the brain can result in language dysfunction. To say that a whole lot of things happen apparently without language doesn't demonstrate that language is an extra. The brain generally isn't very modular and my alternative take would view language as accompaniment that sometimes is "just words" and sometimes touches very specific details (edit: plus allow recursive access these other brain-functionalities, as others point out). We can see how GPT-3's output is a stream of plausible things punctuated by some statement that shows "it doesn't anything" (IE, gets the implicit context wildly wrong).
> Animals with very limited language capabilities have the ability to run their own lives, and to some extent manipulate their environment. Systems built from large language databases don't have that.
This may well point to the possibility that animals have a theory of mind.
This may well point to the possibility that animals have a theory of mind.
I don't know if they have a theory of mind, but some animals absolutely have very complex behavioural prediction models for humans and other animals. I swear my last dog knew whether I intended to go outside based solely on how I stood up from the couch.
This can be explained as associative learning, without needing to reach for ToM, which is still no mean feat.
I believe it's been shown as a general capability in [corvids](https://www.semanticscholar.org/paper/Ravens-attribute-visua...), great apes and maybe possibly in pigs (and goats). It's sort of game theoretic, as in "I know that you know that I stash food here so I'm going to pretend to hide the food here". I've never read anything that goes beyond two levels of "I knows" but it can be sophisticated, such as in the linked paper on Ravens where there is line of sight through a peephole that can be open or closed.
Sophisticated ToM is likely rare in the animal kingdom, not usually going beyond programmed responses to being observed and possibly empathizing with conspecific suffering.
In humans, it's a much more sophisticated process of inferring motivations, anticipating moves and a number of similar inferential capabilities plus abductive reasoning.
Sophisticated ToM is likely rare in the animal kingdom, not usually going beyond programmed responses to being observed and possibly empathizing with conspecific suffering.
In humans, it's a much more sophisticated process of inferring motivations, anticipating moves and a number of similar inferential capabilities plus abductive reasoning.
If you've ever spoken to someone with Alzheimer's, or certain kinds of very severe cognitive disorders, it becomes clear that language fluency tells you little to nothing about effective cognition. There are people who can't add 1+1 who can babble about random stuff convincingly, for a while, anyway, unless you're probing it.
Manipulation involves a few things:
1. Recognize objects in the world, like a keyhole or whatever it is you want to take out of a bin.
2. Recognize your own position -- how close is your hand to the keyhole?
3. Adjust your position until it matches your goal.
My impression of things is that most of our difficulty comes from task 3. It's not so much that we have trouble recognizing where we are, where we want to be, or how to move from where we are to where we want to be. But we have difficulty defining movements that don't cause damage to nearby (or target) objects.
Your human hand has many features that allow it to grip an object securely without damaging the object. And you also have a decent sense for how to move your hand to an object without damaging the object -- or your hand -- though this problem is far from being solved in humans too. Ever stub your toe?
1. Recognize objects in the world, like a keyhole or whatever it is you want to take out of a bin.
2. Recognize your own position -- how close is your hand to the keyhole?
3. Adjust your position until it matches your goal.
My impression of things is that most of our difficulty comes from task 3. It's not so much that we have trouble recognizing where we are, where we want to be, or how to move from where we are to where we want to be. But we have difficulty defining movements that don't cause damage to nearby (or target) objects.
Your human hand has many features that allow it to grip an object securely without damaging the object. And you also have a decent sense for how to move your hand to an object without damaging the object -- or your hand -- though this problem is far from being solved in humans too. Ever stub your toe?
> That's an assumption. The contrary view is that language is just part of the I/O system, not where the hard work gets done.
> Animals with very limited language capabilities have the ability to run their own lives, and to some extent manipulate their environment. Systems built from large language databases don't have that.
I'm tempted to agree with this in some capacity, language is probably just one expression of thought, much like how some people have an internal narration going on during their lives, while others think in abstract concepts that aren't necessarily expressed in language.
Here's the first article DuckDuckGo turned up on the subject, i bet someone can share a better one: https://www.iflscience.com/brain/people-with-no-internal-mon...
Furthermore, there are people who don't have the capacity to visualize things in their "mind's eye" either (aphantasia), which is another curious thing: https://www.sciencefocus.com/the-human-body/aphantasia-life-...
Of course, when you get into it, there are also some niche theories about how language could not only help us express ourselves, but shape our thought (not that dissimilar from the "newspeak" concept in Orwell's 1984): https://en.wikipedia.org/wiki/Linguistic_relativity
> Animals with very limited language capabilities have the ability to run their own lives, and to some extent manipulate their environment. Systems built from large language databases don't have that.
I'm tempted to agree with this in some capacity, language is probably just one expression of thought, much like how some people have an internal narration going on during their lives, while others think in abstract concepts that aren't necessarily expressed in language.
Here's the first article DuckDuckGo turned up on the subject, i bet someone can share a better one: https://www.iflscience.com/brain/people-with-no-internal-mon...
Furthermore, there are people who don't have the capacity to visualize things in their "mind's eye" either (aphantasia), which is another curious thing: https://www.sciencefocus.com/the-human-body/aphantasia-life-...
Of course, when you get into it, there are also some niche theories about how language could not only help us express ourselves, but shape our thought (not that dissimilar from the "newspeak" concept in Orwell's 1984): https://en.wikipedia.org/wiki/Linguistic_relativity
I've had aphantasia all my life, although my dreams are still visual, as though they're real.
The article claims it is more common among academics and computer scientists, so perhaps it's fairly common here.
Perhaps it's helpful for dealing with abstraction, as abstractions don't always map well to visual representations.
Perhaps it's helpful for dealing with abstraction, as abstractions don't always map well to visual representations.
When talking about language (I see in the comments below), I think it is useful to differentiate between the potential capabilities of human languages (i.e., all baby humans seem to be able to learn to adult native level of proficiency any human language, regardless of were where their parents from) and the possible effect that using a particular language might have in the speaker or in the people receiving speech in that particular language, i.e., the expressive capabilities of that individual in that language or of the vocabulary of that language specifically (e.g., crafting a joke: how do you make an egg laugh? Tell him a yoke).
What I mean is that language ability of humans and language ability of animals are not comparable; they are so different that in fact, language ability (regardless of the externalization of that language, by speech, text, signs, touch), looks very much like intelligence (meaning the intrinsic ability for language production and not the specific proficiency at some sort of externalization).
If you happen to spend enough time around animals and child humans you will quickly see that after a couple of years, kids can understand and use recursive / referential structures in a way that pets cannot. Pets seem to understand individual words, tones, and maybe "not". Think of "bring the comfy ones that your grandparents bought you to keep your feet warm" - a sentence a 3 year old can perfectly understand. No matter how much you spend teaching/training a dog that has lived in the same environment since its birth, the dog will not react to it bringing you the yellow slippers (unless you specifically trained the dog to associate the word "comfy" or "feet" to some specific slippers). The current state of parametric language models resemble much more that of a very memorious dog than that of a human.
What I mean is that language ability of humans and language ability of animals are not comparable; they are so different that in fact, language ability (regardless of the externalization of that language, by speech, text, signs, touch), looks very much like intelligence (meaning the intrinsic ability for language production and not the specific proficiency at some sort of externalization).
If you happen to spend enough time around animals and child humans you will quickly see that after a couple of years, kids can understand and use recursive / referential structures in a way that pets cannot. Pets seem to understand individual words, tones, and maybe "not". Think of "bring the comfy ones that your grandparents bought you to keep your feet warm" - a sentence a 3 year old can perfectly understand. No matter how much you spend teaching/training a dog that has lived in the same environment since its birth, the dog will not react to it bringing you the yellow slippers (unless you specifically trained the dog to associate the word "comfy" or "feet" to some specific slippers). The current state of parametric language models resemble much more that of a very memorious dog than that of a human.
> That's an assumption. The contrary view is that language is just part of the I/O system, not where the hard work gets done.
FYI, this is the [Sapir-Whorf hypothesis](1)
> We lack some fundamental understanding about how to interact with the real world. The problem is not on the language side.
This is to a significant extent a sensing issue, not a understanding issue.
1: https://en.wikipedia.org/wiki/Linguistic_relativity
FYI, this is the [Sapir-Whorf hypothesis](1)
> We lack some fundamental understanding about how to interact with the real world. The problem is not on the language side.
This is to a significant extent a sensing issue, not a understanding issue.
1: https://en.wikipedia.org/wiki/Linguistic_relativity
The Sappir-Whorf hypothesis refers to specific languages, and is all but proven wrong in its strong form (a person speaking English and a person speaking Japanese seem to perform exactly the same at all cognitive tasks).
But there is a different theory, due to Noam Chomsky I believe, that language in the general sense is in fact our mechanism for higher thought, with communication being just a secondary function[0]. I believe that this is the theory that the GP was combatting with the animal example.
[0] https://www.youtube.com/watch?v=KEmpRtj34xg
But there is a different theory, due to Noam Chomsky I believe, that language in the general sense is in fact our mechanism for higher thought, with communication being just a secondary function[0]. I believe that this is the theory that the GP was combatting with the animal example.
[0] https://www.youtube.com/watch?v=KEmpRtj34xg
> The Sappir-Whorf hypothesis refers to specific languages, and is all but proven wrong in its strong form (a person speaking English and a person speaking Japanese seem to perform exactly the same at all cognitive tasks).
I believe it's a bit more complicated than this.
https://en.wikipedia.org/wiki/Referent
> In fields such as semantics, semiotics, and the theory of reference, a distinction is made between a referent and a reference. Reference is a relationship in which a symbol or sign (a word, for example) signifies something; the referent is the thing signified. The referent may be an actual person or object, or may be something more abstract, such as a set of actions.
> Reference and referents were considered at length in the 1923 book The Meaning of Meaning by the Cambridge scholars C. K. Ogden and I. A. Richards. Ogden has pointed out that reference is a psychological process, and that referents themselves may be psychological – existing in the imagination of the referrer, and not necessarily in the real world. For further ideas related to this observation, see failure to refer.
In this case, what has been ~"proven wrong" is Whorf's specific hypothesis (the reference) of the underlying phenomenon (the referent) that he was observing, whereas the general notion of whether language fundamentally/substantially affects cognition seems to remain unknown.
As a thought experiment: imagine a culture where this psychological phenomenon of "referent" is a "first class" concept in a language, and the culture that uses it insists that the distinction must be at least taken into consideration whenever a disagreement arises between two or more people....after several decades under such a model, might substantial differences arise? Perhaps this isn't a "proper" example of Whorfism, but it seems to be in the neighbourhood.
I believe it's a bit more complicated than this.
https://en.wikipedia.org/wiki/Referent
> In fields such as semantics, semiotics, and the theory of reference, a distinction is made between a referent and a reference. Reference is a relationship in which a symbol or sign (a word, for example) signifies something; the referent is the thing signified. The referent may be an actual person or object, or may be something more abstract, such as a set of actions.
> Reference and referents were considered at length in the 1923 book The Meaning of Meaning by the Cambridge scholars C. K. Ogden and I. A. Richards. Ogden has pointed out that reference is a psychological process, and that referents themselves may be psychological – existing in the imagination of the referrer, and not necessarily in the real world. For further ideas related to this observation, see failure to refer.
In this case, what has been ~"proven wrong" is Whorf's specific hypothesis (the reference) of the underlying phenomenon (the referent) that he was observing, whereas the general notion of whether language fundamentally/substantially affects cognition seems to remain unknown.
As a thought experiment: imagine a culture where this psychological phenomenon of "referent" is a "first class" concept in a language, and the culture that uses it insists that the distinction must be at least taken into consideration whenever a disagreement arises between two or more people....after several decades under such a model, might substantial differences arise? Perhaps this isn't a "proper" example of Whorfism, but it seems to be in the neighbourhood.
Well, I would say that in your thought experiment it's actually the culture that could produce a substantial difference in cognition, whereas the langauge is merely a consequence of that culture.
For example, there are some Australian langauges if I remember correctly where instead of words for left, right, ahead, behind (relative positioning) they use words for something similar to north/south/east/west (absolute positioning) - as in, "give me that cup to your northeast". This of course requires a habit of keeping track of your absolute positioning at all times.
However, you will not gain this ability or habit just by learning the langauge. In fact, your langauge can even express that same thing today. Instead, you will be unable to naturally use their langauge unless and until you get into the habit of keeping track of your absolute positioning.
Similarly, the people growing up in this culture may not easily learn what we mean by left and right, but they would not gain the ability to understand these concepts merely by learning English - instead, they would naturally speak English while avoiding left/right references, or simply make more mistakes than someone growing in a culture where relative positioning is the preferred form of communication.
For example, there are some Australian langauges if I remember correctly where instead of words for left, right, ahead, behind (relative positioning) they use words for something similar to north/south/east/west (absolute positioning) - as in, "give me that cup to your northeast". This of course requires a habit of keeping track of your absolute positioning at all times.
However, you will not gain this ability or habit just by learning the langauge. In fact, your langauge can even express that same thing today. Instead, you will be unable to naturally use their langauge unless and until you get into the habit of keeping track of your absolute positioning.
Similarly, the people growing up in this culture may not easily learn what we mean by left and right, but they would not gain the ability to understand these concepts merely by learning English - instead, they would naturally speak English while avoiding left/right references, or simply make more mistakes than someone growing in a culture where relative positioning is the preferred form of communication.
> Well, I would say that in your thought experiment it's actually the culture that could produce a substantial difference in cognition, whereas the langauge is merely a consequence of that culture.
Regardless of whether the language is a consequence of the culture (say, they imported it, or had it imposed upon them), how might one determine with accuracy that consistently disciplined speaking (which is preceded by cognition) has no causal dependence on speaking this substantially more strict language itself?
> However, you will not gain this ability or habit just by learning the langauge.
You will not necessarily gain this ability just by learning it, agreed - but using it on a daily basis is another matter.
> In fact, your langauge can even express that same thing today.
It can express it, in some fashion (typically with clumsy, annoying phrases and qualifiers that make people angry, rather than dedicated, socially normal words), but whether one actually does it on a regular basis is another matter.
There are various differences between someone who studied 3 months of 1 hour per week of Jiu Jitsu (or math, physics, etc) classes, and someone who practiced it for several hours a day from an early age.
Even in simple scenarios it can be extremely difficult to accurately predict what the consequences of something are, and language, cognition, and culture are not exactly simple. And, from a scientific/epistemic perspective, one should always keep in mind that an absence of evidence is not proof of absence....which brings to mind another thought experiment that is different but similar: what might a society be like where epistemology and logic are considered crucially valuable skills, and were taught from Grade 1 through 12 to all children, and being incorrect or even speaking epistemically unsoundly were considered seriously shameful? Might it be possible that one would notice anything substantially different in people who were raised in such a culture (and, what might be some differences we might see)? I would be quite shocked if it had no effect.
Regardless of whether the language is a consequence of the culture (say, they imported it, or had it imposed upon them), how might one determine with accuracy that consistently disciplined speaking (which is preceded by cognition) has no causal dependence on speaking this substantially more strict language itself?
> However, you will not gain this ability or habit just by learning the langauge.
You will not necessarily gain this ability just by learning it, agreed - but using it on a daily basis is another matter.
> In fact, your langauge can even express that same thing today.
It can express it, in some fashion (typically with clumsy, annoying phrases and qualifiers that make people angry, rather than dedicated, socially normal words), but whether one actually does it on a regular basis is another matter.
There are various differences between someone who studied 3 months of 1 hour per week of Jiu Jitsu (or math, physics, etc) classes, and someone who practiced it for several hours a day from an early age.
Even in simple scenarios it can be extremely difficult to accurately predict what the consequences of something are, and language, cognition, and culture are not exactly simple. And, from a scientific/epistemic perspective, one should always keep in mind that an absence of evidence is not proof of absence....which brings to mind another thought experiment that is different but similar: what might a society be like where epistemology and logic are considered crucially valuable skills, and were taught from Grade 1 through 12 to all children, and being incorrect or even speaking epistemically unsoundly were considered seriously shameful? Might it be possible that one would notice anything substantially different in people who were raised in such a culture (and, what might be some differences we might see)? I would be quite shocked if it had no effect.
> The Sapir-Whorf hypothesis refers to specific languages, and is all but proven wrong in its strong form (a person speaking English and a person speaking Japanese seem to perform exactly the same at all cognitive tasks).
Japanese people show extremely high performance at "spatial awareness" tasks compared to whites.[1] So on average this isn't true. But it's true that there does not appear to be any benefit from either language -- Japanese-speaking Japanese show no advantage over English-speaking Japanese.
[1] On white-normed IQ tests, East Asians generally show a bit of underperformance (relative to European whites) on verbal tasks and gigantic overperformance on spatial tasks, which is traditionally averaged out to an average IQ score moderately above that of whites. Interestingly, Native Americans show the same slant -- much much stronger spatial than verbal performance -- but at a lower base level.
Japanese people show extremely high performance at "spatial awareness" tasks compared to whites.[1] So on average this isn't true. But it's true that there does not appear to be any benefit from either language -- Japanese-speaking Japanese show no advantage over English-speaking Japanese.
[1] On white-normed IQ tests, East Asians generally show a bit of underperformance (relative to European whites) on verbal tasks and gigantic overperformance on spatial tasks, which is traditionally averaged out to an average IQ score moderately above that of whites. Interestingly, Native Americans show the same slant -- much much stronger spatial than verbal performance -- but at a lower base level.
And that is due to the language and not, say, the writing tradition ?
If we kept the same languages and made japanese people write it massively in 26 half sound symbols and made all of what you call a white european population use 3k+ ideogram drawings for word starting at 3 years old, do you think we d see exactly the same performance ?
I bet you the writing tradition is very influencial when it varies so much and has no relationship with language, especially in Japanese where they frankensteined Chinese symbols over centuries which must require an enormous mental gymnastics and redirect large processing power vs someone who can read and write everything he hears relatively simply?
If we kept the same languages and made japanese people write it massively in 26 half sound symbols and made all of what you call a white european population use 3k+ ideogram drawings for word starting at 3 years old, do you think we d see exactly the same performance ?
I bet you the writing tradition is very influencial when it varies so much and has no relationship with language, especially in Japanese where they frankensteined Chinese symbols over centuries which must require an enormous mental gymnastics and redirect large processing power vs someone who can read and write everything he hears relatively simply?
> And that is due to the language and not, say, the writing tradition ?
No, neither.
> If we kept the same languages and made japanese people write it massively in 26 half sound symbols and made all of what you call a white european population use 3k+ ideogram drawings for word starting at 3 years old, do you think we d see exactly the same performance ?
Yes, the influence of spoken and written language appears to be zero. I mentioned this.
No, neither.
> If we kept the same languages and made japanese people write it massively in 26 half sound symbols and made all of what you call a white european population use 3k+ ideogram drawings for word starting at 3 years old, do you think we d see exactly the same performance ?
Yes, the influence of spoken and written language appears to be zero. I mentioned this.
I interpret what you quoted as the opposite of Sapir-Whorf. Could you explain why they're the same?
I am also quite confused. My understanding (it's been a while since I studied Linguistics) is that strong Sapir-Whorf refers to the argument that what actions your brain can take are constrained by your native language(s). So by strong Sapir-Whorf, the distinction between the French savoir and connaitre should make zero sense to me since that distinction isn't found in English. Weak Sapir-Whorf says the distinction wouldn't make sense until I learned words for it, but that the brain can learn new concepts if it acquires language for them.
So SW argues that language is inherently intertwined with human cognition, rather like how the parts of the brain are all interconnected and working together as we think. Strong SWism is pretty thoroughly debunked, weak is tentatively accepted on a spectrum.
I think that they might be trying to refer to generative theory versus cognitive theory. Again, it's been a while, so I might be wrong, but:
- Generative theory is Chomsky et al. who believe that there is a single inherent underlying pattern to language that restricts it, and that one could, in theory, figure out all possible syntaxes, for example. Similar to how phonology and phonetics are obviously constrained by our biological makeup (we can only make certain sounds), the argument is that other structures are as well. Basically that although they all seem different to us, languages are more alike than they are different because they come from a single source. Also Chomsky and co. consider this approach cognitive; the linguists are just infighting.
- Cognitive Linguists believe that language, and the systems that produce it, are under evolutionary pressure and that they change and shift in response to the environment.
Basically, if you took 10k people and dumped 5k of them on one planet and 5k on a planet that was the physical opposite, forbid them to contact anyone, and visited 2000 years later, Generativists say you could find a system in common between the two and cognitive linguists say you probably couldn't, because the systems themselves would have evolved to have different starting points.
Chomsky + GG: Person is born, person hears speech, person reaches for inherent language patterns in brain and uses them to understand/learn.
Cognitive: Person is born, person hears speech, person picks up on the system from the speech and is encouraged to do so/pattern match by social reinforcement.
No linguist that I'm aware of would claim that language is just an I/O system.
So SW argues that language is inherently intertwined with human cognition, rather like how the parts of the brain are all interconnected and working together as we think. Strong SWism is pretty thoroughly debunked, weak is tentatively accepted on a spectrum.
I think that they might be trying to refer to generative theory versus cognitive theory. Again, it's been a while, so I might be wrong, but:
- Generative theory is Chomsky et al. who believe that there is a single inherent underlying pattern to language that restricts it, and that one could, in theory, figure out all possible syntaxes, for example. Similar to how phonology and phonetics are obviously constrained by our biological makeup (we can only make certain sounds), the argument is that other structures are as well. Basically that although they all seem different to us, languages are more alike than they are different because they come from a single source. Also Chomsky and co. consider this approach cognitive; the linguists are just infighting.
- Cognitive Linguists believe that language, and the systems that produce it, are under evolutionary pressure and that they change and shift in response to the environment.
Basically, if you took 10k people and dumped 5k of them on one planet and 5k on a planet that was the physical opposite, forbid them to contact anyone, and visited 2000 years later, Generativists say you could find a system in common between the two and cognitive linguists say you probably couldn't, because the systems themselves would have evolved to have different starting points.
Chomsky + GG: Person is born, person hears speech, person reaches for inherent language patterns in brain and uses them to understand/learn.
Cognitive: Person is born, person hears speech, person picks up on the system from the speech and is encouraged to do so/pattern match by social reinforcement.
No linguist that I'm aware of would claim that language is just an I/O system.
> Basically, if you took 10k people and dumped 5k of them on one planet and 5k on a planet that was the physical opposite, forbid them to contact anyone, and visited 2000 years later, Generativists say you could find a system in common between the two and cognitive linguists say you probably couldn't, because the systems themselves would have evolved to have different starting points.
Is this not a bit of a false dichotomy (substantially dimensionally compressed representation of the true underlying complexity)? Like sure, one is likely to be able to find substantial commonality, but does cognitive linguistics really unequivocally assert that you will find no commonality whatsoever? If so, that seems a bit crazy.
Is this not a bit of a false dichotomy (substantially dimensionally compressed representation of the true underlying complexity)? Like sure, one is likely to be able to find substantial commonality, but does cognitive linguistics really unequivocally assert that you will find no commonality whatsoever? If so, that seems a bit crazy.
Your understanding is also mine. Great summary.
> No linguist that I'm aware of would claim that language is just an I/O system.
I think you're right. In Chomskyian linguistics there is a distinction between i-language and e-language (https://en.wikipedia.org/wiki/Transformational_grammar#%22I-...). E-language isn't "just I/O" but it's perhaps not far from that whereas i-language is taken as the proper subject for linguists to study.
> No linguist that I'm aware of would claim that language is just an I/O system.
I think you're right. In Chomskyian linguistics there is a distinction between i-language and e-language (https://en.wikipedia.org/wiki/Transformational_grammar#%22I-...). E-language isn't "just I/O" but it's perhaps not far from that whereas i-language is taken as the proper subject for linguists to study.
I like how you've put it.
I have an objection to the idea that understanding is just a pattern recognition, or prediction of the next token. I think fundamental to understanding is the concept of logical contradiction, which I don't believe GPT-3 has. I don't think GPT-3 can say, wait, what I am saying is BS, because logically it's not consistent.
To argue that, I propose a "simplified Chinese room". We have an automaton in the Chinese room, which does only pattern recognition, and can learn by matching patterns and structures, i.e. it will give an answer that is similar to what it has seen before. And we want to teach it to answer a problem "recognize whether the given logical formula is satisfiable". So we give it formulas and it tries to pattern match the formulas it already has seen. Unfortunately, there always exists a formula which is matching closely everything that was shown so far, and yet the correct answer is unexpected. So such an automaton will never be able to discover the actual rule.
What I think has to be added are different levels of discourse, or, a discourse about discourse. With people, we can say, "stop joking around", and it's a meta-discourse signal that we want stop discussing imaginary worlds and focus on the real one, for example, which has to be totally self-consistent. I believe an analogy can be made with Futamura projections, we can have not only systems that learn a specific task (like even GPT-3 does), but also systems that learn to produce an agent to do a specific task, from a metalanguage description that encodes certain rules on how to do it (for example, to follow logical consistency). And if the metalanguage is general enough, you get general intelligence, because then you can apply the agent production process to itself.
To argue that, I propose a "simplified Chinese room". We have an automaton in the Chinese room, which does only pattern recognition, and can learn by matching patterns and structures, i.e. it will give an answer that is similar to what it has seen before. And we want to teach it to answer a problem "recognize whether the given logical formula is satisfiable". So we give it formulas and it tries to pattern match the formulas it already has seen. Unfortunately, there always exists a formula which is matching closely everything that was shown so far, and yet the correct answer is unexpected. So such an automaton will never be able to discover the actual rule.
What I think has to be added are different levels of discourse, or, a discourse about discourse. With people, we can say, "stop joking around", and it's a meta-discourse signal that we want stop discussing imaginary worlds and focus on the real one, for example, which has to be totally self-consistent. I believe an analogy can be made with Futamura projections, we can have not only systems that learn a specific task (like even GPT-3 does), but also systems that learn to produce an agent to do a specific task, from a metalanguage description that encodes certain rules on how to do it (for example, to follow logical consistency). And if the metalanguage is general enough, you get general intelligence, because then you can apply the agent production process to itself.
> I think fundamental to understanding is the concept of logical contradiction, which I don't believe GPT-3 has
are we sure that humans actually have that though ? It's not always my experience
are we sure that humans actually have that though ? It's not always my experience
I think we do. We get a distinct feeling that we don't understand something, and a different feeling when things click together. Or when we hear a syntactically valid (in other words, pattern-matching) but nonsensical sentence (like "colorless green ideas sleep furiously"), or see a Penrose triangle, and we cannot imagine an example, we get a very specific feeling that something is wrong. I think we are actually solving the satisfiability problem every time we imagine something - we find a concrete instance of a situation that satisfies certain conditions (for example, words in a sentence, or pixels in the image).
The book Inside Jokes even speculates that reason why we find jokes pleasurable is that they are actually a learning mechanism to detect logical inconsistencies and further our understanding of the world.
With GPT-3, you can't tell whether it's joking or not, and whether it's aware if it's joking or not. It doesn't (to my knowledge) have that boundary where it could say, wait, I don't understand what we are saying here. Maybe it's easy to add to the system, but as it stands, I don't think GPT-3 does the above.
The book Inside Jokes even speculates that reason why we find jokes pleasurable is that they are actually a learning mechanism to detect logical inconsistencies and further our understanding of the world.
With GPT-3, you can't tell whether it's joking or not, and whether it's aware if it's joking or not. It doesn't (to my knowledge) have that boundary where it could say, wait, I don't understand what we are saying here. Maybe it's easy to add to the system, but as it stands, I don't think GPT-3 does the above.
> With people, we can say, "stop joking around", and it's a meta-discourse signal that we want stop discussing imaginary worlds and focus on the real one, for example, which has to be totally self-consistent.
-
> We get a distinct feeling that we don't understand something, and a different feeling when things click together. Or when we hear a syntactically valid (in other words, pattern-matching) but nonsensical sentence (like "colorless green ideas sleep furiously"), or see a Penrose triangle, and we cannot imagine an example, we get a very specific feeling that something is wrong.
It's true that the human mind has the capability to do these things sometimes (and within certain specifically designed frameworks like the scientific method, do it highly reliably)....but whether we do do it (or even have the capability) with consistent, high level success across various domains is a very different matter.
-
> We get a distinct feeling that we don't understand something, and a different feeling when things click together. Or when we hear a syntactically valid (in other words, pattern-matching) but nonsensical sentence (like "colorless green ideas sleep furiously"), or see a Penrose triangle, and we cannot imagine an example, we get a very specific feeling that something is wrong.
It's true that the human mind has the capability to do these things sometimes (and within certain specifically designed frameworks like the scientific method, do it highly reliably)....but whether we do do it (or even have the capability) with consistent, high level success across various domains is a very different matter.
> With GPT-3, you can't tell whether it's joking or not, and whether it's aware if it's joking or not
But you can't genuinely know that for humans either. At best you can they that they look like they are joking, and you definitely cannot say that they in fact are, even if they say so to your face.
But you can't genuinely know that for humans either. At best you can they that they look like they are joking, and you definitely cannot say that they in fact are, even if they say so to your face.
Well, I think you sorta can. Your boss can say, if you don't stop joking around, and don't follow the rules, I will shoot you. We have an evolutionary predisposition to avoid pain and death and the whole social structure on top of that, and this ensures that sometimes we just have to be honest.
Which is one of the reason why I think we need at least two levels of learning for general intelligence. The bottom learner is the one which learns the actual task at hand, and the top learner is tasked (with learning) to, given a description of a task, produce (train if needed) a bottom learner that can solve that task. This solves the ambiguity, because if we are confident enough that the top learner can produce correct bottom learners for different descriptions, then adding another rule to the description will likely cause the bottom learner (produced by the top learner) to also obey that rule.
It also resolves the understanding problem, because while no learner might be sure if it is itself correct, the top learner has to somehow evaluate the bottom learner, and so the top learner sees where bottom learner is failing to understand (which rules are problematic). What's even more interesting, if the language to express the problems is rich enough, then we can apply the top learner to itself as the bottom learner (express the actual learning setup from within the language), and we get sort of a "philosopher", entity that tries to learn how best to build learners. (This is an interesting analogy you can make with Futamura projection, by thinking about "learners" instead of "programs", and what happens if learners are tasked to produce another learner. The reason why we can be somewhat confident in "correctness" of the bottom learners is similar to reason why we can be somewhat confident that compiled program does what we want, despite it being completely new to the compiler.)
There are multiple ways to map this top/bottom learners onto humans. You can think of the bottom learner as the actual intuitive systems that we have, and the top learner as the conscious system, which drives what we are gonna learn and how. Or the bottom learner as the system that contains the model of the environment, and the top learner as a system that provides episodic learning via some kind of simulation (our dreams, imagination) to it, and reflects on it. Or the bottom learners are individual humans, while top learners are the genes that drive the evolution of the human brain. In practice, there might be more than two levels.
Which is one of the reason why I think we need at least two levels of learning for general intelligence. The bottom learner is the one which learns the actual task at hand, and the top learner is tasked (with learning) to, given a description of a task, produce (train if needed) a bottom learner that can solve that task. This solves the ambiguity, because if we are confident enough that the top learner can produce correct bottom learners for different descriptions, then adding another rule to the description will likely cause the bottom learner (produced by the top learner) to also obey that rule.
It also resolves the understanding problem, because while no learner might be sure if it is itself correct, the top learner has to somehow evaluate the bottom learner, and so the top learner sees where bottom learner is failing to understand (which rules are problematic). What's even more interesting, if the language to express the problems is rich enough, then we can apply the top learner to itself as the bottom learner (express the actual learning setup from within the language), and we get sort of a "philosopher", entity that tries to learn how best to build learners. (This is an interesting analogy you can make with Futamura projection, by thinking about "learners" instead of "programs", and what happens if learners are tasked to produce another learner. The reason why we can be somewhat confident in "correctness" of the bottom learners is similar to reason why we can be somewhat confident that compiled program does what we want, despite it being completely new to the compiler.)
There are multiple ways to map this top/bottom learners onto humans. You can think of the bottom learner as the actual intuitive systems that we have, and the top learner as the conscious system, which drives what we are gonna learn and how. Or the bottom learner as the system that contains the model of the environment, and the top learner as a system that provides episodic learning via some kind of simulation (our dreams, imagination) to it, and reflects on it. Or the bottom learners are individual humans, while top learners are the genes that drive the evolution of the human brain. In practice, there might be more than two levels.
> prediction of the next token
Assessing logical consistency of a following statement would increase prediction accuracy (assuming that human-generated text is more consistent than it is not), but only after you have plucked all lower-hanging fruit. The best model of what comes next in a math paper is a mathematician.
So GPT-N not trying to does not necessarily mean GPT-N+1 can't do it, it just means that it has cheaper ways to increase its prediction scores.
Assessing logical consistency of a following statement would increase prediction accuracy (assuming that human-generated text is more consistent than it is not), but only after you have plucked all lower-hanging fruit. The best model of what comes next in a math paper is a mathematician.
So GPT-N not trying to does not necessarily mean GPT-N+1 can't do it, it just means that it has cheaper ways to increase its prediction scores.
This raises a question whether the best prediction should be a measurement of intelligence. GPT-3 might well be capable of super-intelligence, for what we know, but if we feed it examples on human level of thought, it will interact with us on human (i.e. stupid) level of thought, because that's what we are teaching it. Just like adults choose to talk to children on their level.
However, a truly intelligent system, I think, can transgress this boundary. It can protest against the task of giving the next best prediction, so to speak.
I don't think that mathematician is necessarily the best model for a math paper; I think the best model might as well be the platonic ideal of what it means to do mathematics - logical correctness of all the conclusions.
However, a truly intelligent system, I think, can transgress this boundary. It can protest against the task of giving the next best prediction, so to speak.
I don't think that mathematician is necessarily the best model for a math paper; I think the best model might as well be the platonic ideal of what it means to do mathematics - logical correctness of all the conclusions.
Unless you're substituting GPT3 for AGI, GPT3 is not a super-intelligence. It's a set number of matrix multiplications that completes in polynomial time. Because Transformer context is fixed, and given the limitations of training, adding non-determinism (tree search) means it goes off the rails and is prone to getting trapped in cycles anytime it doesn't quickly terminate.
Has anyone tried prompting GPT-3 with "the following is a transcript of a conversation with a superintelligent artificial intelligence, in which it solves all of humanity's problems"?
It's easy to dismiss stuff like this without having interacted with these models, but there's something visceral about doing so that at least personally left me re-assessing some of my viewpoints on ML/AI. I've had a chance to interact with the model in the article and it is truly a spooky experience.
I'm looking forward to seeing how people react once they get a chance to try some of these increasingly massive chat models in person. GPT-3 is just the tip of the iceberg.
(views my own blah blah)
I'm looking forward to seeing how people react once they get a chance to try some of these increasingly massive chat models in person. GPT-3 is just the tip of the iceberg.
(views my own blah blah)
Sure, but I can't get over how it's picking the next word literally at random from a probability distribution - unless you turn that off and then it often goes into an infinite loop.
I think it's like song lyrics. We can find meaning in vagueness. Particularly if you retry a few times. High-probability completions are likely to seem meaningful.
I think it's like song lyrics. We can find meaning in vagueness. Particularly if you retry a few times. High-probability completions are likely to seem meaningful.
How do you know you are not also picking the next word at random from a probability distribution?
I mean, as I'm typing this sentence, I have some notion about which word is going to come next, and a sense of where the sentence is going to go.
But maybe that is what it feels like to be a large neural network that has been trained on very large data sets for long periods of time.
I mean, as I'm typing this sentence, I have some notion about which word is going to come next, and a sense of where the sentence is going to go.
But maybe that is what it feels like to be a large neural network that has been trained on very large data sets for long periods of time.
One skill we do have is the ability to realize that what we said is not what we meant, and make a correction. This is more goal-oriented than what a transformer is doing. Its only goal is to continue onward in a plausible way and it could be in any direction depending on a random choice. It doesn't care which way it goes.
[deleted]
[deleted]
Yeah, and I guess that's kind of the point. You can still see the seams in many of these models, but you could easily imagine a version that's truly indistinguishable from a human. Or at least indistinguishable from something we'd consider an "intelligent entity". And in that case, how does what's under the hood affect the "meaning"? Should it?
This isn't a new idea by any means, science fiction has tackled this ad-nauseam. The thing that caused me to have a strong reaction is that previously it's never seemed like a question we'd seriously have to contend with. It's an interesting idea in a nicely wrapped up 2 hour movie, but damn does it seem scary and complicated out here in the real world.
This isn't a new idea by any means, science fiction has tackled this ad-nauseam. The thing that caused me to have a strong reaction is that previously it's never seemed like a question we'd seriously have to contend with. It's an interesting idea in a nicely wrapped up 2 hour movie, but damn does it seem scary and complicated out here in the real world.
I don't think they will get there by scaling this technology. Something more is needed to have an interesting world model. But machine learning research is a hot field and it could be invented any year now.
I feel so strongly that I have a soul that starts with self-aware consciousness and I get such push back on this idea from high tech people, that I'm starting to wonder if perhaps most don't possess a soul and are effectively biorobots. This is also the theme of Kurt Vonnegut's "Breakfast of Champions" in which the author (Vonnegut inserts himself into the novel) is going to kill himself (like his mother) because he feels everyone is a robot except him. He is saved when he meets an artist and senses the soul in the man.
Everyone feels that way, but people hold different perspectives when they scrutinize exactly what that means and how it would work.
There is no evidence of souls existing, they are unable to be measured in any way. Unless they have several magical rules and violate physics (but only sometimes and are still somehow undetectable), then your brain would still have all those same thoughts about how it feels strongly about having a soul, because that's part of how the ego hard-coded into your brain works.
I don't particularly like the view you're espousing, not only because its inconsistent without explanation but because it creates a group of "others" where those with a different philosophical viewpoint are now literally "unfeeling robots" to you.
There is no evidence of souls existing, they are unable to be measured in any way. Unless they have several magical rules and violate physics (but only sometimes and are still somehow undetectable), then your brain would still have all those same thoughts about how it feels strongly about having a soul, because that's part of how the ego hard-coded into your brain works.
I don't particularly like the view you're espousing, not only because its inconsistent without explanation but because it creates a group of "others" where those with a different philosophical viewpoint are now literally "unfeeling robots" to you.
> There is no evidence of souls existing, they are unable to be measured in any way.
This has a dependency on what is considered valid evidence, as well as our current ability to measure things.
> Unless they have several magical rules and violate physics (but only sometimes and are still somehow undetectable), then your brain would still have all those same thoughts about how it feels strongly about having a soul, because that's part of how the ego hard-coded into your brain works.
The same phenomenon applies to reality itself, including our perception of the state and behavior of what is inside of it, as compared to what is actually there (the question of whether souls exist is subject to this problem).
A somewhat relevant post from SSC the other day: https://astralcodexten.substack.com/p/the-phrase-no-evidence...
This has a dependency on what is considered valid evidence, as well as our current ability to measure things.
> Unless they have several magical rules and violate physics (but only sometimes and are still somehow undetectable), then your brain would still have all those same thoughts about how it feels strongly about having a soul, because that's part of how the ego hard-coded into your brain works.
The same phenomenon applies to reality itself, including our perception of the state and behavior of what is inside of it, as compared to what is actually there (the question of whether souls exist is subject to this problem).
A somewhat relevant post from SSC the other day: https://astralcodexten.substack.com/p/the-phrase-no-evidence...
I don't actually believe that. I was referring to a work of fiction that is philosophically meaningful to me.
What I do believe is there are components to intelligence that includes emotional intelligence, self-awareness and memory that some people don't seem to have in great quantities, but I don't think they are robots.
What I do believe is there are components to intelligence that includes emotional intelligence, self-awareness and memory that some people don't seem to have in great quantities, but I don't think they are robots.
My mistake for misunderstanding, I suppose. But you initially said that you (1) were beginning to think it, then said it is also (2) the theme of your literature:
> I'm (1) starting to wonder if perhaps most don't possess a soul and are effectively biorobots. This is also (2) the theme...
> I'm (1) starting to wonder if perhaps most don't possess a soul and are effectively biorobots. This is also (2) the theme...
I'm sorry. That was an awkward way of saying it that I didn't notice until you pointed it out.
No need to apologize, I hope you have a fantastic evening!
> There’s something fundamentally unanswerable about the question “What are the minimum requirements for personhood?”, or more colloquially, “When does an ‘it’ become a ‘who’?”
This is the crux of the question, and I'd argue the abstraction and generalization of person and personhood itself is not only flawed, but the base error that diverts the course and trajectory of our ability to reason about ethics.
It's perhaps easier to meander around with analogies, contrasts, and comparisons than to delve down into that harder question, and as long as we're circumspect about it, philosophically we're just treading water.
This is the crux of the question, and I'd argue the abstraction and generalization of person and personhood itself is not only flawed, but the base error that diverts the course and trajectory of our ability to reason about ethics.
It's perhaps easier to meander around with analogies, contrasts, and comparisons than to delve down into that harder question, and as long as we're circumspect about it, philosophically we're just treading water.
[deleted]
> Since the interior state of another being can only be understood through interaction, no objective answer is possible to the question of when an “it” becomes a “who”
A large language model can in theory be understood at an algorithmic level by reverse engineering. If the algorithm turned out to be a giant lookup table, it's an "it". If the algorithm contained an obvious model of self, it's a "who".
A large language model can in theory be understood at an algorithmic level by reverse engineering. If the algorithm turned out to be a giant lookup table, it's an "it". If the algorithm contained an obvious model of self, it's a "who".
We assume other people are "who" and not "it" because their behavior is predictably similar to our own, and so we model their experiences and perspectives based on our own subjective experience.
Occam's razor suggests that if we know other people have the same basic configuration as ourselves, then the subjective experiences of others will be more or less comparable to our own.
Additional assumptions are necessary to propose that Chalmersian/philosophical zombies could exist, having the same human hardware and behavior but lacking subjective experience. Under the principle of least complexity and the absence of evidence for alternate explanations, I think it's rational to assume the consciousness of other humans and certain animals with nearly 100% confidence.
That means something happening in the neocortex is causing consciousness. We know it's in the cortex because of the history of injury or absence of other parts of the brain, leaving us with empirical evidence. We know that hippocampus injury can result in a person losing the ability to remember more then 5 minutes of their past. We also know these individuals retain their personalities and the long-term memory from before their injury. This suggests that long term memory is encoded in the synaptic structure of the neocortex and that consciousness is an emergent result of neocortical operation.
There might be some involvement of particular brain regions or the thalamus or other organ, but it looks like the answer is part of whatever algorithm is encoded in the structure and processing of the neocortex.
It could be possible that consciousness doesn't require an explicit model of self. This is implied by ego death and other experiences by meditators and psychedelic users. The thing that is having a subjective experience could simply be a consequence of processing a particular configuration of information. The self concept seems to be a separate model that can be perceived as part of the process of awareness, but it seems to be a discrete thing.
To me that presents an interesting ethical question - if gpt-3 has a sense of self, then is it OK to subject it to the incoherent flashes of single moments of consciousness it undergoes each time you run it?
If it has no self concept, it might still have subjective experience, but would have no persistent contiguous experience . If it did have a self concept, then it would have a static past that informed the results of each run, as if you could isolate a single moment of awareness, then instantly reset the brain to a saved state and run it again. The output could be convincingly continuous, but the subjective experiences would be a myriad of similar but unrelated singularities of awareness.
I think gpt-3 lacks consciousness, and that a persistence mechanism is missing that plays a part in whatever is happening that causes awareness.
Hopefully, at some point, a m mathematician or scientist will be able to identify and explain the process of consciousness so that we can be reasonably certain we're not subjecting entities to a really weird tortured existence.
Occam's razor suggests that if we know other people have the same basic configuration as ourselves, then the subjective experiences of others will be more or less comparable to our own.
Additional assumptions are necessary to propose that Chalmersian/philosophical zombies could exist, having the same human hardware and behavior but lacking subjective experience. Under the principle of least complexity and the absence of evidence for alternate explanations, I think it's rational to assume the consciousness of other humans and certain animals with nearly 100% confidence.
That means something happening in the neocortex is causing consciousness. We know it's in the cortex because of the history of injury or absence of other parts of the brain, leaving us with empirical evidence. We know that hippocampus injury can result in a person losing the ability to remember more then 5 minutes of their past. We also know these individuals retain their personalities and the long-term memory from before their injury. This suggests that long term memory is encoded in the synaptic structure of the neocortex and that consciousness is an emergent result of neocortical operation.
There might be some involvement of particular brain regions or the thalamus or other organ, but it looks like the answer is part of whatever algorithm is encoded in the structure and processing of the neocortex.
It could be possible that consciousness doesn't require an explicit model of self. This is implied by ego death and other experiences by meditators and psychedelic users. The thing that is having a subjective experience could simply be a consequence of processing a particular configuration of information. The self concept seems to be a separate model that can be perceived as part of the process of awareness, but it seems to be a discrete thing.
To me that presents an interesting ethical question - if gpt-3 has a sense of self, then is it OK to subject it to the incoherent flashes of single moments of consciousness it undergoes each time you run it?
If it has no self concept, it might still have subjective experience, but would have no persistent contiguous experience . If it did have a self concept, then it would have a static past that informed the results of each run, as if you could isolate a single moment of awareness, then instantly reset the brain to a saved state and run it again. The output could be convincingly continuous, but the subjective experiences would be a myriad of similar but unrelated singularities of awareness.
I think gpt-3 lacks consciousness, and that a persistence mechanism is missing that plays a part in whatever is happening that causes awareness.
Hopefully, at some point, a m mathematician or scientist will be able to identify and explain the process of consciousness so that we can be reasonably certain we're not subjecting entities to a really weird tortured existence.
> If the algorithm turned out to be a giant lookup table, it's an "it". If the algorithm contained an obvious model of self, it's a "who".
It seems to be that (at least a naive understanding of those two possibilities) leaves a large middle ground where an language model isn't a lookup table but doesn't have a model of self.
It seems to be that (at least a naive understanding of those two possibilities) leaves a large middle ground where an language model isn't a lookup table but doesn't have a model of self.
[deleted]
I'm surprised by no mention of the Chinese room argument: https://en.wikipedia.org/wiki/Chinese_room - it's closely related to the topic.
Searle goes into the room with the copy of the program instructions. He gets passed a note written in Chinese, cracks his knuckles, and then opens the manual.
He reads, "Step-1: prior to translating the note, we are going to train your neural networks on a Chinese corpus consisting of 410 billion tokens from Common Crawl, 19 billion tokens from WebText, 67 billion tokens from Books, and 3 billion tokens from Wikipedia. Training in 5, 4, 3..."
He shouts, "wait but if you do that wo shuō zhōngwén."
He reads, "Step-1: prior to translating the note, we are going to train your neural networks on a Chinese corpus consisting of 410 billion tokens from Common Crawl, 19 billion tokens from WebText, 67 billion tokens from Books, and 3 billion tokens from Wikipedia. Training in 5, 4, 3..."
He shouts, "wait but if you do that wo shuō zhōngwén."
The search result I'm finding for that phrase is "Wǒ bù huì shuō zhōngwén. I kept saying it over and over again, the phrase I had memorized to explain my inability to speak their language".
So I guess it means "I don't speak Mandarin"?
So I guess it means "I don't speak Mandarin"?
The "Wǒ bù huì[...]" part means "I negate/not can[...]"
It means precisely the opposite.
It means precisely the opposite.
The Chinese room argument is very unconvincing unfortunately, as brains are actually biological computing machines.
Instead of the computer/book telling the non-Chinese speaker what to write, what if it’s a human that speaks Chinese?
Does it matter then that the writer has no understanding of the language, when the backend clearly does?
(I think I get it actually, after writing these two questions, but I’ll post them anyway.)
Does it matter then that the writer has no understanding of the language, when the backend clearly does?
(I think I get it actually, after writing these two questions, but I’ll post them anyway.)
I think at "the writer has no understanding of the language, when the backend clearly does?" you circle back to the argument's main problem - is it possible for the non-human backend to "understand" the language, even though it behaves like it does.
This isn't aimed at you, viraptor, but at Searle and his argument.
Searle and subsequent proponents fail to define mind, intelligence, or understanding. The argument is founded in anthropocentric arrogance.
Gpt-3 has convincingly passed the Turing test, with various and sundry incarnations passing as human for days or even weeks on reddit and Twitter and other social media. You can keep moving the goalposts, but the herd has left the barn. The argument is over. The Turing test has been passed.
There is no way to frame human mind, intelligence, or understanding such that you exclude the equivalent functionality in software without engaging in magical thinking.
Searle's assertion that he would not understand Chinese while executing the rules isn't relevant. Searle could conceivably execute the inference pass over a gpt-3 model based on Chinese, and produce results that surpass the Turing threshold as conceived by Searle at the time. The intelligence and understanding and mind is encoded in the model and logic of the processing.
This demonstrates that substrate is not involved in the process of intelligence. There's nothing unique or magical about the human brain that suggests a shred of evidence that it can't be replicated by any Turing complete processing system, or computer. Our minds are an algorithm being run continuously by a messy wet computer in a bone vat connected to a meat suit.
Your sense of continuous perception is the state of a model stored in and processed by the physical, electrical, and chemical configuration of your brain and the laws of physics.
Any methodical definition of understanding arrives at logic and semantic relationships between concepts and information theory. The brain has to be operating as a computer because there is literally nothing else that could be reasonably asserted.
Understanding and consciousness and intelligence and mind are consequences of algorithms, or software. Gpt-3 and similar models are undeniably different types of minds then that of humans, lacking persistent state, online feedback, and a myriad of systems and capabilities. Transformers in their current form are not agi and interact with the world in a radically different way than humans do. They do something that human minds do, however.
Consciousness could be the result of a certain class of algorithms being processed, with a self referential model processed concurrent with external input. Additional systems might be required, but consciousness has to be accomplished using no more than the capabilities offered by the brain, so it's likely to be an emergent consequence of the holistic execution of the cortical algorithm.
Each inference pass might represent a single moment of mind, a disjoint and ephemeral flash of consciousness, unrelated to its prior or future states, deterministically replicable and static.
The Chinese room is a failed argument and we should move on to more interesting things. Brain meat is not magic.
Searle and subsequent proponents fail to define mind, intelligence, or understanding. The argument is founded in anthropocentric arrogance.
Gpt-3 has convincingly passed the Turing test, with various and sundry incarnations passing as human for days or even weeks on reddit and Twitter and other social media. You can keep moving the goalposts, but the herd has left the barn. The argument is over. The Turing test has been passed.
There is no way to frame human mind, intelligence, or understanding such that you exclude the equivalent functionality in software without engaging in magical thinking.
Searle's assertion that he would not understand Chinese while executing the rules isn't relevant. Searle could conceivably execute the inference pass over a gpt-3 model based on Chinese, and produce results that surpass the Turing threshold as conceived by Searle at the time. The intelligence and understanding and mind is encoded in the model and logic of the processing.
This demonstrates that substrate is not involved in the process of intelligence. There's nothing unique or magical about the human brain that suggests a shred of evidence that it can't be replicated by any Turing complete processing system, or computer. Our minds are an algorithm being run continuously by a messy wet computer in a bone vat connected to a meat suit.
Your sense of continuous perception is the state of a model stored in and processed by the physical, electrical, and chemical configuration of your brain and the laws of physics.
Any methodical definition of understanding arrives at logic and semantic relationships between concepts and information theory. The brain has to be operating as a computer because there is literally nothing else that could be reasonably asserted.
Understanding and consciousness and intelligence and mind are consequences of algorithms, or software. Gpt-3 and similar models are undeniably different types of minds then that of humans, lacking persistent state, online feedback, and a myriad of systems and capabilities. Transformers in their current form are not agi and interact with the world in a radically different way than humans do. They do something that human minds do, however.
Consciousness could be the result of a certain class of algorithms being processed, with a self referential model processed concurrent with external input. Additional systems might be required, but consciousness has to be accomplished using no more than the capabilities offered by the brain, so it's likely to be an emergent consequence of the holistic execution of the cortical algorithm.
Each inference pass might represent a single moment of mind, a disjoint and ephemeral flash of consciousness, unrelated to its prior or future states, deterministically replicable and static.
The Chinese room is a failed argument and we should move on to more interesting things. Brain meat is not magic.
> Gpt-3 has convincingly passed the Turing test
Has it? I can't find a convincing source for that. Note that twitter and reddit don't really count, the Turing test always was a conversation, not limited to bite-sized soundbites like those. You can make probably make a decent Reddit bot by selecting random old popular posts from that sub, and that would never pass the Turing test.
The only interaction I've had with GPT-3 is AI Dungeon, and it's very easy to discover that's not a human.
Has it? I can't find a convincing source for that. Note that twitter and reddit don't really count, the Turing test always was a conversation, not limited to bite-sized soundbites like those. You can make probably make a decent Reddit bot by selecting random old popular posts from that sub, and that would never pass the Turing test.
The only interaction I've had with GPT-3 is AI Dungeon, and it's very easy to discover that's not a human.
Ask them, why would they lie?
Trolling by some google AI honcho. What's the world coming to?
I think this would be like asking if calculators understand math.
Do submarines swim?
Do we understand large language models?
Obviously not.
That's an assumption. The contrary view is that language is just part of the I/O system, not where the hard work gets done.
Animals with very limited language capabilities have the ability to run their own lives, and to some extent manipulate their environment. Systems built from large language databases don't have that.
After 50 years, robotic manipulation in unstructured situations is still terrible. Rod Brooks tried to crack that and his startup, Rethink Robotics, failed. DARPA has had decades of manipulation projects. Fifty years ago, they could put a bolt in a hole. Five years ago, they got a robot hand to put a key in a lock. Amazon is still trying to get robots to pick things from bins fast and reliably.
We lack some fundamental understanding about how to interact with the real world. The problem is not on the language side.