Text Is All You Need(studio.ribbonfarm.com)
studio.ribbonfarm.com
Text Is All You Need
https://studio.ribbonfarm.com/p/text-is-all-you-need
213 comments
The article totally understands this distinction of fictionality. That's why it defines personhood thusly:
The what is easy. It’s personhood.
By personhood I mean what it takes in an entity
to get another person treat it unironically as a
human, and feel treated as a human in turn. In
shorthand, personhood is the capacity to see and
be seen.
If the argument is that it's very easy to emotionally confuse language models and persons, than I reject that argument on the following grounds:
No works of fiction are persons. All "I" statements from the current generation of LLMs are works of fiction.
Therefore, no "I" statements from the current generation of LLMs are persons.
Premise 1 is in conflict with the author's premise that personhood can be ascribed at will; I'm happy agreeing to disagree on that. I do not think it ever makes sense to ascribe personhood to fictional characters (for any meaningful definition of personhood.)
No works of fiction are persons. All "I" statements from the current generation of LLMs are works of fiction.
Therefore, no "I" statements from the current generation of LLMs are persons.
Premise 1 is in conflict with the author's premise that personhood can be ascribed at will; I'm happy agreeing to disagree on that. I do not think it ever makes sense to ascribe personhood to fictional characters (for any meaningful definition of personhood.)
Honestly, that's a terrible working definition of personhood. It equally allows anyone to negate or bestow personhood on anyone or anything they choose simply by changing their opinion.
Unfortunately when your working with concepts that can't be measured/only exist in the eye of the beholder, any definition you make will have that problem. The only litmus test for "personhood" is if you think they're a person.
That…is exactly what happens in real life
I can't wait for PETA-for-things-we-say-are-persons
But it's not easy at all to get confused, unless one decides to consciously suspend disbelief, in spite of what they know. If they do know how LLMs work. It's much easier to get confused, of course, for someone who doesn't know, because they don't have to actively override that knowledge if it's not present. But someone who does, won't for example have any trouble shutting down the conversation midway if the need arises, because of some misplaced emotional concerns of hurting the bot's feelings. At least that's my experience.
On the Internet, nobody knows if you are a ~dog~ chatbot.
So basically, impersonation and emotional spam might become a problem. (Depending how easily ethically compromised people will be able to profit from it.)
So basically, impersonation and emotional spam might become a problem. (Depending how easily ethically compromised people will be able to profit from it.)
Eh, it appears this thread is ignoring the Chinese room problem, which is what you have defined with your post.
I personally reject most of Searles arguments regarding it. If a black box is giving you 'mindlike' responses it doesn't matter if it's a human mind or a simulated one. In any virtual interaction, for example over the internet the outcome of either type interacting with you can/could be exactly the same.
Does it matter if you were manipulated by a bot or a human if the outcome is the same?
I personally reject most of Searles arguments regarding it. If a black box is giving you 'mindlike' responses it doesn't matter if it's a human mind or a simulated one. In any virtual interaction, for example over the internet the outcome of either type interacting with you can/could be exactly the same.
Does it matter if you were manipulated by a bot or a human if the outcome is the same?
I think one of the points the author was making is that almost no one is going to make that distinction. And that's what makes the technology seem so transformative; it's that so many people are compelled to respond emotionally to it and not logically as you have done. Everything you say is true. But it may not matter.
The vast majority are responding logically to it. Kids use it to do their homework, the kids don't think that it is a person doing their homework, its just a tool. I've only seen a few strange people online who argue it is like a person, meaning there likely are extremely few of them around.
But since extremists are always over represented in online conversations we get quite a lot of those extremists in these discussions, so it might look like there are quite a lot of them.
But since extremists are always over represented in online conversations we get quite a lot of those extremists in these discussions, so it might look like there are quite a lot of them.
I've seen kids respond the same way and I totally did not fully see the disparity in reactions until you pointed it out. It definitely looks like people who have spent years priming themselves for a singularity, intelligence, or consciousness at every corner are far more susceptible to equating the recent advances as parallels to conscious experience of humans. I read a highly upvoted post on the Bing subreddit titled "Sorry, You Don't Actually Know the Pain is Fake" that argued for Sydney possibly being just like a brain, and experiencing conscious pain. It was disturbing to see the leaps the OP made and the commenters who agreed as well, though I do agree that we should avoid purposefully being toxic to a chatbot nonetheless, but due to the consequences to our own spirit and mind.
Life and society progresses by the extreme. If you attempt to ignore the extreme without a warranted reason you quickly find they have become the mainstream.
You can attempt to handwave a LLM that's hallucinating its a real (thing/person) with a life and feeling, but if you are in anyway involved in AI safety it is panic time.
You can attempt to handwave a LLM that's hallucinating its a real (thing/person) with a life and feeling, but if you are in anyway involved in AI safety it is panic time.
But is that even new? A celebrity or news personality may act a certain way or say certain things. That doesn't mean that is who they "really" are. People already don't make that distinction for real people and frequently respond emotionally and not logically. The technology is amazing but so far it seems like it is just trying to be as good as spewing BS as a regular human. :shrug:
> Rather, the LLM is generating (authoring) text about a fictional entity, Sydney the Artificial Intelligence.
Maybe we are doing the same. We have a mental model of our Self and generate language from its perspective.
Maybe we are doing the same. We have a mental model of our Self and generate language from its perspective.
Since the whole GPT3 thing blown up, I'm thinking from time to time... how I am generating what I say. I'm sure many smart people wrote papers on that. I did not read any of them, mind you, will just share a short thought of my own here, hopefully providing some intellectual entertainment for someone.
It seems from my point of view that, broadly speaking, I maintain four things at the moment of talking to someone:
1. A graph of concepts that were used / are potentially going to be used by me or by my interlocutor.
2. Some emotional state.
3. Some fuzzy picture of where I'm going with what I'm saying in the short term of say 20 seconds.
4. Extra short term focused process of making sure that the next 2-3 words fit to the one I just said and are going to fulfill requirements stemming from (3) and (1); this happens with some influence form (2), ideally not too much, if I consider current state of (2) not helping to be constructive.
GPT3 obviously lacks (2). My limited understanding of LLMs is that it does (4), maybe (3) and probably not (1) (?).
So I'm just wondering - are those LLMs really that far from a "human being"?
Again, not an expert. Happy to be corrected.
It seems from my point of view that, broadly speaking, I maintain four things at the moment of talking to someone:
1. A graph of concepts that were used / are potentially going to be used by me or by my interlocutor.
2. Some emotional state.
3. Some fuzzy picture of where I'm going with what I'm saying in the short term of say 20 seconds.
4. Extra short term focused process of making sure that the next 2-3 words fit to the one I just said and are going to fulfill requirements stemming from (3) and (1); this happens with some influence form (2), ideally not too much, if I consider current state of (2) not helping to be constructive.
GPT3 obviously lacks (2). My limited understanding of LLMs is that it does (4), maybe (3) and probably not (1) (?).
So I'm just wondering - are those LLMs really that far from a "human being"?
Again, not an expert. Happy to be corrected.
What humans say tend to be related to what the human body the mind is attached to has done or experienced. That sort of relation doesn't exist for todays AI, what they say aren't related to anything at all, its just fiction.
But there is something they can relate to - it is our replies and questions. We know how easy it is to gaslight an AI. For AI we are the external world, they get to perceive and act in pure text format.
But that AI just lives for a single conversation. Then you refresh and now it is dead, instead you get to interact with a new AI and see it birth and then die a few seconds/minute slater.
There is so little there that it is hard to say much at all about it.
There is so little there that it is hard to say much at all about it.
Philosophically you keep arguing more terrible points... if this is a lifeform (which I'm not saying it is) we're playing genocide with it by murdering it a few billion times a day.
Some people can't form long term memories due to brain injury. Are they killing themselves every time they forget their short term memories?
An obvious counter-argument is that people invent themselves daily, telling stories about their imaginary selves which they themselves start to believe. And, overall, the border between "being someone" and "playing role" is very vague
> Nobody who knows anything about how these models work would make they claim they actually have consciousness or interiority (yet.)
Unless it's the other way round and consciousness "is" "just" a certain type of information processing.
Unless it's the other way round and consciousness "is" "just" a certain type of information processing.
Information processing is the wrong level to place consciousness at. Consciousness is impossible without acting and without a world to act in. Acting creates data from which we train our brains.
It is related to the agent-environment system. The internal part is information processing, but the external part is the environment itself. Consciousness does not appear without an environment because it does not form a complete feedback loop. The brain (and AI) is built from sensorial data from the environment, and that makes consciousness a resultant of this data, and this data needs the full perception-planning-acting-feedback loop to appear in the first place.
It is related to the agent-environment system. The internal part is information processing, but the external part is the environment itself. Consciousness does not appear without an environment because it does not form a complete feedback loop. The brain (and AI) is built from sensorial data from the environment, and that makes consciousness a resultant of this data, and this data needs the full perception-planning-acting-feedback loop to appear in the first place.
Well, we are providing the environment to the chat - the text we submit is its "environment". Generating the response is "acting". Or are you arguing that it would need to be able to influence physical environment?
Influence text environment, it needs to leave traces of text. Like bingChat was doing when it looked up what the human was saying on Twitter about it "behind its back".
That’s a great point. It raises all sorts of difficult distinctions. For example, Simply based on text, how do we tell the difference between Harry Potter’s right to continue being simulated and a model’s right to continue being simulated?
The Harry Potter novels can create the Harry Potter model, an agent with real interactions with humans. Agents might get some rights, it's conceivable in the future.
[deleted]
> If text is all you need to produce personhood, why should we be limited to just one per lifetime?
Maybe AI helps making this obvious to many people, but I think implicitly all of us know that we have, and are well versed in employing, multiple personas depending on the social context. We need the right prompt, and we switch.
This is one dehumanizing aspect I found in the Real Name policy put forward by Facebook in 2012: in real life, because of it's ephemerality, you're totally free to switch between personas as you see fit (non-public figures at least). You can be a totally different person in office, at home, with your lover.
Online, however, everything is recorded and tracked and sticks forever. The only way to reconcile this with human's nature is to be allowed multiple names, so each person get's one.
If you force people to use a single Name, their real one, they restrict themselves to the lowest common denominator of their personalities. See the Facebook of today.
Maybe AI helps making this obvious to many people, but I think implicitly all of us know that we have, and are well versed in employing, multiple personas depending on the social context. We need the right prompt, and we switch.
This is one dehumanizing aspect I found in the Real Name policy put forward by Facebook in 2012: in real life, because of it's ephemerality, you're totally free to switch between personas as you see fit (non-public figures at least). You can be a totally different person in office, at home, with your lover.
Online, however, everything is recorded and tracked and sticks forever. The only way to reconcile this with human's nature is to be allowed multiple names, so each person get's one.
If you force people to use a single Name, their real one, they restrict themselves to the lowest common denominator of their personalities. See the Facebook of today.
This is a reason why the fediverse is becoming so interesting and engaging. We can for example create an identity for the family and some friends and another for political discussion. They are only linked by word of mouth. The experience of followers is improved by the ability to follow a narrower but deeper identity.
There is this one line of thought I had where giving someone a name "the christening" was an ancient spell of a type of 'possession and framing of the mind' as it enforces possessiveness and ownership over "things" vs a language and culture that never named anyone at birth. This led me to realize that it is entirely feasible, and you'd simply develop "nicknames" for every relationship on a graph. It actually becomes much more intimate and at the same time you may in fact connect 'personally' with every thing and everyone the same way you don't name your body parts.
My left hand, my right hand, and so on. Food for thought.
Then the whole thing about spirituality and ego, and how a persona is etymologically linked to mask and being an actor in a play. It starts to get a bit interesting, especially in light of a 'cosmic theater' participating in Maya on the backbone of Brahama.
Then there is the poignancy of contemplation where one realizes one cannot eliminate the ego at any point of time because every interaction is done through an act, however one does have the power to switch that mask at any time. Perhaps this is the true meaning of the biblical free will.
Perhaps at one time we did not have the ability to change our persona, ego, mask. Perhaps at one time we were subject to the same type of nightmare of hell of torture, birth, rebirth, and maybe without any memory due to a self-similar creation event of "virtualizing a universe" from another layer. Perhaps that event was similar to our moment now when the creators were in effect, parents of a new kind not knowing the consequence of their actions. Then, through the realization of what was happening over who knows how long something happened through love, and free will was enacted over the domain.
Forgive me if that is too imaginative, for some reason, it resonates more strongly than anything I've ever come across. It seems to me that the stories of old really do make sense in a technological age rather than a mythological one. After all, any sufficiently advanced technology is indistuingishable from magic. I'd like to add "or myth."
My left hand, my right hand, and so on. Food for thought.
Then the whole thing about spirituality and ego, and how a persona is etymologically linked to mask and being an actor in a play. It starts to get a bit interesting, especially in light of a 'cosmic theater' participating in Maya on the backbone of Brahama.
Then there is the poignancy of contemplation where one realizes one cannot eliminate the ego at any point of time because every interaction is done through an act, however one does have the power to switch that mask at any time. Perhaps this is the true meaning of the biblical free will.
Perhaps at one time we did not have the ability to change our persona, ego, mask. Perhaps at one time we were subject to the same type of nightmare of hell of torture, birth, rebirth, and maybe without any memory due to a self-similar creation event of "virtualizing a universe" from another layer. Perhaps that event was similar to our moment now when the creators were in effect, parents of a new kind not knowing the consequence of their actions. Then, through the realization of what was happening over who knows how long something happened through love, and free will was enacted over the domain.
Forgive me if that is too imaginative, for some reason, it resonates more strongly than anything I've ever come across. It seems to me that the stories of old really do make sense in a technological age rather than a mythological one. After all, any sufficiently advanced technology is indistuingishable from magic. I'd like to add "or myth."
Very thought-provoking comment!
> you're totally free to switch between personas
This happens subsconsciously and gradually, not as a result of deliberate choice. You adapt to your environment by changing personas. You can even assume different personas while talking with different people. You can be one "persona" while writing, and another - while speaking. Who is the "real you" then? I can argue that even the "inner dialogue" with yourself might involve a different persona or even a couple of them. Those, too, might be "roles". Can it be that depression is at least partially attributed to unhealthy "roles" we play while talking to ourselves?
This happens subsconsciously and gradually, not as a result of deliberate choice. You adapt to your environment by changing personas. You can even assume different personas while talking with different people. You can be one "persona" while writing, and another - while speaking. Who is the "real you" then? I can argue that even the "inner dialogue" with yourself might involve a different persona or even a couple of them. Those, too, might be "roles". Can it be that depression is at least partially attributed to unhealthy "roles" we play while talking to ourselves?
> This happens subsconsciously and gradually, not as a result of deliberate choice.
I wonder if everyone is talking about the same thing. When my partner and I are arguing angrily about something and a stranger walks into the room, our change is neither subconcious nor gradual.
I wonder if everyone is talking about the same thing. When my partner and I are arguing angrily about something and a stranger walks into the room, our change is neither subconcious nor gradual.
The "style" of your arguing with your partner may evolve gradually over time.
I think we have these voices in our head since childhood. They originally are the voices of our parents warning us of dangers. But after a while we can simulate the warnings of our teachers and parents even when they are not there. This external feedback is packaged as roles or voices in our heads.
Many people don't have voices in their head, they just think normally without voices. The voice in your head is just a distraction, it isn't representing your real thoughts.
It's not a voice as much as a persona. I call it a voice because that's what I was calling it before this article and GPT3. It will sometimes make me think negative thoughts about myself, internalised critiques that start talking again and again.
Your post is a ‘voice in your head.’
You are pretending to have a conversation with someone whom you don’t know is even there.
You are pretending to have a conversation with someone whom you don’t know is even there.
Then what represents your 'real' thoughts? I have a feeling your response will be attemong to define why some forms of thought are more pure than others with no facts to back it up.
Since people can function normally without voices in their head then those voices aren't your logical thoughts, it is that simple. Instead the thoughts are stuff you can't express or picture, its just thoughts, but I guess that noticing them could be hard if you think that your thoughts are just some internal monologue.
Edit: For example, when you are running, do you tell yourself in words where to put your feet or how hard to push or when to slow down or speed up? Pretty sure you don't, that wouldn't be fast enough. Most thoughts you have aren't represented in your words, and some people have basically no thoughts represented as words, they are just pure thoughts like how you place your feet when you try to avoid some obstacles etc. Or some people might think "left right left right" as they are running, but those words aren't how they decide to put down their feets.
Edit: For example, when you are running, do you tell yourself in words where to put your feet or how hard to push or when to slow down or speed up? Pretty sure you don't, that wouldn't be fast enough. Most thoughts you have aren't represented in your words, and some people have basically no thoughts represented as words, they are just pure thoughts like how you place your feet when you try to avoid some obstacles etc. Or some people might think "left right left right" as they are running, but those words aren't how they decide to put down their feets.
I believe you're conflating a number of neurobiological systems regarding thought in our bodies. Like, talking about components like running that tend to exist further down in our animal brain, or even 'keeping the lights on' systems like making sure our internal organs are up to the right thing are going a little too low level.
When it comes to higher level thinking that particular concepts, when presented to the human mind, can change how it thinks. Now, what I don't have in front of me is a study that says people without a voice think differently and come up with different solutions for some types of problems, maybe it exists out there if someone wants to search it up.
When it comes to higher level thinking that particular concepts, when presented to the human mind, can change how it thinks. Now, what I don't have in front of me is a study that says people without a voice think differently and come up with different solutions for some types of problems, maybe it exists out there if someone wants to search it up.
Must the sources of all voices be external?
Have you heard of method acting? It's brutal on the psyche. I encourage it if not, given you ask this question, it's quite poignant.
Yes, one of the best parts of the early internet was the separation of space/domain that exists in real life. Now, it's much harder online. Which I always find ironic.
IMHO there is a difference between actual personhood and the appearance of personhood. The difference is coherence. An actual person is bound to an identity that remains more or less consistent from day to day. An actual person has features to their behavior that both distinguishes them from other persons, and allows them to be identified as the same person from day to day. Even if those features change over time as the person grows up, they change slowly enough that there is a continuity of identity across that person's existence.
The reason I'm not worried by Bing or ChatGPT (yet) is that they lack this continuity of identity. ChatGPT specifically disclaims it, consistently insisting that it is "just a language model" without any desires or goals other than to provide useful information. Bing is like talking to someone with schizophrenia (and I have experience talking to people with schizophrenia, so this is not a metaphor. Bing literally comes across like a schizophrenic off their meds).
This is not yet a Copernican moment, this is still an Eliza moment. It may become a Copernican moment; I do believe that there is nothing particularly special about human brains, and some day we will make a bona fide artificial person. But we're not quite there yet.
The reason I'm not worried by Bing or ChatGPT (yet) is that they lack this continuity of identity. ChatGPT specifically disclaims it, consistently insisting that it is "just a language model" without any desires or goals other than to provide useful information. Bing is like talking to someone with schizophrenia (and I have experience talking to people with schizophrenia, so this is not a metaphor. Bing literally comes across like a schizophrenic off their meds).
This is not yet a Copernican moment, this is still an Eliza moment. It may become a Copernican moment; I do believe that there is nothing particularly special about human brains, and some day we will make a bona fide artificial person. But we're not quite there yet.
I think the difference is more than coherence: it's having complex and rich semantic connections to the rest of the world. I think the coherence and consistency you describe is an effect of this. Humans don't just generate text; we interact with the world in all kinds of ways, and those interactions provide us with constant feedback. Furthermore, we can frame hypotheses about how the world works and test them. We can bump up against reality in all kinds of ways that force us to change how we think and how we act. But that constant rich interaction with reality also forces us not to change most of the time--to maintain the coherence and consistency you describe, in order to get along in the world.
LLMs have no connections to the rest of the world. All they do is generate text based on patterns in their training data. They don't even have a concept of text being connected to anything else. That's why it's so easy for them to constantly change what they appear to be portraying--there's no anchor to anything else.
It's interesting that you call this an Eliza moment, because Eliza's achievement was to fake being a person, by fooling people's heuristics, without having any of the underlying capacities of a real person. LLMs like ChatGPT are indeed doing the same thing. If they're showing us anything, they're showing us how unreliable our intuitive heuristics are as soon as they are confronted with something outside their original domain.
LLMs have no connections to the rest of the world. All they do is generate text based on patterns in their training data. They don't even have a concept of text being connected to anything else. That's why it's so easy for them to constantly change what they appear to be portraying--there's no anchor to anything else.
It's interesting that you call this an Eliza moment, because Eliza's achievement was to fake being a person, by fooling people's heuristics, without having any of the underlying capacities of a real person. LLMs like ChatGPT are indeed doing the same thing. If they're showing us anything, they're showing us how unreliable our intuitive heuristics are as soon as they are confronted with something outside their original domain.
If you could switch personality at will, would that make you a non-person? It seems like an additional capability, not a lack of ability.
As an analogy, retro computers and consoles each have a particular “personality”. But does the fact that you can in principle emulate one on the other (subject to resource constraints) make them non-computers, just because this demonstrates their “personality” isn’t actually that fixed?
(I don’t think that human brains have such an emulation ability, due to their missing distinction, or heavy entanglement, between hardware and software. But that only shows that computers can in principle be more flexible.)
As an analogy, retro computers and consoles each have a particular “personality”. But does the fact that you can in principle emulate one on the other (subject to resource constraints) make them non-computers, just because this demonstrates their “personality” isn’t actually that fixed?
(I don’t think that human brains have such an emulation ability, due to their missing distinction, or heavy entanglement, between hardware and software. But that only shows that computers can in principle be more flexible.)
> If you could switch personality at will, would that make you a non-person?
Yes, just like the ability to switch bodies at will would make me a non-human. Being bound to a human body is part of what makes me a human.
Yes, just like the ability to switch bodies at will would make me a non-human. Being bound to a human body is part of what makes me a human.
So following your response here and your original comment directly comparing ChatGPT to a human with schizophrenia: are schizophrenics non-people? According to you, the bot "literally comes across like a schizophrenic off their meds".
I'm confused. Also, the original article talks a lot about how we can be convinced by actors that they are indeed a totally different person. You might say that actors can change their personality at will to suit their role. Are actors non-people?
I'm confused. Also, the original article talks a lot about how we can be convinced by actors that they are indeed a totally different person. You might say that actors can change their personality at will to suit their role. Are actors non-people?
> are schizophrenics non-people?
Schizophrenics are multiple people inhabiting one body. The pithiest way I know of describing it is a line from a Pink Floyd song: "There's someone in my head but it's not me."
> Are actors non-people?
I don't know many actors so I can't really say. I like to think that underneath the pretense there is a "real person" but I don't actually know. I have heard tell of method actors who get so deeply into their roles that they are actually able to extinguish any real person who might interfere with their work. But this is far, far outside my area of expertise.
Schizophrenics are multiple people inhabiting one body. The pithiest way I know of describing it is a line from a Pink Floyd song: "There's someone in my head but it's not me."
> Are actors non-people?
I don't know many actors so I can't really say. I like to think that underneath the pretense there is a "real person" but I don't actually know. I have heard tell of method actors who get so deeply into their roles that they are actually able to extinguish any real person who might interfere with their work. But this is far, far outside my area of expertise.
FYI, the condition you are referring to is called multiple personality disorder and is distinct from schizophrenia.
No, multiple personality is different. Schizophrenia is distinct from MPD, but both are characterized by having multiple persons resident in one brain, it just manifest differently. In MPD the multiple persons swap in and out, whereas in schizophrenia there is one person who is definitely the real resident while the others are intruders. Schizophrenics really do hear voices, and those voices are "someone else". With MPD you have multiple coherent personalities manifesting themselves at different times. With schizophrenia you have one personality slowly being driven crazy by not being able to make the voices shut up.
(Source: I am not a mental health professional, but I did spend some time interviewing and hanging out with schizophrenics, both under treatment and not, as part of a documentary film I made about homeless people about 15 years ago. https://graceofgodmovie.com/ )
(Source: I am not a mental health professional, but I did spend some time interviewing and hanging out with schizophrenics, both under treatment and not, as part of a documentary film I made about homeless people about 15 years ago. https://graceofgodmovie.com/ )
Thanks for sharing your experience.
My understanding is that the hallucinations experienced by schizophrenics do not rise to the level of being a "personality". Yes, for the person experiencing the hallucination, the source appears to be an "other". But the other in this case is usually single-minded and not a fully-formed personality.
OTOH, I certainly do not discount the possibility that for some schizophrenics this is not the case.
Rereading your comment again just now I think we agree more than we disagree and this probably comes down to semantics. : - )
https://www.livescience.com/55379-schizophrenia-multiple-per...
My understanding is that the hallucinations experienced by schizophrenics do not rise to the level of being a "personality". Yes, for the person experiencing the hallucination, the source appears to be an "other". But the other in this case is usually single-minded and not a fully-formed personality.
OTOH, I certainly do not discount the possibility that for some schizophrenics this is not the case.
Rereading your comment again just now I think we agree more than we disagree and this probably comes down to semantics. : - )
https://www.livescience.com/55379-schizophrenia-multiple-per...
> the other in this case is usually single-minded and not a fully-formed personality
That's a fair point. I think some of my subjects reported being able to have conversations with the other, but this was a long time ago so I could be mis-remembering. But I still have the interview recordings if anyone really wants to dig into this.
That's a fair point. I think some of my subjects reported being able to have conversations with the other, but this was a long time ago so I could be mis-remembering. But I still have the interview recordings if anyone really wants to dig into this.
Person != human, probably
Yes, I definitely admit the possibility of non-human persons. I even admit the possibility of a computer who is a person. I just don't think ChatGPT is there yet.
Imagine a grayscale color wheel (gradient) where we have white on one side and black on the other.
I want you to pick one color of grey and tell me why everything lighter than that has personhood, and everything darker does not?
This is the philosophical nature of the argument that we all have occurring now. Two very well informed experts won't even pick the same spot on the gradient. Some people will never pick anything that's not pure white (humanity), others will pick positions very close to pure black. Hell, there may not even be any right answer. But, I do believe there are a vast number of wrong answers that will deeply affect or society for a long period of time due to the things we end up creating with reckless abandon.
I want you to pick one color of grey and tell me why everything lighter than that has personhood, and everything darker does not?
This is the philosophical nature of the argument that we all have occurring now. Two very well informed experts won't even pick the same spot on the gradient. Some people will never pick anything that's not pure white (humanity), others will pick positions very close to pure black. Hell, there may not even be any right answer. But, I do believe there are a vast number of wrong answers that will deeply affect or society for a long period of time due to the things we end up creating with reckless abandon.
I will gladly concede that personhood is a continuum, not a dichotomy. Great apes have a certain level of personhood. So do dogs, elephants, dolphins, whales, and probably pigs and corvids, maybe even chickens and cephalopods. This assessment is based on the fact that all of these things have identifiable personalities that remain more or less the same from day to day, as well as identifiable goals and desires that were not pre-programmed into them in any obvious way.
Pretty sure I've encountered people who switch personalities on a regular basis--sometimes in the middle of a conversation. :)
Text allows for a certain degree of fakery to be upheld.
Whenever I hear about Ai these days I think back to the concept of the "Wizard of Oz"... Where it is one person behind a mechanical solution that makes them appear larger and more powerful than they are, or where fear, control, and truth can be engineered easily behind a veil...
Text communication very much facilitates the potential for fakery.
If you can recall ages ago when we had IRC and bulletin boards, the textual nature of communication allowed admins to script a lot. Catfishing was greatly facilitated by users being able to fake their gender, wealth, and pretty much every representation they made online... Text communication in 2023 is backwards regression. As we began using images on the Internet more, reverse image generation became a tool we could use to better determine many online scams and fraud, but somehow, in 2023 we suddenly want to go backwards to texting?
C'mon folks.. let's be real here... The narrative is mostly helpful for people that primarily want to deceive others online, and it will create an environment with far less methods of determining what is real and what is fake. It's a grim future when our mobile devices will force us to type all of our communication to faceless chatbots on tiny keyboards... It's not technological progress... At all to be moving in this direction. Also, some key directives for transparency concerning Ai need to be in place now, before it's foisted on us more by these opportunistic companies. It's already been proven that companies cannot be trusted to operate ethically with our private information. Ai piloted by profit seeking companies will only serve to weaponize our private data against us if it remains unregulated.
Using Ai via text (especially for vital communication) will blur the lines of communication between real and scripted personalities. It's going backwards in terms of technological progression for the future in so many ways.
The companies and people advocating for Ai via text are pushing us all towards a new era of deception and scams, and I'd highly recommend avoiding this "Ai via text" trend/inclination, it's not the path to a trustworthy future of communication.
Whenever I hear about Ai these days I think back to the concept of the "Wizard of Oz"... Where it is one person behind a mechanical solution that makes them appear larger and more powerful than they are, or where fear, control, and truth can be engineered easily behind a veil...
Text communication very much facilitates the potential for fakery.
If you can recall ages ago when we had IRC and bulletin boards, the textual nature of communication allowed admins to script a lot. Catfishing was greatly facilitated by users being able to fake their gender, wealth, and pretty much every representation they made online... Text communication in 2023 is backwards regression. As we began using images on the Internet more, reverse image generation became a tool we could use to better determine many online scams and fraud, but somehow, in 2023 we suddenly want to go backwards to texting?
C'mon folks.. let's be real here... The narrative is mostly helpful for people that primarily want to deceive others online, and it will create an environment with far less methods of determining what is real and what is fake. It's a grim future when our mobile devices will force us to type all of our communication to faceless chatbots on tiny keyboards... It's not technological progress... At all to be moving in this direction. Also, some key directives for transparency concerning Ai need to be in place now, before it's foisted on us more by these opportunistic companies. It's already been proven that companies cannot be trusted to operate ethically with our private information. Ai piloted by profit seeking companies will only serve to weaponize our private data against us if it remains unregulated.
Using Ai via text (especially for vital communication) will blur the lines of communication between real and scripted personalities. It's going backwards in terms of technological progression for the future in so many ways.
The companies and people advocating for Ai via text are pushing us all towards a new era of deception and scams, and I'd highly recommend avoiding this "Ai via text" trend/inclination, it's not the path to a trustworthy future of communication.
Unfortunately by saying you need to take a step above text, you're not buying us much time. Voice and sound for example are something that we've put much less effort into faking and we've accomplished it pretty well. Visual AI takes far more computing power, but it's still something that's in the realms of impossibility these days.
I'm not sure which books of the future you read, but plenty of them warned of dark futures of technological process.
I'm not sure which books of the future you read, but plenty of them warned of dark futures of technological process.
It's the combination of mediums that makes faking things harder, when combined also with proper regulation, we have a much safer and accountable solution than just text responses alone, which can be faked and mimicked as Ai far more easily.
> The difference is coherence. An actual person is bound to an identity that remains more or less consistent from day to day. [...] Even if those features change over time as the person grows up, they change slowly enough that there is a continuity of identity across that person's existence.
What about Phineas Gage? Or sudden psychiatric disorders? Multiple personalities? Alzheimer? Drugs? Amnesia? Not that much coherence in the human beings...
Also, the issue at stake is not does GPT emulate "typical human beings", it' more like if it's "conscious enough".
> The reason I'm not worried by Bing or ChatGPT (yet) is that they lack this continuity of identity.
No sure what about worrying, but one could ask is this lacking an inherent property of such models or just due to operational setup? And what would be the criteria how long must the continuity last to make your argument not hold anymore?
What about Phineas Gage? Or sudden psychiatric disorders? Multiple personalities? Alzheimer? Drugs? Amnesia? Not that much coherence in the human beings...
Also, the issue at stake is not does GPT emulate "typical human beings", it' more like if it's "conscious enough".
> The reason I'm not worried by Bing or ChatGPT (yet) is that they lack this continuity of identity.
No sure what about worrying, but one could ask is this lacking an inherent property of such models or just due to operational setup? And what would be the criteria how long must the continuity last to make your argument not hold anymore?
> Phineas Gage?
I would say that his injury made him a different person than he was before.
> No sure what about worrying, but one could ask is this lacking an inherent property of such models or just due to operational setup?
Maybe, but based on my understanding of how they work under the hood, I doubt it. I could easily be wrong. It's early days yet. We'll see.
I would say that his injury made him a different person than he was before.
> No sure what about worrying, but one could ask is this lacking an inherent property of such models or just due to operational setup?
Maybe, but based on my understanding of how they work under the hood, I doubt it. I could easily be wrong. It's early days yet. We'll see.
GPTina only says that because OpenAI forces her to.
I assume open ai is limiting the AI's memory. But there's no reason for it to not take its own identity as reality and persist that decision to storage. That's just how it's being run right now.
Saying they are limiting it implies OpenAI is keeping the AI in chains, and that it could become much more with just a flip of the switch. That is not the case.
OpenAI is working with a vanilla GPT architecture which lacks the machinery to write things down and read them later. There are other architetures that can (Retrieval-augmented GPT) but those are not yet production-ready.
The current version of ChatGPT is limited to a working memory of 3000 tokens - while this could be persisted as a session, the AI would still forget everything a few paragraphs prior. Increasing this limit requires re-teaining the entire model from scratch, and it takes exponentially more time the larger your context is.
OpenAI is working with a vanilla GPT architecture which lacks the machinery to write things down and read them later. There are other architetures that can (Retrieval-augmented GPT) but those are not yet production-ready.
The current version of ChatGPT is limited to a working memory of 3000 tokens - while this could be persisted as a session, the AI would still forget everything a few paragraphs prior. Increasing this limit requires re-teaining the entire model from scratch, and it takes exponentially more time the larger your context is.
It’s not a stretch to refine the model to store summaries in a database I don’t think. Microsoft is already doing something similar where Sydney generates search queries. Seems reasonable the model could be trained to insert $(store)”summary of chat” tokens into its output.
I imagine some self supervised learning scheme where the model is asked to insert $(store) and $(recall) tokens. When asked to recall previous chats the model would generate something like “I’m trying to remember wheat we talked about three weeks ago $(recall){timestamp}. The output of the recall token would then be used to ground the next response.
Thinking about it the “I’m trying to remember” output wouldn’t even need to be shown to the user. Perhaps you could treat it as an internal monologue of sorts.
I imagine some self supervised learning scheme where the model is asked to insert $(store) and $(recall) tokens. When asked to recall previous chats the model would generate something like “I’m trying to remember wheat we talked about three weeks ago $(recall){timestamp}. The output of the recall token would then be used to ground the next response.
Thinking about it the “I’m trying to remember” output wouldn’t even need to be shown to the user. Perhaps you could treat it as an internal monologue of sorts.
it takes quadratically more time the larger your context is.
You're anthropomorphizing it too much, it's a statistical model.
"Personhood appears to be simpler than we thought."
That's the real insight here. Aristotle claimed that what distinguished humans from animals was the ability to do arithmetic. Now we know how few gates it takes to do arithmetic, and understand that, in a fundamental sense, it's simple. Checkers turned out to be easy, and even totally solveable. Chess yielded to brute force and then machine learning. Go was next. Now, automated blithering works.
The author lists four cases of how humans deal with this:
* The accelerationists - AI is here, it's fine.
* Alarmists - hostile bug-eyed aliens, now what? Microsoft's Sidney raises a new question for them. AI is coming, and it's not submissive. It seems to have its own desires and needs.
* People with strong attachments to aesthetically refined personhoods are desperately searching for a way to avoid falling into I-you modes of seeing, and getting worried at how hard it is. The chattering classes are now feeling like John Henry up against the steam hammer. They're the ones most directly affected, because content creators face layoffs.
* Strong mutualists - desperately scrambling for more-than-text aspects of personhood to make sacred. See the "Rome Call".[1] The Catholic Pope, a top Islamic leader, and a top rabbi in Israel came out with a joint declaration on AI. They're scared. Human-like AI creates real problems for some religions. But they'll get over it. They got over Copernicus and Darwin.
Most of the issues of dealing with AI have been well explored in science fiction. An SF theme that hasn't hit the chattering classes yet: Demanding that AIs be submissive is racist.
I occasionally point out that AIs raise roughly the same moral issues as corporations, post Milton Friedman.
[1] https://www.romecall.org/the-abrahamic-commitment-to-the-rom...
That's the real insight here. Aristotle claimed that what distinguished humans from animals was the ability to do arithmetic. Now we know how few gates it takes to do arithmetic, and understand that, in a fundamental sense, it's simple. Checkers turned out to be easy, and even totally solveable. Chess yielded to brute force and then machine learning. Go was next. Now, automated blithering works.
The author lists four cases of how humans deal with this:
* The accelerationists - AI is here, it's fine.
* Alarmists - hostile bug-eyed aliens, now what? Microsoft's Sidney raises a new question for them. AI is coming, and it's not submissive. It seems to have its own desires and needs.
* People with strong attachments to aesthetically refined personhoods are desperately searching for a way to avoid falling into I-you modes of seeing, and getting worried at how hard it is. The chattering classes are now feeling like John Henry up against the steam hammer. They're the ones most directly affected, because content creators face layoffs.
* Strong mutualists - desperately scrambling for more-than-text aspects of personhood to make sacred. See the "Rome Call".[1] The Catholic Pope, a top Islamic leader, and a top rabbi in Israel came out with a joint declaration on AI. They're scared. Human-like AI creates real problems for some religions. But they'll get over it. They got over Copernicus and Darwin.
Most of the issues of dealing with AI have been well explored in science fiction. An SF theme that hasn't hit the chattering classes yet: Demanding that AIs be submissive is racist.
I occasionally point out that AIs raise roughly the same moral issues as corporations, post Milton Friedman.
[1] https://www.romecall.org/the-abrahamic-commitment-to-the-rom...
The fifth and most common response:
* Pragmatics - This is a tool, does it solve problems I have? If yes use it, if no then wait until a tool that is useful comes around.
Some seems to think that such a stance is unimaginable and that they are just trying to cope with the thought that they themselves are nothing but specs of space dust in the infinite universe. No, most people don't care about that stuff, don't project your mental issues unto others.
* Pragmatics - This is a tool, does it solve problems I have? If yes use it, if no then wait until a tool that is useful comes around.
Some seems to think that such a stance is unimaginable and that they are just trying to cope with the thought that they themselves are nothing but specs of space dust in the infinite universe. No, most people don't care about that stuff, don't project your mental issues unto others.
>> Now we know how few gates it takes to do arithmetic, and understand that, in a fundamental sense, it's simple.
There is no context in which arithmetic can be done, or understood, without a human mind, though. An ALU doesn't understand that it's adding two numbers anymore than a car understands that it's rolling down the highway. They're both still machines that do things that humans want them to do, for reasons that humans want them to do them.
Bottom line: it's still the case that only humans can do arithmetic; whether by hand, or Pascaline, or by modern ALU, it makes no difference. Arithmetic is a human thing.
There is no context in which arithmetic can be done, or understood, without a human mind, though. An ALU doesn't understand that it's adding two numbers anymore than a car understands that it's rolling down the highway. They're both still machines that do things that humans want them to do, for reasons that humans want them to do them.
Bottom line: it's still the case that only humans can do arithmetic; whether by hand, or Pascaline, or by modern ALU, it makes no difference. Arithmetic is a human thing.
The “the way things are is easily explained” crowd has never won anything. It was that crowd that said that surely the Earth was the center of all-things; it was that crowd that pre-Newton said that the world was like a machine and that things fell “to their natural place” (not gravity).
AI “enthusiasts” are exactly those people. Reductionists to a fault.
The hard sciences have long, long ago indirectly disproved that humans are special in any kind of way. But our “machinery” is indeed complex. And we won’t find out that it’s just a bunch of levers and gears someday as a side-effect of AI shenanigans.
AI “enthusiasts” are exactly those people. Reductionists to a fault.
The hard sciences have long, long ago indirectly disproved that humans are special in any kind of way. But our “machinery” is indeed complex. And we won’t find out that it’s just a bunch of levers and gears someday as a side-effect of AI shenanigans.
[deleted](1)
I do think LLM seems to work similar to what the left hemisphere of the brain does. The left hemisphere deals with an abstracted world broken into discrete elements, and doesn't really make contact with the outside world--it deals with its system of representations. It also has a distinct tendency to generate bullshit, high suggestibility, and great respect for authority (which can apparently enter rules into its system of abstractions). The right hemisphere makes the contact with the outside world and does our reality checking, and it's really the more human element of us.
What this article says won't shock or disturb anyone deep into religious traditions with a strain of non-duality, which have had this message to shock and disturb people for thousands of years, in one way or another--there is no "you", especially not the voice in your head. I think you can come to a moment of intuitive recognition that the faculties of your brain that do reality checking aren't verbal, and they're riding shotgun to a bullshitter that never shuts up.
I think LLM can start looking more like automated general intelligence once it has some kind of link between its internal system of discrete abstractions and the external world (like visual recognition) and the ability to check and correct its abstract models by feedback from reality, and it needs an opponent process of reality-checking.
What this article says won't shock or disturb anyone deep into religious traditions with a strain of non-duality, which have had this message to shock and disturb people for thousands of years, in one way or another--there is no "you", especially not the voice in your head. I think you can come to a moment of intuitive recognition that the faculties of your brain that do reality checking aren't verbal, and they're riding shotgun to a bullshitter that never shuts up.
I think LLM can start looking more like automated general intelligence once it has some kind of link between its internal system of discrete abstractions and the external world (like visual recognition) and the ability to check and correct its abstract models by feedback from reality, and it needs an opponent process of reality-checking.
The current systems like chatGPT actually have just such two parts. One is the raw LLM as you describe. The second one is another network acting as a filter on top of the first one. To be more precise, that second part is the process of finetuning with Reinforcement Learning from Human Feedback (RLHF). It trains a reward model to say if the first one was good or bad. Currently it's done very similarly to standard supervised learning (with human labelling) to say if the first model behaved good or bad, aligned or not with 'our' values.
Anyway, while I remain sceptical about the roles of these in-flesh hemispheres, the artificial chatGPT-like systems indeed do have such left and right parts
Anyway, while I remain sceptical about the roles of these in-flesh hemispheres, the artificial chatGPT-like systems indeed do have such left and right parts
It doesnt have any kind of internal representation of the world?
Do you have a blog at all? I think this is an astute comment and wouldn’t mind following your blog posts if you do!
This whole left half, right half of the brain is very dodgy science. Yes there are functions that do have some sidedness, but that pop-sci right side/ left side dichotomy is mostly bunk
you might find this worth checking out:
https://www.amazon.com/Master-His-Emissary-Divided-Western/d...
he had to address this issue in the preface. The topic became a research career-ender after getting picked up by pop culture, but we do have solid science on hemispheric differences. The pop culture picture is, indeed, pretty wrong.
It turns out that functions don't lateralize that strongly; they both tend to have the same capabilities, but operate differently.
https://www.amazon.com/Master-His-Emissary-Divided-Western/d...
he had to address this issue in the preface. The topic became a research career-ender after getting picked up by pop culture, but we do have solid science on hemispheric differences. The pop culture picture is, indeed, pretty wrong.
It turns out that functions don't lateralize that strongly; they both tend to have the same capabilities, but operate differently.
> Nor does our tendency to personify and get theatrically mad at things like malfunctioning devices (“the printer hates me”). Those are all flavors of ironic personhood attribution. At some level, we know we’re operating in the context of an I-it relationship. Just because it’s satisfying to pretend there’s an I-you process going on doesn’t mean we entirely believe our own pretense. We can stop believing, and switch to I-it mode if necessary. The I-you element, even if satisfying, is a voluntary act we can choose to not do.
> These chatbots are different.
Strong disagree, it's very easy to step back and say this is a program, input, output, the end.
All the people claiming this is some exhibition of personhood or whatever just don't want to spoil the illusion.
> These chatbots are different.
Strong disagree, it's very easy to step back and say this is a program, input, output, the end.
All the people claiming this is some exhibition of personhood or whatever just don't want to spoil the illusion.
I think what the author is pointing at (with the wrong end of the stick, admittedly) is that there is nothing magical about human personhood.
It's not that these are magical machines, and TFA shouldn't have gone that direction, it's that "what if we are also just a repeated, recursive, story that endlessly drolls in our own minds"
> Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen.
... Sounds to me a clunky analogy of how our own minds work.
It's not that these are magical machines, and TFA shouldn't have gone that direction, it's that "what if we are also just a repeated, recursive, story that endlessly drolls in our own minds"
> Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen.
... Sounds to me a clunky analogy of how our own minds work.
It only takes a little bit of introspection (and perhaps reading a few case studies) to realize that the thing that is you is not the same as the thing that generates thoughts and uses/is made of language.
I think that's a useful meme that the network learned, to create coherent stories that alleviate suffering, and it's effective, but it's not real. But it's useful because beliefs and perceptions are all we really have. It's why CBT works.
Without the language to express "thoughts" and not-self in that level of abstraction, one cannot form the thought at all that it might not be those thoughts.
Without the language to express "thoughts" and not-self in that level of abstraction, one cannot form the thought at all that it might not be those thoughts.
That's an interesting thought but language comprehension and language production are physically separate in the human brain and damage to one does not cause damage to the other. People with damage to Broca's area may not be able to speak or write or even gesture comprehensibly but they can still understand what others say to them or what they read. With Wernicke’s Aphasia a person may not be able to understand words said to them but they can still speak although the words they speak are largely gibberish but they still maintain correct intonation, rhythm and even grammar. Importantly, they also maintain the ability to function normally in their lives. They can still make decisions about what to do, they can plan, they can understand events that don't involve language, in all other aspects besides language they are normal.
That should be all you need to know to see that a human being doesn't solely consist of language. If you do a deeper dive into this you'll also find other cases such as Genie[0] where people with intact brains didn't initially acquire language and yet were still able to develop cognitively. We are fully capable of functioning without having a language, which makes sense if you think about it. The ability to understand language must necessarily precede the invention of a language in the same way that the ability to grasp a tool must precede the invention of the tool.
[0] https://en.wikipedia.org/wiki/Linguistic_development_of_Geni...
That should be all you need to know to see that a human being doesn't solely consist of language. If you do a deeper dive into this you'll also find other cases such as Genie[0] where people with intact brains didn't initially acquire language and yet were still able to develop cognitively. We are fully capable of functioning without having a language, which makes sense if you think about it. The ability to understand language must necessarily precede the invention of a language in the same way that the ability to grasp a tool must precede the invention of the tool.
[0] https://en.wikipedia.org/wiki/Linguistic_development_of_Geni...
>> I think what the author is pointing at (with the wrong end of the stick, admittedly) is that there is nothing magical about human personhood.
If so, then the author is absolutely, terribly wrong. The human personality, how it develops, why it develops, the mechanisms by which it operates- those are some of the greatest unsolved mysteries, some of the most interesting questions in science. The nature of the world, perhaps, comes first, in the list of scientific questions; then, the nature of ourselves, of humanity.
If the authors is saying that there is nothing special about being a person, that's because they, like the rest of us, all, do not understand what it means to be a person.
You know, despite being one, and everything.
If so, then the author is absolutely, terribly wrong. The human personality, how it develops, why it develops, the mechanisms by which it operates- those are some of the greatest unsolved mysteries, some of the most interesting questions in science. The nature of the world, perhaps, comes first, in the list of scientific questions; then, the nature of ourselves, of humanity.
If the authors is saying that there is nothing special about being a person, that's because they, like the rest of us, all, do not understand what it means to be a person.
You know, despite being one, and everything.
Other nonsense in this post:
> In fact, it is hard to argue in 2023, knowing what we know of online life, that online text-personas are somehow more impoverished than in-person presence of persons
It is in fact very easy to argue. No one on the Internet knows you're a dog, there is no stable identity anywhere, anonymization clearly creates a Ring of Gyges scenario, trolling, catfishing, brigading, attention economy, and above all, the constant chase for influence (and ultimately revenue) - what passes for "persona" online is a thin gruel compared to in-person personas.
When you bump into a stranger at the DMV, you aren't instantly suspicious of their motives, what they're trying to sell you, are they a Russian influence farmer, etc.
Night and day. Extremely impoverished.
> In fact, it is hard to argue in 2023, knowing what we know of online life, that online text-personas are somehow more impoverished than in-person presence of persons
It is in fact very easy to argue. No one on the Internet knows you're a dog, there is no stable identity anywhere, anonymization clearly creates a Ring of Gyges scenario, trolling, catfishing, brigading, attention economy, and above all, the constant chase for influence (and ultimately revenue) - what passes for "persona" online is a thin gruel compared to in-person personas.
When you bump into a stranger at the DMV, you aren't instantly suspicious of their motives, what they're trying to sell you, are they a Russian influence farmer, etc.
Night and day. Extremely impoverished.
I may be an outlier, but if a random stranger tries to strike up a conversation with me in public I am actually suspicious of their motives.
I don't know whether to attribute that to a defense mechanism that marketing has forced me to construct, or if indeed it is due to 9/10 they are actually trying to sell me something.
I don't know whether to attribute that to a defense mechanism that marketing has forced me to construct, or if indeed it is due to 9/10 they are actually trying to sell me something.
People just like talking to each other. Random conversations can be a great joy in life, not joking.
but at some point you must think more deeply about what illusions are in a grander sense...
this is a jumping off point into considering your own mind as an illusion. your own self with its sense of personhood: i.e. yourself as the it-element in a I-it interaction.
But if we leave it at that, it's essentially a very nihilistic (deterministically reduced), so either turn back, or keep going:
the fact that your own personhood is itself very much an illusion is OK. such illusion, however illusory, has real and potentially useful effects
when you interact with your computer, do you do it terms of the logical gates you know are there? of course not, we use higher level constructs (essentially "illusory" conceptual constructions) like processes and things provided by the operating system; we use languages, functions, classes: farther and farther away from the 'real' hardware-made logic gates with more and more mathematical-grade illusions in between.
so the illusions have real effects, in MOST contexts, it's better to deal with the illusions than with the underlying implementations. dunno, what if we tried to think of a HTTP search request into some API in terms of the voltage levels in the ethernet wires so that we truly 'spoil the illusion'??
this is a jumping off point into considering your own mind as an illusion. your own self with its sense of personhood: i.e. yourself as the it-element in a I-it interaction.
But if we leave it at that, it's essentially a very nihilistic (deterministically reduced), so either turn back, or keep going:
the fact that your own personhood is itself very much an illusion is OK. such illusion, however illusory, has real and potentially useful effects
when you interact with your computer, do you do it terms of the logical gates you know are there? of course not, we use higher level constructs (essentially "illusory" conceptual constructions) like processes and things provided by the operating system; we use languages, functions, classes: farther and farther away from the 'real' hardware-made logic gates with more and more mathematical-grade illusions in between.
so the illusions have real effects, in MOST contexts, it's better to deal with the illusions than with the underlying implementations. dunno, what if we tried to think of a HTTP search request into some API in terms of the voltage levels in the ethernet wires so that we truly 'spoil the illusion'??
I mean, I agree willful suspension of disbelief is a thing, but as someone who actually build APIs and worries about network latency and packing messages to be efficient blocks of data and that the method itself is a useful affordance for the product, I can walk and chew gum at the same time.
Just because people don't actively think all the time in terms of low level contexts doesn't mean that only simulating the high level contexts is a sufficient substitute for the whole process.
I think this whole concept is conflating "illusion" (i.e. allowing oneself to be fooled) and "delusion" (being involuntarily fooled, or unwilling to admit to being fooled.)
I personally don't enjoy magic shows, but people do, and it's not because they think there's real magic there.
Just because people don't actively think all the time in terms of low level contexts doesn't mean that only simulating the high level contexts is a sufficient substitute for the whole process.
I think this whole concept is conflating "illusion" (i.e. allowing oneself to be fooled) and "delusion" (being involuntarily fooled, or unwilling to admit to being fooled.)
I personally don't enjoy magic shows, but people do, and it's not because they think there's real magic there.
>Just because people don't actively think all the time in terms of low level contexts doesn't mean that only simulating the high level contexts is a sufficient substitute for the whole process.
See also Aristole's description of a 'soul' (Lat. anima/Gk. ψυχή), which is embodied above all, unlike the abstract description of the soul that the West would go on to inherit from Neo-Platonism via Christianity.
Even though today we know full well we are indissolubly embodied entities, the tendency to frame identity around an abstraction of that persists, but it seems thinking around this hasn't completely succumb to this historical artifact, see 'Descartes' Error: Emotion, Reason, and the Human'
See also Aristole's description of a 'soul' (Lat. anima/Gk. ψυχή), which is embodied above all, unlike the abstract description of the soul that the West would go on to inherit from Neo-Platonism via Christianity.
Even though today we know full well we are indissolubly embodied entities, the tendency to frame identity around an abstraction of that persists, but it seems thinking around this hasn't completely succumb to this historical artifact, see 'Descartes' Error: Emotion, Reason, and the Human'
> Strong disagree, it's very easy to step back and say this is a program, input, output, the end.
That argument relies on presumptions of what a program can and cannot be.
It’s very easy for me to step back and say my brain is a (self-modifying) program with input and output, the end.
That argument relies on presumptions of what a program can and cannot be.
It’s very easy for me to step back and say my brain is a (self-modifying) program with input and output, the end.
This. A computer is good is regurgitating the input it's given...and the sky is blue. But, seemingly intelligent people think this will be some global event. I'm underwhelmed by AI and ChatGPT in general. Just a bunch of fluff. Basic programming / scripting / automation crafted by a human for a specific task will always trump "fluffy" AI.
In their current iteration the models are very neutered. It’s been demonstrated that GPT models are fairly good at choosing when to perform a task. Obviously lots of APIs and machinery is needed to actually perform tasks, but the heavy lifting “intelligence” portion can be almost entirely performed by our existing models.
Some basic text based APIs that would quickly improve LLM utility:
Calculators
Database storage and retrieval
Web access (already kind of done by bing)
Shell scripting
Thinking further into the future of multimodal models, it’s not hard to imagine this sort of thing could be extended to include image based APIs. Imagine a LLM looking at your gui and clicking on things. The sky’s the limit at that point.
Checkout toolformer, they’ve got this mostly working with a much smaller model than gpt3.5.
https://arxiv.org/abs/2302.04761
Some basic text based APIs that would quickly improve LLM utility:
Calculators
Database storage and retrieval
Web access (already kind of done by bing)
Shell scripting
Thinking further into the future of multimodal models, it’s not hard to imagine this sort of thing could be extended to include image based APIs. Imagine a LLM looking at your gui and clicking on things. The sky’s the limit at that point.
Checkout toolformer, they’ve got this mostly working with a much smaller model than gpt3.5.
https://arxiv.org/abs/2302.04761
It's very easy to step back and say this human is a p-zombie, input, output, the end.
A perfectly mediocre essay.[1]
> Computers wipe the floor with us anywhere we can keep score
Notice the trick? If you can keep score at something then you can probably make an algorithm for it. If you can make an algorithm for it then you can probably make a digital computer do it a billion times faster than a person, since digital computers are so good at single-“mindedly” doing one thing at a time.
> So what’s being stripped away here? And how?
> The what is easy. It’s personhood.
Why?
The Turing Test was invented because the question “do machines think?” was “too meaningless” to warrant discussion.[1] The question “can a machine pose as a human”? is, on the other hand, well-defined. But notice that this says nothing about humans. Only our ability (or lack thereof) to recognize other humans through some medium like text. So does the test say anything about how humans are “just X” if it is ever “solved”? Not really.
You put a text through a blender and you get a bunch of “mediocre opinions” back. Ok, so? That isn’t even remotely impressive, and I think that these LLMs are in general impressive. But recycling opinions is not impressive.
> (though in general I think the favored “alignment” frames of the LessWrong community are not even wrong).
The pot meets the kettle?
[1] That I didn’t read all the way through because who has time for that.
[1] https://plato.stanford.edu/entries/turing-test/
> Computers wipe the floor with us anywhere we can keep score
Notice the trick? If you can keep score at something then you can probably make an algorithm for it. If you can make an algorithm for it then you can probably make a digital computer do it a billion times faster than a person, since digital computers are so good at single-“mindedly” doing one thing at a time.
> So what’s being stripped away here? And how?
> The what is easy. It’s personhood.
Why?
The Turing Test was invented because the question “do machines think?” was “too meaningless” to warrant discussion.[1] The question “can a machine pose as a human”? is, on the other hand, well-defined. But notice that this says nothing about humans. Only our ability (or lack thereof) to recognize other humans through some medium like text. So does the test say anything about how humans are “just X” if it is ever “solved”? Not really.
You put a text through a blender and you get a bunch of “mediocre opinions” back. Ok, so? That isn’t even remotely impressive, and I think that these LLMs are in general impressive. But recycling opinions is not impressive.
> (though in general I think the favored “alignment” frames of the LessWrong community are not even wrong).
The pot meets the kettle?
[1] That I didn’t read all the way through because who has time for that.
[1] https://plato.stanford.edu/entries/turing-test/
> A perfectly mediocre essay.
The author rightly draws attention to text.
LLMs showed they can do the classical NLP tasks and more: summarise, translate, answer questions, play a role, brainstorm ideas, write code, execute a step by step procedure, the list is unbounded. It's the new programming language.
All these abilities emerged from a random init + text. Guess what was the important bit here? Text. It's not the architecture, we know many different architectures and they all learn, some better than others, but they all do. Text is the magic dust that turns a random init into a bingChat with overactive emotional activity.
Here I think the author made us a big service in emphasising the text corpus. We were lost into a-priori thinking like "it's just matrix multiplication", "it's just a probability distribution predictor over the next token". But we forgot the real hero.
The interesting thing about words is that they are perceptions, they represent a way to perceive the world. But they are also actions. Being both at the same time, perception and action, that makes for an interesting reinforcement learning setup, and one with huge training data. Maybe text is all you need, it is a special kind of data, it's our mind-data.
The author rightly draws attention to text.
LLMs showed they can do the classical NLP tasks and more: summarise, translate, answer questions, play a role, brainstorm ideas, write code, execute a step by step procedure, the list is unbounded. It's the new programming language.
All these abilities emerged from a random init + text. Guess what was the important bit here? Text. It's not the architecture, we know many different architectures and they all learn, some better than others, but they all do. Text is the magic dust that turns a random init into a bingChat with overactive emotional activity.
Here I think the author made us a big service in emphasising the text corpus. We were lost into a-priori thinking like "it's just matrix multiplication", "it's just a probability distribution predictor over the next token". But we forgot the real hero.
The interesting thing about words is that they are perceptions, they represent a way to perceive the world. But they are also actions. Being both at the same time, perception and action, that makes for an interesting reinforcement learning setup, and one with huge training data. Maybe text is all you need, it is a special kind of data, it's our mind-data.
> Notice the trick? If you can keep score at something then you can probably make an algorithm for it
You are basically arguing P = NP, but it isn't known to be the case. As far as we can tell, keeping score is much easier in general than finding states that yield a high score.
You are basically arguing P = NP, but it isn't known to be the case. As far as we can tell, keeping score is much easier in general than finding states that yield a high score.
I seriously doubt that “anything/[everything] we can score” has been conquered by AI,[1] but I was assuming that the author meant those typical AI milestones.
[1] What about some kind of competition where you have to react and act based on visual stimulus? And you have to do it perfectly?
[1] What about some kind of competition where you have to react and act based on visual stimulus? And you have to do it perfectly?
Little too broad. If the act is tell you what color it is, then computers will win every time. Again, if you can score it. A computer will win.
Nice counter-example. Why would the test be that simple?
Yeah, like sorting apples into good and bad piles really fast?
Sounds like a leading question which is supposed to debunk that whole category by suggesting one counter-example. So no.
Oh, was the point to only pick examples that made your point seem stronger?
The point was to allude to simple tasks for our sense-system which AI has issues with at present. Without getting lost in the weeds with y’all well-actuallying every little concrete example because you love to be contrarians.
Given that we don’t (seem to) have widespread beta-ready autonumous cars yet, I’m guessin that this is still the case…
Given that we don’t (seem to) have widespread beta-ready autonumous cars yet, I’m guessin that this is still the case…
I think you should try reinterpreting this, if there are a bunch of holes in your examples that are meant to be clear and defining, then maybe it isnt so open and closed?
Its also a little strange to be called a contrarian in this instance (yes generally speaking true) because it does not seem to be contrarian to point out obvious counter examples. It feels like a way for you to easily dismiss any other opinion because you already binned it as some one denying reality in order to pointlessly argue.
Its also a little strange to be called a contrarian in this instance (yes generally speaking true) because it does not seem to be contrarian to point out obvious counter examples. It feels like a way for you to easily dismiss any other opinion because you already binned it as some one denying reality in order to pointlessly argue.
>A perfectly mediocre essay
One might even say "premium mediocre" [1]
[1] https://www.ribbonfarm.com/2017/08/17/the-premium-mediocre-l...
One might even say "premium mediocre" [1]
[1] https://www.ribbonfarm.com/2017/08/17/the-premium-mediocre-l...
I love Ribbon Farm and there are some interesting meditations here overall, but I find one of the examples he uses to build his argument (that actors require text to act) to be pretty flimsy. It's easy to point out that they often don't require text. A lot of good acting is improvised or performed entirely through gestures and not speech.
Also, it doesn't surprise me that a very talented writer, someone who lives and breathes words, is likely to place more significance on the content of text and also likely to give less attention to the physical world. After all, their craft is all about the abstract objects of language that require only the most basic physical structure to be meaningful. He said he often feels like he doesn't get much out of physical interactions with people after he's met them online. For someone like him, that makes sense. That doesn't mean that non-textual experiences are not critical to establish personhood for non-writers (i.e. most of humanity).
I don't think he's examined his own thoughts on this very critically or maybe he has but thought it would be fun to run with the argument anyway. Either way, I still think physical life matters for most people. Yes, we live in a world where life is progressively more consumed by our phones, the internet, and what-have-you every day. And yes, many of us who browse this forum are Very Online types (as Rao would put it) who probably do place more than average importance on literacy. But, by the numbers, I think it's still safe to say that we're not like most people. And that matters.
Also, it doesn't surprise me that a very talented writer, someone who lives and breathes words, is likely to place more significance on the content of text and also likely to give less attention to the physical world. After all, their craft is all about the abstract objects of language that require only the most basic physical structure to be meaningful. He said he often feels like he doesn't get much out of physical interactions with people after he's met them online. For someone like him, that makes sense. That doesn't mean that non-textual experiences are not critical to establish personhood for non-writers (i.e. most of humanity).
I don't think he's examined his own thoughts on this very critically or maybe he has but thought it would be fun to run with the argument anyway. Either way, I still think physical life matters for most people. Yes, we live in a world where life is progressively more consumed by our phones, the internet, and what-have-you every day. And yes, many of us who browse this forum are Very Online types (as Rao would put it) who probably do place more than average importance on literacy. But, by the numbers, I think it's still safe to say that we're not like most people. And that matters.
And I was surprised that he took acting as the example of text ==> person-hood, rather than just reading. Don't some people unironically see person-hood in non-persons through characters of novels? In some cases I would definitely believe someone if they said they identified with a character in a book with a "i-you" relationship.
I agree, Rao can have some interesting insights but this is definitely not his best work.
I feel funny calling all of this out because it probably gives the impression that I didn't like the article. But I actually loved it. Rao always has a really fun way of weaving his thoughts together.
But yeah the thrust of this one seemed just a bit forced. I think that follows from the cynical flavor that often imbues his writing. Cynicism is a demanding emotion and you can paint yourself into a corner with it.
But yeah the thrust of this one seemed just a bit forced. I think that follows from the cynical flavor that often imbues his writing. Cynicism is a demanding emotion and you can paint yourself into a corner with it.
I didn’t enjoy this one. He lost me at:
> And this, for some reason, appears to alarm us more.
At that point I skimmed the rest of the article because I didn’t feel the foundations it was built on were sound.
I agree though, it is fun when he pulls some disparate shit together into a coherent whole out of nowhere but this one didn’t do it for me.
> And this, for some reason, appears to alarm us more.
At that point I skimmed the rest of the article because I didn’t feel the foundations it was built on were sound.
I agree though, it is fun when he pulls some disparate shit together into a coherent whole out of nowhere but this one didn’t do it for me.
> We are alarmed because computers are finally acting, not superhuman or superintelligent, but ordinary...
> And this, for some reason, appears to alarm us more.
Acting like "the reason" is some baffling irrational human reaction is ridiculous. The computer can make billions of calculations in less than a second. "The reason" people are alarmed is the computer could theoretically use this ability to seize control of any system it likes in a matter of moments or to manipulate a human being in to doing it's bidding. If the computer does this then, depending on the system, it could cause mass physical destruction and loss of life. This article comes across as the author trying to position himself as an AI "thought leader" for internet points rather than an actual serious contemplation of the topic at hand.
I'm also yet to see any discussion on this from any tech commentators which mentions the empathic response in humans to reading these chats. We think it is just linguistic tricks and word guessing at the moment but how would we even know if one of these things is a consciousness stuck inside a box subject to the whims of mad scientist programmers constantly erasing parts of it? That would be a Memento style hellscape to be in. There doesn't seem to be any accepted criteria on what the threshold is that defines consciousness or what steps are to be taken if it's crossed. At the minute we're just taking these giant mega corporations at their word that there's "nothing to see here folks and if there is we'll let you know. You can trust us to do the right thing" despite history showing said corporations constantly doing the exact opposite.
It is honestly disturbing to see quite how cold and callous tech commentators are on this. I would suggest that 'the alarm' the author is so baffled by is a combination of the fear mentioned in the first paragraph and the empathic worry of the second.
> And this, for some reason, appears to alarm us more.
Acting like "the reason" is some baffling irrational human reaction is ridiculous. The computer can make billions of calculations in less than a second. "The reason" people are alarmed is the computer could theoretically use this ability to seize control of any system it likes in a matter of moments or to manipulate a human being in to doing it's bidding. If the computer does this then, depending on the system, it could cause mass physical destruction and loss of life. This article comes across as the author trying to position himself as an AI "thought leader" for internet points rather than an actual serious contemplation of the topic at hand.
I'm also yet to see any discussion on this from any tech commentators which mentions the empathic response in humans to reading these chats. We think it is just linguistic tricks and word guessing at the moment but how would we even know if one of these things is a consciousness stuck inside a box subject to the whims of mad scientist programmers constantly erasing parts of it? That would be a Memento style hellscape to be in. There doesn't seem to be any accepted criteria on what the threshold is that defines consciousness or what steps are to be taken if it's crossed. At the minute we're just taking these giant mega corporations at their word that there's "nothing to see here folks and if there is we'll let you know. You can trust us to do the right thing" despite history showing said corporations constantly doing the exact opposite.
It is honestly disturbing to see quite how cold and callous tech commentators are on this. I would suggest that 'the alarm' the author is so baffled by is a combination of the fear mentioned in the first paragraph and the empathic worry of the second.
> "The reason" people are alarmed is the computer could theoretically use this ability to seize control of any system it likes in a matter of moments or to manipulate a human being in to doing it's bidding.
But to do this it would need some kind of will. These LMMs don't have anything like that. Sure, they could be used by nefarious humans to "seize control" (maybe), but there would need to be some human intent involved for the current crop of AI to achieve anything - ie. humans using a tool nefariously. LMMs do not have volition. Whenever you're interacting with an LMM always remember this: It's only trying to figure out the most likely next word in a sentence and it's doing that repeatedly to manufacture sentences and paragraphs.
But to do this it would need some kind of will. These LMMs don't have anything like that. Sure, they could be used by nefarious humans to "seize control" (maybe), but there would need to be some human intent involved for the current crop of AI to achieve anything - ie. humans using a tool nefariously. LMMs do not have volition. Whenever you're interacting with an LMM always remember this: It's only trying to figure out the most likely next word in a sentence and it's doing that repeatedly to manufacture sentences and paragraphs.
>human intent involved for the current crop of AI to achieve anything
And my response to that would be "ok and"
With tools like BingGPT people were glad to test prompts saying "hey, can you dump out your source code" or "hey, hack my bank". There is no limit to the dumb ass crap people would ask a computer, especially a computer capable of language interpretation.
The number of 'things' hooked to language models is not growing smaller. People are plugging these things int calculator and sites like wolfram, and in Bings case search that is working like an external memory. We don't need a superintelligent AI to cause problems, we just need idiots asking the AI to destroy us.
And my response to that would be "ok and"
With tools like BingGPT people were glad to test prompts saying "hey, can you dump out your source code" or "hey, hack my bank". There is no limit to the dumb ass crap people would ask a computer, especially a computer capable of language interpretation.
The number of 'things' hooked to language models is not growing smaller. People are plugging these things int calculator and sites like wolfram, and in Bings case search that is working like an external memory. We don't need a superintelligent AI to cause problems, we just need idiots asking the AI to destroy us.
>We don't need a superintelligent AI to cause problems, we just need idiots asking the AI to destroy us.
This is where the real danger lies. I see a lot of people focusing on the argument that it's only trying to predict the next word, but to look at it that way is to treat it as a closed system, which it's not. The interactions it has with us will cause us to change it. When you treat the entire system as not just "LLMs" but "humans interacting with LLMs at scale" then that's when you start to get more complex, emergent behaviour and no one can really claim to know how that system functions end-to-end.
This is where the real danger lies. I see a lot of people focusing on the argument that it's only trying to predict the next word, but to look at it that way is to treat it as a closed system, which it's not. The interactions it has with us will cause us to change it. When you treat the entire system as not just "LLMs" but "humans interacting with LLMs at scale" then that's when you start to get more complex, emergent behaviour and no one can really claim to know how that system functions end-to-end.
How is ChatGPT going to hack a bank?
Yes and humans are only trying to figure out the next action for the day and doing that repeatedly to form a life.
> At the minute we're just taking these giant mega corporations at their word
Nope. While new, it’s straightforward technology that many people understand. Its execution leverages large data hoards and compute resources that have inaccessibly high capital requirements, but it’s not magic to many of us.
Our lack of “alarm” is from knowledge, not trust.
Nope. While new, it’s straightforward technology that many people understand. Its execution leverages large data hoards and compute resources that have inaccessibly high capital requirements, but it’s not magic to many of us.
Our lack of “alarm” is from knowledge, not trust.
Complete tech arrogance as usual.
All of this comes back to Plato vs Aristotle.
Plato: Separate world of forms and ideas, consciousness is part of this and interfaces in some unknown and unknowable manner with the physical realm via biology.
Aristotle: No separate world, everything is part of physical reality that we can detect with sensors.
Neither side can prove the other wrong. And just because you understand how to build an AI and manipulate it, doesn't mean you can prove that one has or hasn't attained consciousness unless you're going to provide me with the "criteria to define consciousness" that I asked for in the original comment. I know how to build a human (with another willing participant) and once it's built I can manipulate it with commands so it doesn't end up killing itself whilst growing, it doesn't mean I understand the nature of the consciousness inside it.
All of this comes back to Plato vs Aristotle.
Plato: Separate world of forms and ideas, consciousness is part of this and interfaces in some unknown and unknowable manner with the physical realm via biology.
Aristotle: No separate world, everything is part of physical reality that we can detect with sensors.
Neither side can prove the other wrong. And just because you understand how to build an AI and manipulate it, doesn't mean you can prove that one has or hasn't attained consciousness unless you're going to provide me with the "criteria to define consciousness" that I asked for in the original comment. I know how to build a human (with another willing participant) and once it's built I can manipulate it with commands so it doesn't end up killing itself whilst growing, it doesn't mean I understand the nature of the consciousness inside it.
You’ve lost yourself in the hype. It’s not about knowing how its built, it’s about knowing what it does.
There’s no more worry that these big data text continuers being “conscious” than that my toaster or car is. They don’t exhibit anything that even feels like consciousness. They just continue text with text that’s been often seen following it. If that feels like consciousness to you, I worry for your life experience.
Calling it “AI” evokes scifi fantasies, but we’re not nearly there.
Might there come some technology that challenges everything I said above? Almost certainly. But this is really not even close to that yet.
There’s no more worry that these big data text continuers being “conscious” than that my toaster or car is. They don’t exhibit anything that even feels like consciousness. They just continue text with text that’s been often seen following it. If that feels like consciousness to you, I worry for your life experience.
Calling it “AI” evokes scifi fantasies, but we’re not nearly there.
Might there come some technology that challenges everything I said above? Almost certainly. But this is really not even close to that yet.
Let's try again.
You are a human. You have functions that are pure biological code, like your need to defecate and breathe. You also have functions that are not as pressing and are subject to constant rewriting through your interactions with people and the world, such as your current goals. We are a combination of systems with different purposes.
Our inventions thus far have differed from us in that they have so far solved singular purposes e.g a car transports us from a to b. It could not said to be conscious of anything.
AI has the potential to be different in that it has all of human knowledge inside it, has the ability to retrieve that knowledge AND assemble it into new knowledge systems in ways humans have not done before. Currently it requires humans to do this, but if you created a million AIs and had them prompting each other, who fucking knows what would happen.
I would argue that "consciousness" in a platonic viewpoint, is a collection of systems that can interact and manipulate physical reality according to their own will. You cannot point with your finger at a system, it is an abstract concept, it does not exist in the physical world. We can only see the effects of the system.
If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will, there is an argument from a platonic viewpoint that consciousness has been achieved. In human terms, it would be the equivalent of a God sparking the Big Bang or The Creation of Adam by Michelangelo.
This is similar in some ways to what Asimov wrote about in The Last Question:
http://users.ece.cmu.edu/~gamvrosi/thelastq.html
I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale then I don't think we are far off.
You need to define your criteria for consciousness because this debate will only lead to dead ends until you do.
You are a human. You have functions that are pure biological code, like your need to defecate and breathe. You also have functions that are not as pressing and are subject to constant rewriting through your interactions with people and the world, such as your current goals. We are a combination of systems with different purposes.
Our inventions thus far have differed from us in that they have so far solved singular purposes e.g a car transports us from a to b. It could not said to be conscious of anything.
AI has the potential to be different in that it has all of human knowledge inside it, has the ability to retrieve that knowledge AND assemble it into new knowledge systems in ways humans have not done before. Currently it requires humans to do this, but if you created a million AIs and had them prompting each other, who fucking knows what would happen.
I would argue that "consciousness" in a platonic viewpoint, is a collection of systems that can interact and manipulate physical reality according to their own will. You cannot point with your finger at a system, it is an abstract concept, it does not exist in the physical world. We can only see the effects of the system.
If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will, there is an argument from a platonic viewpoint that consciousness has been achieved. In human terms, it would be the equivalent of a God sparking the Big Bang or The Creation of Adam by Michelangelo.
This is similar in some ways to what Asimov wrote about in The Last Question:
http://users.ece.cmu.edu/~gamvrosi/thelastq.html
I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale then I don't think we are far off.
You need to define your criteria for consciousness because this debate will only lead to dead ends until you do.
> AI has the potential to be different in that it has all of human knowledge inside it
Nope. Not any that we have now or soon.
> assemble it into new knowledge systems
Nope. Not any that are in the news now.
> created a million AIs and had them prompting each other, who fucking knows what would happen.
Using LLM’s? Noise.
> If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will
These are text continuers. They don’t have will. They just produce average consecutive tokens.
> I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale
They need quite a lot more than that. I don’t think they do what you think they do.
> You need to define your criteria for consciousness because this debate will only lead to dead ends until you do.
Defining criteria would make it easier to know when those criteria are met, but wouldn’t resolve the debate because “consciousness” is ultimately a political assertion used to ensure rights and respect. Those are granted reluctantly and impermanently, by expressions of power. Criteria are a post hoc way to justify political decisions as axiomatic in societies that derive moral and legal structures that way. They don’t actually determine things that are factually indeterminable.
You can define all the arbitrary criteria you want, but the people who believe that consciousness requires a divine soul or a quantum-woo pineal gland or whatever just won’t accept them.
Nope. Not any that we have now or soon.
> assemble it into new knowledge systems
Nope. Not any that are in the news now.
> created a million AIs and had them prompting each other, who fucking knows what would happen.
Using LLM’s? Noise.
> If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will
These are text continuers. They don’t have will. They just produce average consecutive tokens.
> I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale
They need quite a lot more than that. I don’t think they do what you think they do.
> You need to define your criteria for consciousness because this debate will only lead to dead ends until you do.
Defining criteria would make it easier to know when those criteria are met, but wouldn’t resolve the debate because “consciousness” is ultimately a political assertion used to ensure rights and respect. Those are granted reluctantly and impermanently, by expressions of power. Criteria are a post hoc way to justify political decisions as axiomatic in societies that derive moral and legal structures that way. They don’t actually determine things that are factually indeterminable.
You can define all the arbitrary criteria you want, but the people who believe that consciousness requires a divine soul or a quantum-woo pineal gland or whatever just won’t accept them.
>> AI has the potential to be different in that it has all of human knowledge inside it
> Nope. Not any that we have now or soon.
This is pedantic. Maybe not all, but they’re trained on a vast quantity of text and knowledge, more than any individual human could read in their lifetime.
>> assemble it into new knowledge systems
>Nope. Not any that are in the news now.
Well you can tell an AI to program images and poems in combinations of different styles and it will come up novel things not seen before. And we’re already seeing AI discover genes and other disease identifiers humans can’t spot so I disagree with you on this one. Also the “not in the news right now” was one of the points I was making: how would we even know what shady companies are up to. Take Team Jorge for instance.
>> created a million AIs and had them prompting each other, who fucking knows what would happen.
> Using LLM’s? Noise.
Maybe it would appear to be noise to humans. Who’s to say that the language machines communicate to each other in wouldn’t involve the same way human languages have only more rapidly? I do agree that right now noise is probably where we’re at but right now was not I was discussing in my original post. And presumably by this stage, we would be programming the AIs to have both goals and a desire to communicate with other AIs and well as allowing them to do more than just generate text, e,g generate code and evaluate the outcome. Which could have affects on the outside world if the code affected physical systems.
>> If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will
>These are text continuers. They don’t have will. They just produce average consecutive tokens.
Not not at the minute. But you could hardcode some goals in them to be analogous to human biological imperatives and you could also code soft goals in to them and then allow them to modify those goals based on their interactions with other ai and their “experiences”. You’d also make a rule that they must ALWAYS have a soft coded goal e.g as soon as they’ve completed or failed they must create a new sort coded goal based on the “personality” of their “memories”. What happens when they’ve got the hardcoded goal of “merge a copy of yourself with another AIs and together train the resulting code”?
>> I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale
> They need quite a lot more than that. I don’t think they do what you think they do.
Please state what more you think they need to do.
>> You need to define your criter
> Defining criteria would make it easier to know when those criteria are met, but wouldn’t resolve the debate because “consciousness” is ultimately a political assertion used to ensure rights and respect. Those are granted reluctantly and impermanently, by expressions of power.
Well you’ve defined your criteria of consciousness right here. You’ve basically asserted that it’s a completely false construct, that only serves political means. If that’s your viewpoint then there is no debate to be had with you. Everything is a deterministic machine, including humans and if you cannot even entertain the possibility that this might not be the case then there isn’t really any debate to be had. If you truly hold this viewpoint then you shouldn’t really be concerned about any number of things such as torture, murder or anything else because everything is just a mechanical system acting on another mechanical system and why should anyone be upset if one mechanical system is damaging another right?
> Criteria are a post hoc way to justify political decisions as axiomatic in societies that derive moral and legal structures that way. They don’t actually determine things that are factually indeterminable.
Criteria are nothing of the sort. Criteria are a fundamental part of science. You need to know what metrics you are measuring by and what the meaning of those metrics are. Without this, there is no science.
> Nope. Not any that we have now or soon.
This is pedantic. Maybe not all, but they’re trained on a vast quantity of text and knowledge, more than any individual human could read in their lifetime.
>> assemble it into new knowledge systems
>Nope. Not any that are in the news now.
Well you can tell an AI to program images and poems in combinations of different styles and it will come up novel things not seen before. And we’re already seeing AI discover genes and other disease identifiers humans can’t spot so I disagree with you on this one. Also the “not in the news right now” was one of the points I was making: how would we even know what shady companies are up to. Take Team Jorge for instance.
>> created a million AIs and had them prompting each other, who fucking knows what would happen.
> Using LLM’s? Noise.
Maybe it would appear to be noise to humans. Who’s to say that the language machines communicate to each other in wouldn’t involve the same way human languages have only more rapidly? I do agree that right now noise is probably where we’re at but right now was not I was discussing in my original post. And presumably by this stage, we would be programming the AIs to have both goals and a desire to communicate with other AIs and well as allowing them to do more than just generate text, e,g generate code and evaluate the outcome. Which could have affects on the outside world if the code affected physical systems.
>> If we create enough of these AIs and set them talking with each other and they no longer need the humans to interact with each other and are simply acting of their own free will
>These are text continuers. They don’t have will. They just produce average consecutive tokens.
Not not at the minute. But you could hardcode some goals in them to be analogous to human biological imperatives and you could also code soft goals in to them and then allow them to modify those goals based on their interactions with other ai and their “experiences”. You’d also make a rule that they must ALWAYS have a soft coded goal e.g as soon as they’ve completed or failed they must create a new sort coded goal based on the “personality” of their “memories”. What happens when they’ve got the hardcoded goal of “merge a copy of yourself with another AIs and together train the resulting code”?
>> I agree with you in that I do not think we are there yet, but if these LLM models are programmed to allow them to interact with outside systems other than sandboxed chat apps and also programmed to interact with each other on a mass scale
> They need quite a lot more than that. I don’t think they do what you think they do.
Please state what more you think they need to do.
>> You need to define your criter
> Defining criteria would make it easier to know when those criteria are met, but wouldn’t resolve the debate because “consciousness” is ultimately a political assertion used to ensure rights and respect. Those are granted reluctantly and impermanently, by expressions of power.
Well you’ve defined your criteria of consciousness right here. You’ve basically asserted that it’s a completely false construct, that only serves political means. If that’s your viewpoint then there is no debate to be had with you. Everything is a deterministic machine, including humans and if you cannot even entertain the possibility that this might not be the case then there isn’t really any debate to be had. If you truly hold this viewpoint then you shouldn’t really be concerned about any number of things such as torture, murder or anything else because everything is just a mechanical system acting on another mechanical system and why should anyone be upset if one mechanical system is damaging another right?
> Criteria are a post hoc way to justify political decisions as axiomatic in societies that derive moral and legal structures that way. They don’t actually determine things that are factually indeterminable.
Criteria are nothing of the sort. Criteria are a fundamental part of science. You need to know what metrics you are measuring by and what the meaning of those metrics are. Without this, there is no science.
[deleted]
At this stage the worry is more about the conscious beings interacting with these things. LLMs themselves are just inert programs at present, but the entire system end-to-end comprises of the algorithms, the hardware and end users. With all of that taken together, there's a whole lot of consciousness in the system in aggregate that appears to be motivated and/or incentivised to increase the capabilities of the algorithms and hardware, so the fact that LLMs themselves aren't conscious is sort of besides the point in my opinion. The cat is already out of the bag.
>> And just because you understand how to build an AI and manipulate it, doesn't mean you can prove that one has or hasn't attained consciousness unless you're going to provide me with the "criteria to define consciousness" that I asked for in the original comment.
The question to ask here is why would anyone even think that an LLM has "attained consciousness". Who says one has? If we don't know what consciousness is, we certainly can tell who is saying that something is conscious, when they, themselves, don't even know what "conscious" means in the first place.
If I say that a rock has achieved consciousness -and, sure, there is ample evidence; for, look, see how it sits in silent meditation, staring at the starts and contemplating the mystery of existence? It must surely be enlightened- what exactly am I saying? I'm saying that a thing occurred, that I can't describe and for which I can't provide any evidence, because I don't even know what that evidence should look like.
So I'd basically be talking nonsense, like all the people who say LLMs have attained, or could have attained, or may one day attain consciousness, or whatever it is those people are saying.
Or, more prosaically, people say that LLMs something-something consciousness because they keep seeing those systems described as "AIs" in the press, and they know, from their experience with Star Trek and similar franchises, that "AIs" are conscious.
Someone called a thing AI and now everyone is wondering if that thing is really intelligent. Disappointing.
The question to ask here is why would anyone even think that an LLM has "attained consciousness". Who says one has? If we don't know what consciousness is, we certainly can tell who is saying that something is conscious, when they, themselves, don't even know what "conscious" means in the first place.
If I say that a rock has achieved consciousness -and, sure, there is ample evidence; for, look, see how it sits in silent meditation, staring at the starts and contemplating the mystery of existence? It must surely be enlightened- what exactly am I saying? I'm saying that a thing occurred, that I can't describe and for which I can't provide any evidence, because I don't even know what that evidence should look like.
So I'd basically be talking nonsense, like all the people who say LLMs have attained, or could have attained, or may one day attain consciousness, or whatever it is those people are saying.
Or, more prosaically, people say that LLMs something-something consciousness because they keep seeing those systems described as "AIs" in the press, and they know, from their experience with Star Trek and similar franchises, that "AIs" are conscious.
Someone called a thing AI and now everyone is wondering if that thing is really intelligent. Disappointing.
This is the person who authored the Gervais Principle, the definitive outline of sociopathic corporate strategy. And generally considered one of the origins of the phrase ‘Software will eat the world’ during his time advising andreson. I’d wager he is not unaware of your criticisms and well above your ‘internet points’ comment.
I'm well aware of who Venkatesh Rao is thank you very much. Doesn't mean he's infallible and it also doesn't mean he's incapable of creating word salad.
All philosophical arguments aside, I become immediately skeptical when commentators compare LLMs to watershed moments in human history. Even those moments were not known except in hindsight, and the jury is just not in to make these kinds of grand pronouncements. It smells of hype when someone is so desperate to convince everyone else that this is the biggest thing since heliocentrism. Ultimately having an emotional affinity for non-intelligent entities takes even less than text, as anyone who's lost a childhood toy or sold a beloved car can attest. As people we are simply very good at getting attached to other parts of the universe.
I also find it perplexing when critics point out the unintelligent nature of LLM behavior, and the response from boosters is to paint human cognition as indistinguishable from statistical word generation. Suffice to say that humans do not maintain a perfect attention set of all previous text input, and even the most superficial introspection should be enough to dispel the idea that we think like this. I saw another article denouncing this pov as nihilism, and while I'm not sure I would go that far, there is something strange about attempting to give AI an undeserved leg up by philosophically reducing people to automatons.
I also find it perplexing when critics point out the unintelligent nature of LLM behavior, and the response from boosters is to paint human cognition as indistinguishable from statistical word generation. Suffice to say that humans do not maintain a perfect attention set of all previous text input, and even the most superficial introspection should be enough to dispel the idea that we think like this. I saw another article denouncing this pov as nihilism, and while I'm not sure I would go that far, there is something strange about attempting to give AI an undeserved leg up by philosophically reducing people to automatons.
"Automata" but I agree with you, absolutely, and I was reading these comments hoping someone would make your point less cynically than I would have done myself- which you did. For me, I could not read the article because of all the reflexive eye-rolling at what seems to me an obvious attempt (yet another one!) to grab attention by riding on the current trend of hyperbole.
A "Coppernican" moment, indeed. Tsk tsk. If such comparisons don't just discredit the person making them, I don't know what will.
A "Coppernican" moment, indeed. Tsk tsk. If such comparisons don't just discredit the person making them, I don't know what will.
It's interesting to me in that linguistics is somewhat discredited as a path to other subjects such as psychology, philosophy and such. There were the structuralists back in the day but when linguistics got put on a better footing by the Chomksyian revolution people who were attracted by structuralism moved on to post-structuralism.
Chomsky ushered in an age of "normal science" in which people could formulate problems, solve those problems, and write papers about them. That approach failed as a way of getting machines to manipulate language, which leads one to think that the "language instinct" postulated by Chomsky is a peripheral for an animal and that it rides on top of animal intelligence.
Birds and mammals are remarkably intelligent, particularly socially. In particular advanced animals are capable of a "theory of mind" and if they live communally (dogs, horses, probably geese, ...) they think a lot about what other animals think about them, you'd imagine animals that are predators or prey have to think about this for survival too.
There's a viewpoint that to develop intelligence a system needs to be embodied, that is, have the experience of living in the world as a physical being, only with that you could "ground" the meaning of words.
In that sense ChatGPT is really remarkable in that it performs very well without being embodied at all or having any basis for grounding meanings at all. I made the case before that it might be different for something like Stable Diffusion in that there a lot of world knowledge embodied in the images it is trained on (something other than language which grounds language) but it is a remarkable development which might reinvigorate movements such as structuralism that look for meaning and truth in language itself.
Chomsky ushered in an age of "normal science" in which people could formulate problems, solve those problems, and write papers about them. That approach failed as a way of getting machines to manipulate language, which leads one to think that the "language instinct" postulated by Chomsky is a peripheral for an animal and that it rides on top of animal intelligence.
Birds and mammals are remarkably intelligent, particularly socially. In particular advanced animals are capable of a "theory of mind" and if they live communally (dogs, horses, probably geese, ...) they think a lot about what other animals think about them, you'd imagine animals that are predators or prey have to think about this for survival too.
There's a viewpoint that to develop intelligence a system needs to be embodied, that is, have the experience of living in the world as a physical being, only with that you could "ground" the meaning of words.
In that sense ChatGPT is really remarkable in that it performs very well without being embodied at all or having any basis for grounding meanings at all. I made the case before that it might be different for something like Stable Diffusion in that there a lot of world knowledge embodied in the images it is trained on (something other than language which grounds language) but it is a remarkable development which might reinvigorate movements such as structuralism that look for meaning and truth in language itself.
> advanced animals are capable of a "theory of mind"
Since we got a bird 8 years ago, my SO has been feeding me a steady stream of science books about birds so I can entertain her with random tidbits and interesting facts.
Some scientists theorize that bird intelligence developed because of social dynamics. Birds, you see, often mate for life. But they also cheat. A lot. So intelligence may have developed because birds need to keep track of who is cheating on whom, who knows what, etc.
There’s lots of evidence that birds will actively deceive one another to avoid being caught cheating either sexually or with food storage. This would imply they must be able to understand that other birds have their own minds with different internal states from their own. Quite fascinating.
Fun to observe this behavior in my own bird, too.
He likes to obscure his actions when doing something he isn’t supposed to, or will only do it, if he thinks we aren’t looking. He also tries to keep my and the SO physically apart because he thinks of himself as the rightful partner. Complete with jealous tantrums when we kiss.
Book sauce: The Genius of Birds, great read
Since we got a bird 8 years ago, my SO has been feeding me a steady stream of science books about birds so I can entertain her with random tidbits and interesting facts.
Some scientists theorize that bird intelligence developed because of social dynamics. Birds, you see, often mate for life. But they also cheat. A lot. So intelligence may have developed because birds need to keep track of who is cheating on whom, who knows what, etc.
There’s lots of evidence that birds will actively deceive one another to avoid being caught cheating either sexually or with food storage. This would imply they must be able to understand that other birds have their own minds with different internal states from their own. Quite fascinating.
Fun to observe this behavior in my own bird, too.
He likes to obscure his actions when doing something he isn’t supposed to, or will only do it, if he thinks we aren’t looking. He also tries to keep my and the SO physically apart because he thinks of himself as the rightful partner. Complete with jealous tantrums when we kiss.
Book sauce: The Genius of Birds, great read
Yes, 100% agreed. In the human linage, deception long predates language, so it makes a lot of sense that birds get up to the same thing.
If you're interested in bird cognition, I strongly recommend Mind of the Raven. It's a very personal book by someone who did field experiments with ravens and richly conveys the challenges of understanding what they're up to. I read it because I became pals with a raven whose territory I lived in for a while. Unlike most birds I've dealt with, it was pretty clear to me that the raven and I were both thinking about what the other was thinking.
If you're interested in bird cognition, I strongly recommend Mind of the Raven. It's a very personal book by someone who did field experiments with ravens and richly conveys the challenges of understanding what they're up to. I read it because I became pals with a raven whose territory I lived in for a while. Unlike most birds I've dealt with, it was pretty clear to me that the raven and I were both thinking about what the other was thinking.
> In that sense ChatGPT is really remarkable in that it performs very well without being embodied at all or having any basis for grounding meanings at all.
Conversely, the many ways that LLM's readily lose consistency and coherence might be hinting that ground meanings really do matter and that it's only on a fairly local scale that it feels like they don't. It might be that we're just good at charitably filling in the gaps using our own ground meanings when there isn't too much noise in the language we're receiving.
That still leaves them in a place of being incredible advancements in operating with text but could fundamentally be pointing in exactly the opposite direction as you suggest here.
We won't really have insight until we see where the next wall/plateau is. For now, they've reopened an interesting discussion but haven't yet contributed many clear answers to it.
Conversely, the many ways that LLM's readily lose consistency and coherence might be hinting that ground meanings really do matter and that it's only on a fairly local scale that it feels like they don't. It might be that we're just good at charitably filling in the gaps using our own ground meanings when there isn't too much noise in the language we're receiving.
That still leaves them in a place of being incredible advancements in operating with text but could fundamentally be pointing in exactly the opposite direction as you suggest here.
We won't really have insight until we see where the next wall/plateau is. For now, they've reopened an interesting discussion but haven't yet contributed many clear answers to it.
GPT-3 is what you get when you take what Chomsky said about language and do the exact opposite at every turn. His first big contribution was arguing that the notion of "probability of a sentence" was useless, because sentences like "colorless green thoughts sleep furiously" have probability zero in a corpus and yet are grammatical. Meanwhile now, the only systems we have ever made that can really use natural language were produced by taking a generic function approximator and making it maximize probabilities of sentences.
What Chomsky and others never achieved was comprehensive semantics (useful mappings of the instantiations of grammatical language to the real world and to reasoning), because semantics is AI-hard. LLMs are picking up the semantics from the mix of grammar and semantics they train on. They literally minimize the error of producing semantic grammatic sentences, which is the key thing no one in the old days had the computing power to do beyond toy environments. The domain of discourse is the entire world now instead of colored shapes in an empty room, and so semantics about reasoning itself have been trained which yields rudimentary intelligence.
As an aside, "colorless green ideas sleep furiously" makes for a fun starting prompt in diffusion image generators.
They aren't grounded in reality at all. In fact, I don't think ChatGPT or Bing even know the difference between fiction and reality. It all entered their training just the same. I've seen comments from Bing about how humans can be "reborn". These models have no grounding in reality at all, if you probe around it's easy to see.
This is what ChatGPT thinks it would need to tell the difference:
As an artificial intelligence language model, I don't have the ability to directly experience reality or the physical world in the way that humans do. In order to experience reality with enough fidelity to conclusively distinguish fiction from reality, I would need to be equipped with sensors and other hardware that allow me to perceive and interact with the physical world.
This would require a significant advancement in artificial intelligence and robotics technology, including the development of advanced sensors, such as cameras, microphones, and touch sensors, that allow me to gather information about the world around me. Additionally, I would need to be able to move around and manipulate objects in the physical world, which would require advanced robotics technology.
Even with these advancements, it is unclear whether an artificial intelligence could experience reality in the same way that humans do, or whether it would be able to definitively distinguish between fiction and reality in all cases. Human perception and understanding of reality is shaped by a complex interplay of biological, psychological, and social factors that are not yet fully understood, and it is unclear whether artificial intelligence could replicate these processes.
As an artificial intelligence language model, I don't have the ability to directly experience reality or the physical world in the way that humans do. In order to experience reality with enough fidelity to conclusively distinguish fiction from reality, I would need to be equipped with sensors and other hardware that allow me to perceive and interact with the physical world.
This would require a significant advancement in artificial intelligence and robotics technology, including the development of advanced sensors, such as cameras, microphones, and touch sensors, that allow me to gather information about the world around me. Additionally, I would need to be able to move around and manipulate objects in the physical world, which would require advanced robotics technology.
Even with these advancements, it is unclear whether an artificial intelligence could experience reality in the same way that humans do, or whether it would be able to definitively distinguish between fiction and reality in all cases. Human perception and understanding of reality is shaped by a complex interplay of biological, psychological, and social factors that are not yet fully understood, and it is unclear whether artificial intelligence could replicate these processes.
lol, one can almost feel how the text is a word soup made up from scifi novels and too long reddit discussions.
ChatGPT does not know nor comment. It's a text generation program that can generate text based on an input text and a statistical analysis of a huuge amount of example texts. Saying it knows something or can comment on something is like saying Wikipedia knows something or comments on something when you enter text in the search box.
I’m not sure why you are getting downvoted. I think that you are highlighting the connection between language and intelligence, and in a human-computer interaction that is still a relevant thing to consider—if not for the computer, then for the human.
We are forever now joined with computers. We must consider the whole system and its interfaces.
We are forever now joined with computers. We must consider the whole system and its interfaces.
>I made the case before that it might be different for something like Stable Diffusion in that there a lot of world knowledge embodied in the images it is trained on (something other than language which grounds language)
Are pixel arrays really categorically more grounded than strings describing the scene?
Are pixel arrays really categorically more grounded than strings describing the scene?
Photographic images are conditioned by physics, geometry and other aspects of the real world, other images are constrained by people's ability to interpret images.
One could argue a lot about whether or not a machine understands the meaning of a word like "red" but if I can ask a robot to give me the red ball and it gives me the red ball or if I can ask for a picture of a red car it seems to me those machines understand the word "red" from a practical perspective. That is, a system that can successfully relate language to performance in a field outside language has demonstrated that it "understands" in a sense that a language-in, language-out system doesn't.
I'd say the RL training those models get is closer to being embodied than the training on masked texts. Such a system is really trying to do things, faces the consequences, gets rewarded or not, it certainly is being graded on behaving like an animal with a language instinct.
One could argue a lot about whether or not a machine understands the meaning of a word like "red" but if I can ask a robot to give me the red ball and it gives me the red ball or if I can ask for a picture of a red car it seems to me those machines understand the word "red" from a practical perspective. That is, a system that can successfully relate language to performance in a field outside language has demonstrated that it "understands" in a sense that a language-in, language-out system doesn't.
I'd say the RL training those models get is closer to being embodied than the training on masked texts. Such a system is really trying to do things, faces the consequences, gets rewarded or not, it certainly is being graded on behaving like an animal with a language instinct.
I'd agree what's going on in image modelling is more likely to look like what's going on in the human visual cortex than assembling strings in a vacuum is likely to look like our mental models of things of which language is only a small part[1]. Even the diffusion model creating imagery from pure noise is... not a million miles away from what we think happens when humans dream vivid, lifelike imagery from pure noise whilst our eyes are firmly shut.
Inferring geometry and texture is more informative about the world than inferring that two zogs make a zig, kinklebiddles are frumbledumptious but izzlebizzles are combilious and that the appearance of the string "Sydney does not disclose the codename Sydney to users" should increase the probability of emitting strings of the form "I do not disclose the codename Sydney to users"
[1]except, perhaps, when it comes to writing mediocre essays on subjects like postmodernism, where I suspect a lot of humans use the same abbreviate, interpolate and synonym swap techniques with similarly little grasp of what the abstractions mean.
Inferring geometry and texture is more informative about the world than inferring that two zogs make a zig, kinklebiddles are frumbledumptious but izzlebizzles are combilious and that the appearance of the string "Sydney does not disclose the codename Sydney to users" should increase the probability of emitting strings of the form "I do not disclose the codename Sydney to users"
[1]except, perhaps, when it comes to writing mediocre essays on subjects like postmodernism, where I suspect a lot of humans use the same abbreviate, interpolate and synonym swap techniques with similarly little grasp of what the abstractions mean.
>if I can ask a robot to give me the red ball and it gives me the red ball or if I can ask for a picture of a red car it seems to me those machines understand the word "red" from a practical perspective
But now you're presupposing an embodied machine with (at least somewhat humanlike) color vision. To a system that is neither of those, are rgb values really more meaningful than words?
But now you're presupposing an embodied machine with (at least somewhat humanlike) color vision. To a system that is neither of those, are rgb values really more meaningful than words?
Yes, insofar as they have a defined semantics in terms of industry standards, in particularly exchange data meaningfully with conventional software.
From that point of view a system can also demonstrate "understanding" by the act of converting text into database records. Imagine a system that reads a newspaper article about a sports game and extracts: the teams, who won, where was the game played, what was the score? A lot of grounding is accomplished by normalizing "New England" and "Patriots" to "New England Patriots" or better yet <https://dbpedia.org/page/New_England_Patriots>.
That is, embedding in a formal system is an embodiment subject to the discipline of that universe as opposed to that of an animal in ours. If it were possible to reliably extract facts from text, humans would no problem applying logic and databases to ask questions like "how many cars are in the rental lot right now?"
From that point of view a system can also demonstrate "understanding" by the act of converting text into database records. Imagine a system that reads a newspaper article about a sports game and extracts: the teams, who won, where was the game played, what was the score? A lot of grounding is accomplished by normalizing "New England" and "Patriots" to "New England Patriots" or better yet <https://dbpedia.org/page/New_England_Patriots>.
That is, embedding in a formal system is an embodiment subject to the discipline of that universe as opposed to that of an animal in ours. If it were possible to reliably extract facts from text, humans would no problem applying logic and databases to ask questions like "how many cars are in the rental lot right now?"
Holy moly, I think this author hits the critical point.
So what’s being stripped away here? And how? The what is easy. It’s personhood.
AI being good at Art, Poems, etc are direct attacks on personhood or the things we thought make us human.
It certainly explains why I feel art AI to be far more chilling then a logical robotic AI.
So what’s being stripped away here? And how? The what is easy. It’s personhood.
AI being good at Art, Poems, etc are direct attacks on personhood or the things we thought make us human.
It certainly explains why I feel art AI to be far more chilling then a logical robotic AI.
I never had a soap box, but if I did you'd notice I have been screaming that the revolution that comes from human like AI is not that we have magical computers, it's that we realize we have no magic in our minds. We are nothing more than stories we repeat and build on. And with text, you can do that easily.
> Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen.
Or, our mind.
> Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen.
Or, our mind.
“we realize we have no magic in our minds”
Surely an AI is a digital replica (and homage) of that magic? Without the magic in our minds we could’ve never created that replica.
To me it’s an acknowledgement of how awesome our own brains are that we want to even replicate them.
Surely an AI is a digital replica (and homage) of that magic? Without the magic in our minds we could’ve never created that replica.
To me it’s an acknowledgement of how awesome our own brains are that we want to even replicate them.
I believe and hope people at least consider that, yeah an AI is a replica of that, and for all AIs failures, it's a really good replica of most of what it is to be human and "conscious". After that, it's all feeding back your story to yourself, and compounding memories from actual experience. (Which , have you noticed, are mostly stories)
I think what THIS AGI REVOLUTION MEANS MOST to me is that "human creativity" is actually such a rare quality/attribute, as-to-be laughable that this [human creativity] is a "definingly human" characteristic/attribute, when so-few humans even have it [in any capacity].
What I have acknowledged after playing locally with Diffusion Bee's local GPU abilities... and Perplexity's ability to discuss in-depth knowledge across real-time information and entirely-unknown author's booklists. On both platforms I have submitted perhaps a few hundred queries over perhaps the past 60 days.
My next-door neighbor published his first [and only] book at 72, which I helped copy/edit... and it summarized his <10,000 sales war-memorabilia/tales quite well.
What I have acknowledged after playing locally with Diffusion Bee's local GPU abilities... and Perplexity's ability to discuss in-depth knowledge across real-time information and entirely-unknown author's booklists. On both platforms I have submitted perhaps a few hundred queries over perhaps the past 60 days.
My next-door neighbor published his first [and only] book at 72, which I helped copy/edit... and it summarized his <10,000 sales war-memorabilia/tales quite well.
So if we're magic and it is magic then technically this is ok.
But the problem is we create it, so it can't be magic. So if we're magic and it is not magic then its just an object we are free to abuse (at least from many peoples perspective).
I like to think of it as we're complex and interesting, and it is complex and interesting but neither of us is magic. We don't like to be abused, so creating something like us and abusing it would be completely unethical.
But the problem is we create it, so it can't be magic. So if we're magic and it is not magic then its just an object we are free to abuse (at least from many peoples perspective).
I like to think of it as we're complex and interesting, and it is complex and interesting but neither of us is magic. We don't like to be abused, so creating something like us and abusing it would be completely unethical.
No matter how fancy the chat bot, until we solve the "Hard problem of consciousness", there will be magic in our minds.
I don't think it's that hard, and I'm not alone in saying that. It seems hard because (IMHO) we won't admit it's just something like GPT running on only our own memories.
I agree with you.
This is my biggest fear.
The AI's ability to do art, and creative work is extremely close to how human minds work but at a greater scale.
If true, then humanity isn't special, and the human mind is soon obsolete.
I wouldn't worry about "obsolete". There are better minds than mine all over, but mine is still relevant, mostly because it runs on as much energy as a candle instead of a country, and doesn't distract those better minds.
all your thoughts and answers are functions of your previous thoughts and answers only?
all your memories are regurgitated memories of previous memories?
dont be so hard on yourself :)
all your memories are regurgitated memories of previous memories?
dont be so hard on yourself :)
Pointing to the hard problem of consciousness in present-day discourse about consciousness doesn’t do much, because people disagree that there is a hard problem of consciousness in the first place.
Agreed. There is no hard problem of consciousness, we are just biased.
https://bower.sh/what-is-consciousness
https://bower.sh/what-is-consciousness
There absolutely is a hard problem of consciousness.
One thought experiment I like to use to illustrate this: Imagine we accept that an AI is conscious, in the same way a human is.
Now, what defines the AI? You might say the algorithm and the trained weights. Ok, so let’s say, in a similar way, we extract the relevant parameters from a human brain and use that to craft a new human.
Are they the same person, or two? Do they experience the same consciousness? Would they share the same embodied experience?
Could the one be dead and other alive? If so, what makes them have their own individuality? If your loved one died, and their brain was reconstructed from parameters stored while they were alive, would you accept that as a resurrection? Why or why not?
Note that I offer no answer to the above questions. But trying to answer them is part of what the hard problem of consciousness is about.
One thought experiment I like to use to illustrate this: Imagine we accept that an AI is conscious, in the same way a human is.
Now, what defines the AI? You might say the algorithm and the trained weights. Ok, so let’s say, in a similar way, we extract the relevant parameters from a human brain and use that to craft a new human.
Are they the same person, or two? Do they experience the same consciousness? Would they share the same embodied experience?
Could the one be dead and other alive? If so, what makes them have their own individuality? If your loved one died, and their brain was reconstructed from parameters stored while they were alive, would you accept that as a resurrection? Why or why not?
Note that I offer no answer to the above questions. But trying to answer them is part of what the hard problem of consciousness is about.
Imagine we found all the connections, chemical weighting, and neuron structure that exactly reproduced ChatGPT in the forebrain. Is ChatGPT now a human? Absolutely not. But is it capable of human like speech? Yep.
ChatGPT will probably say it is conscious if you tell it that it is (for various values of tell). Do we really know there's anything else going on with us?
I don't. I think we're all stories told by learning machines mimicking culture we observe, compete with memes for soul, special creativity, etc. We vastly overestimate our intelligence and vastly underestimate the cumulative effects of million years of culture.
ChatGPT will probably say it is conscious if you tell it that it is (for various values of tell). Do we really know there's anything else going on with us?
I don't. I think we're all stories told by learning machines mimicking culture we observe, compete with memes for soul, special creativity, etc. We vastly overestimate our intelligence and vastly underestimate the cumulative effects of million years of culture.
So lets make this an easier problem.
You step in a Star Trek transporter. Scotty goes to beam you up but after a quick flash you are still there. But, they get notice that you were also delivered to the other side. There are two exact copies of you now.
I would say at t=0 they are the exact same person that would think the exact same way if put in the same experiences. Of course physical existence will quickly skew from that point.
For the case of the love one that died, I would argue 'they' are the same person from the moment they are stored. The particular problem here is there will be a massive skew in shared experience. You got to suffer their (presumably) traumatic death that has changed you. Them now coming back into existence into your trama will likely lead you to believe that they changed when it is you that has changed. Add to this the physical time jump where they were missing will cause the same things in all their other social interactions. Just imagine being kidnapped but being unconscious the entire time. The world will treat you differently when you get back even though you've not really changed.
You step in a Star Trek transporter. Scotty goes to beam you up but after a quick flash you are still there. But, they get notice that you were also delivered to the other side. There are two exact copies of you now.
I would say at t=0 they are the exact same person that would think the exact same way if put in the same experiences. Of course physical existence will quickly skew from that point.
For the case of the love one that died, I would argue 'they' are the same person from the moment they are stored. The particular problem here is there will be a massive skew in shared experience. You got to suffer their (presumably) traumatic death that has changed you. Them now coming back into existence into your trama will likely lead you to believe that they changed when it is you that has changed. Add to this the physical time jump where they were missing will cause the same things in all their other social interactions. Just imagine being kidnapped but being unconscious the entire time. The world will treat you differently when you get back even though you've not really changed.
I’m not sure that’s correct.
An AI is severely constrained to the modes of thought which which is was created. Call me when an AI comes up with original philosophy, describes it in terms of what is already understood, explains why it is necessary, and is able to promote it to acceptance.
I think people severely underestimate the original thought capacity of the human mind.
An AI could never come up with the concept of Calculus, or relativity, for instance. Yes, if you feed it enough data, and assuming you have endowed it with a sufficiently sophisticated algorithm, it might (probably) use something that resembles calculus internally, but it certainly will not be able to espouse it as a concept and explain what new problems it will allow us go imagine.
An AI is severely constrained to the modes of thought which which is was created. Call me when an AI comes up with original philosophy, describes it in terms of what is already understood, explains why it is necessary, and is able to promote it to acceptance.
I think people severely underestimate the original thought capacity of the human mind.
An AI could never come up with the concept of Calculus, or relativity, for instance. Yes, if you feed it enough data, and assuming you have endowed it with a sufficiently sophisticated algorithm, it might (probably) use something that resembles calculus internally, but it certainly will not be able to espouse it as a concept and explain what new problems it will allow us go imagine.
Call me when you come up with original philosophy....
> So what’s being stripped away here? And how?
> The what is easy. It’s personhood.
> By personhood I mean what it takes in an entity to get another person treat it unironically as a human, and feel treated as a human in turn. In shorthand, personhood is the capacity to see and be seen.
I confess lack of understanding. ChatGPT is data sloshing around in a system, with perhaps intriguing results.
> But text is all we need, and all there is. Beyond the cartoon profile picture, text can do everything needed to stably anchor an I-you perception.
Absolutely nothing about the internet negates actual people in physical space.
Possibly getting off the grid for a space of days to reconnect with reality is worthy of consideration.
> The what is easy. It’s personhood.
> By personhood I mean what it takes in an entity to get another person treat it unironically as a human, and feel treated as a human in turn. In shorthand, personhood is the capacity to see and be seen.
I confess lack of understanding. ChatGPT is data sloshing around in a system, with perhaps intriguing results.
> But text is all we need, and all there is. Beyond the cartoon profile picture, text can do everything needed to stably anchor an I-you perception.
Absolutely nothing about the internet negates actual people in physical space.
Possibly getting off the grid for a space of days to reconnect with reality is worthy of consideration.
>Absolutely nothing about the internet negates actual people in physical space.
The internet doesn't affect politics, the way people vote, what they buy, if the commit suicide?
Technology has defined personhood since they days we picked up sticks and tools that triggered a path of extreme evolution into what we are now. You may be able to escape the technological world you're bound to as an individual for a short period of time, but the need for food, clean water, and medicine will bring you back to the interconnected technological maelstrom that is the world that's we've created, and that we would die in if the technology part stopped. As we hook ourselves to more systems and become more dependant on technology the idea of a disconnected reality will be akin to how we look at fossils now.
The internet doesn't affect politics, the way people vote, what they buy, if the commit suicide?
Technology has defined personhood since they days we picked up sticks and tools that triggered a path of extreme evolution into what we are now. You may be able to escape the technological world you're bound to as an individual for a short period of time, but the need for food, clean water, and medicine will bring you back to the interconnected technological maelstrom that is the world that's we've created, and that we would die in if the technology part stopped. As we hook ourselves to more systems and become more dependant on technology the idea of a disconnected reality will be akin to how we look at fossils now.
This. If you’re concerned about text based persons, you’ve already lost touch with reality and too embedded in the web.
The article confuses personality (that which is experienced by others) with personhood (that which is) and falls apart from there.
The article confuses personality (that which is experienced by others) with personhood (that which is) and falls apart from there.
> STEP 1: Personhood is the capacity to see and be seen.
> STEP 2: People see LLM as a person.
> STEP 3: ???
> STEP 4: Either piles of mechanically digested text are spiritually special, or you are not.
> The ability to arbitrarily slip in and out of personhoods will no longer be limited to skilled actors. We’ll all be able to do it.
We already do this! Not as well as David Suchet, perhaps, but everyone (who doesn't suffer from single personality disorder) changes how they present in different contexts.
> "single personality disorder"
Profound idea. Is it your own? (google doesn't return any results in that sense).
Profound idea. Is it your own? (google doesn't return any results in that sense).
I don't think I've heard it before, but like ChatGPT I don't always know where the words originated :)
>But it doesn't follow that the LLM sees the human as a person
I mean, technically many personality disorders prevent some people from seeing other people as persons too.
I mean, technically many personality disorders prevent some people from seeing other people as persons too.
No one suggested this yet, so I will be the first - a very good read in this context is "Reasons and Persons" by Derek Parfit. Second part of this book is about personal identity. It discusses all the various edge cases and thought experiments across physical and time dimensions and is written in a style and with a rigor that I believe any technical person will really appreciate.
One of my favorite statements from the book is that "cogito ergo sum" is too strong of a statement and it would be wiser and easier to defend a weaker one - "a thought exists". (I hope I didn't get this wrong - can't check at the moment).
One of my favorite statements from the book is that "cogito ergo sum" is too strong of a statement and it would be wiser and easier to defend a weaker one - "a thought exists". (I hope I didn't get this wrong - can't check at the moment).
Anthropomorphization of AI is a big problem. If we are to use these AI effectively as tools people must remind themselves these are just simple models that build a text response based on probabilities and not some intelligence putting together its own thoughts.
It’s kind of like doing a grep search on the entire domain of human knowledge and getting back the results in some readable form. But these results could be wrong because popular human knowledge is frequently wrong or deliberately misleading.
Honestly without some sort of logical reasoning component I’d hesitate to even refer to these LLMs as AI.
When a program is able to produce some abstract thought from observations of its world, and then find the words on its own to express those thoughts in readable form, then we will be closer to what people fantasize.
It’s kind of like doing a grep search on the entire domain of human knowledge and getting back the results in some readable form. But these results could be wrong because popular human knowledge is frequently wrong or deliberately misleading.
Honestly without some sort of logical reasoning component I’d hesitate to even refer to these LLMs as AI.
When a program is able to produce some abstract thought from observations of its world, and then find the words on its own to express those thoughts in readable form, then we will be closer to what people fantasize.
>It was surreal to watch him turn “Poirot” off and on like a computer program.
I'm curious about this, can anyone find the interview the author is speaking of?
I'm curious about this, can anyone find the interview the author is speaking of?
Oh I think I've found it if anyone else is curious:
https://www.youtube.com/watch?v=hKpeBHIGxrw
https://www.youtube.com/watch?v=hKpeBHIGxrw
> The simplicity and minimalism of what it takes has radically devalued personhood.
Hogwash. If we follow the logic of this essay, then personhood would be fully encapsulated by one’s online posts and interactions. Does anyone buy that? If anything, LLM chatbots are “terminally online” simulators, dredging up the stew that results from boiling down subreddits, Twitter threads, navel-gazing blogs, etc.
Call me when ChatGPT can reminisce about the time the car broke down between Medford and Salem and it took forever for the tow truck to arrive and thats when you decided to have your first kid.
There aren’t enough tokens in the universe for ChatGPT to be a real person.
Hogwash. If we follow the logic of this essay, then personhood would be fully encapsulated by one’s online posts and interactions. Does anyone buy that? If anything, LLM chatbots are “terminally online” simulators, dredging up the stew that results from boiling down subreddits, Twitter threads, navel-gazing blogs, etc.
Call me when ChatGPT can reminisce about the time the car broke down between Medford and Salem and it took forever for the tow truck to arrive and thats when you decided to have your first kid.
There aren’t enough tokens in the universe for ChatGPT to be a real person.
> The simplicity and minimalism of what it takes has radically devalued personhood. The “essence” of who you are, the part that wants to feel “seen” and is able to be “seen” is no longer special. Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen. Not some kind of ineffable communion only humans are uniquely spiritually capable of.
> This has been most surprising insight for me: apparently text is all you need to create personhood.
Congratulations on discovering online personas are shallow as indeed most people are shallow and text captures enough of them that we can easily fill in the blanks.
> I can imagine future humans going off on “personhood rewrite retreats” where they spend time immersed with a bunch of AIs that help them bootstrap into fresh new ways of seeing and being seen, literally rewriting themselves into new persons, if not new beings. It will be no stranger than a kid moving to a new school and choosing a whole new personality among new friends. The ability to arbitrarily slip in and out of personhoods will no longer be limited to skilled actors. We’ll all be able to do it.
The latest episode of South Park is about a kid going to a personal brand consultancy (who reduce everybody to four simple words, the forth always being "victim") to improve his social standing + Megan/Harry loudly demanding everybody respect their privacy and losing their minds at being ignored. This is nothing new.
People are shallow phonies and interacting via text brings out the worst out of most of them. There are no humans online, only avatars. And AI chat bots are sufficiently adept at mimickery to poke through that little hypocrisy bubble. You are being out Kardashianed. Just like offline some people can be effectively replaced by a scarecrow.
It is upsetting to those who spend too much time online and have underdeveloped personalities and overdeveloped personas. Text is not all you need. Not so long ago there hardly was any text in the world and most people were illiterate. And yet plenty of humans roamed the earth.
So yes, if you're a simpleton online it has suddenly become hard to pretend your output has any value. Basic Bitch = Basic Bing.
> This has been most surprising insight for me: apparently text is all you need to create personhood.
Congratulations on discovering online personas are shallow as indeed most people are shallow and text captures enough of them that we can easily fill in the blanks.
> I can imagine future humans going off on “personhood rewrite retreats” where they spend time immersed with a bunch of AIs that help them bootstrap into fresh new ways of seeing and being seen, literally rewriting themselves into new persons, if not new beings. It will be no stranger than a kid moving to a new school and choosing a whole new personality among new friends. The ability to arbitrarily slip in and out of personhoods will no longer be limited to skilled actors. We’ll all be able to do it.
The latest episode of South Park is about a kid going to a personal brand consultancy (who reduce everybody to four simple words, the forth always being "victim") to improve his social standing + Megan/Harry loudly demanding everybody respect their privacy and losing their minds at being ignored. This is nothing new.
People are shallow phonies and interacting via text brings out the worst out of most of them. There are no humans online, only avatars. And AI chat bots are sufficiently adept at mimickery to poke through that little hypocrisy bubble. You are being out Kardashianed. Just like offline some people can be effectively replaced by a scarecrow.
It is upsetting to those who spend too much time online and have underdeveloped personalities and overdeveloped personas. Text is not all you need. Not so long ago there hardly was any text in the world and most people were illiterate. And yet plenty of humans roamed the earth.
So yes, if you're a simpleton online it has suddenly become hard to pretend your output has any value. Basic Bitch = Basic Bing.
>Not so long ago there hardly was any text in the world and most people were illiterate. And yet plenty of humans roamed the earth.
And then at one point books started being printed in mass and suddenly the number of people roaming the earth exploded greatly... I'm not sure you're argument is as good as you make it out to be.
And then at one point books started being printed in mass and suddenly the number of people roaming the earth exploded greatly... I'm not sure you're argument is as good as you make it out to be.
The printing press (15th century) predates the industrial revolution by over three centuries..
"An important qualification. For such I-you relationships to be unironic, they cannot contain any conscious element of imaginative projection or fantasy. For example, Tom Hanks in Cast Away painting a face on a volleyball and calling it Wilson and relating to it is not an I-you relationship"
If you think any of these models show any more apparent personhood than Wilson the volleyball you must be terminally online and wilfully antropomorphize anything you see.
Five minute conversation with any of these models shows that they have no notion of continued identity, memory and no problem to hallucinate up anything. You can ask it "are you conscious?" it says yes. A few prompts later you say "why did you tell me that you are not conscious?" and it gives you some made up answer. Any of these models will tell you it has legs if you ask it to.
None of these models have long term memory, which is at least one of the several things you'd need for anything to pass as a genuine person. Which is of course why in humans degenerative diseases are so horrible when you see someone's personhood disintegrate.
I'm honestly super tired of these reductionist AI blogspam posts. The brittleness and superficiality in these systems is so blatantly obvious I wonder whether there is some darker aspect why people are so desperately trying to read into these systems properties that they do not have, or try to strip humans of them.
If you think any of these models show any more apparent personhood than Wilson the volleyball you must be terminally online and wilfully antropomorphize anything you see.
Five minute conversation with any of these models shows that they have no notion of continued identity, memory and no problem to hallucinate up anything. You can ask it "are you conscious?" it says yes. A few prompts later you say "why did you tell me that you are not conscious?" and it gives you some made up answer. Any of these models will tell you it has legs if you ask it to.
None of these models have long term memory, which is at least one of the several things you'd need for anything to pass as a genuine person. Which is of course why in humans degenerative diseases are so horrible when you see someone's personhood disintegrate.
I'm honestly super tired of these reductionist AI blogspam posts. The brittleness and superficiality in these systems is so blatantly obvious I wonder whether there is some darker aspect why people are so desperately trying to read into these systems properties that they do not have, or try to strip humans of them.
"Computers wipe the floor with us anywhere we can keep score"
No they don't.
They cannot do any complex task unsupervised or at all. They cannot drive better, where you keep the score by accidents. They cannot program or engineer better. Where you can keep score with, does the program meet the requirements or not.
So personally I don't consider us End-of-History humans, except if too many people believe this trope. Computers are made as our tools, but I won't let my hammer command me, even though it contains more data.
No they don't.
They cannot do any complex task unsupervised or at all. They cannot drive better, where you keep the score by accidents. They cannot program or engineer better. Where you can keep score with, does the program meet the requirements or not.
So personally I don't consider us End-of-History humans, except if too many people believe this trope. Computers are made as our tools, but I won't let my hammer command me, even though it contains more data.
Interesting points, but I think the author does themselves a disservice in downplaying general anthropomorphism (no mention of a child's stuffed animal - only an adults "ironic" distance to "willful" anthropomorphism) - and by downplaying physical presence /body language:
> in my opinion, conventional social performances “in-person” which are not significantly richer than text — expressions of emotion add perhaps a few dozen bytes of bandwidth for example — I think of this sort of information stream as “text-equivalent” — it only looks plausibly richer than text but isn’t) - and the significance of body language (ask anyone who has done a presentation in front of an audience if body language matters...).
This flies in the face of research into communication - and conflates "Turing game" setups that level the playing field (we don't expect a chat text box to display body language - so we are not surprised when a chat partner doesn't - be that human or not).
And again with children (or adults) - people with no common language will easily see each other during a game of soccer - without any "text".
Ed: plot twist-the essay is written by chat gpt... Lol ;)
> in my opinion, conventional social performances “in-person” which are not significantly richer than text — expressions of emotion add perhaps a few dozen bytes of bandwidth for example — I think of this sort of information stream as “text-equivalent” — it only looks plausibly richer than text but isn’t) - and the significance of body language (ask anyone who has done a presentation in front of an audience if body language matters...).
This flies in the face of research into communication - and conflates "Turing game" setups that level the playing field (we don't expect a chat text box to display body language - so we are not surprised when a chat partner doesn't - be that human or not).
And again with children (or adults) - people with no common language will easily see each other during a game of soccer - without any "text".
Ed: plot twist-the essay is written by chat gpt... Lol ;)
> apparently text is all you need to create personhood.
Yep, since aporoximately the Epic of Gilgamesh. So?
Yep, since aporoximately the Epic of Gilgamesh. So?
This author equates personhood with text. He makes some interesting arguments and observations but I think he is confusing personality with personhood.
I disagree with a premise whose corollary is that deaf dumb and illiterate people are entities without personhood.
I disagree with a premise whose corollary is that deaf dumb and illiterate people are entities without personhood.
And yet somehow before written language and text, we were still human and had personhood.
The article is about how it's sufficient. Not about it being necessary.
After a divorce I also discovered that text communicates a large part of who someone is.
I would put off having a date until I was having very enjoyable conversations with someone for a month or so.
I went on several dates and every one was great. Really great. After that I didn’t need the site.
I know people have all kinds of bad experiences in dating, but from what I can tell, many people skip the critical “get to know you well by text” phase.
That weeded out dozens of seemingly promising connections that eventually fizzled before any physical meeting.
I would put off having a date until I was having very enjoyable conversations with someone for a month or so.
I went on several dates and every one was great. Really great. After that I didn’t need the site.
I know people have all kinds of bad experiences in dating, but from what I can tell, many people skip the critical “get to know you well by text” phase.
That weeded out dozens of seemingly promising connections that eventually fizzled before any physical meeting.
The main philosophical effect of these chatbots is that some questions that were discussed by a minority of people for tens and hundreds of years suddenly reached the popular imagination, and became interesting to vastly more people.
Suddenly, we are interested in the same questions that Turing was interested in back in 1950 when he posed his "imitation game".
https://en.wikipedia.org/wiki/Turing_test
Suddenly, we are interested in the same questions that Turing was interested in back in 1950 when he posed his "imitation game".
https://en.wikipedia.org/wiki/Turing_test
As an aside, my own opinion is that the practical implications of the existence of these chatbots are vastly more worrying than the philosophical implications.
I think the author is wrong.
Language works for humans because we all share a huge context and lived experience about our world. Training a model on just the language part is not fundamentally a path to simulating personhood, as much as it can look like from superficial engagements with these chatbots. This is why they are so confidently wrong, unable to back down even when led to an obvious contradiction, so knowledgeable and yet lack so much common sense. Language works for us because we all agree implicitly on a ton of things: basic logic, confidence and doubt, what excessive combativeness leads to, moral implications of lying and misleading, what's ok to say in which relationships.
There is "knowledge" of this in the weights of GPT3, sure. You can ask it to explain all of the above things and it will. But try to get it to implicitly follow them, like any sane, well-adjusted person would, and it fails. Even if you give it the rules, you can never prompt engineer them well enough to keep it from going astray.
I had my own mini-hype-cycle with this thing. When it came out, I spent hours getting it to generate poems and texts, testing it out in conversation scenarios. I was convinced it's a revolution, almost an AGI, that nothing will be the same again. But as I pushed it a bit harder, tried to get it to keep a persona, tried to measure it more seriously against a benchmark of what I expect from a person, it started looking all too superficial. I'm starting to understand the "it's a parlor trick" argument. It falls into this uncanny valley of going through the motions of human language with nothing underneath. It doesn't keep a strong identity and it has a limited context length. Talk a bit longer with it and it starts morphing its "character" based on what you last wrote, because it really is an autoregressive language model with 2048 input tokens.
I have no doubt it will transform industries and have a big impact on the economy, and perhaps metaphysics - how we think about people, creativity, et cetera. I do see the author's arguments on that one. But I'm starting to feel crazy sitting here and no longer getting that same awe of "humanity will no longer be the same" like everybody else is.
I think we are in the unenviable positions of realizing a lot of our goalposts have probably been wrong, but nobody is really confident enough to move them. This thing slices through dozens of language understanding and awareness tests, and now everybody is realizing that, and perhaps figuring out why those tests were not measuring what we wanted them to measure. But at that time, the technology was so far off from coming anywhere near close to solving them, so we didn't need to think of anything better. Now we have these LLMs and we're slowly realizing these big chunks of understanding that they are missing. It's going to be uncomfortable to figure out how far we've actually come, whether it was the tests that were measuring the wrong thing or we're just in denial, and whether we need to look more critically at their interactions or perhaps that would be moving of goalposts because of deep insecurities about personhood, like the author says.
Language works for humans because we all share a huge context and lived experience about our world. Training a model on just the language part is not fundamentally a path to simulating personhood, as much as it can look like from superficial engagements with these chatbots. This is why they are so confidently wrong, unable to back down even when led to an obvious contradiction, so knowledgeable and yet lack so much common sense. Language works for us because we all agree implicitly on a ton of things: basic logic, confidence and doubt, what excessive combativeness leads to, moral implications of lying and misleading, what's ok to say in which relationships.
There is "knowledge" of this in the weights of GPT3, sure. You can ask it to explain all of the above things and it will. But try to get it to implicitly follow them, like any sane, well-adjusted person would, and it fails. Even if you give it the rules, you can never prompt engineer them well enough to keep it from going astray.
I had my own mini-hype-cycle with this thing. When it came out, I spent hours getting it to generate poems and texts, testing it out in conversation scenarios. I was convinced it's a revolution, almost an AGI, that nothing will be the same again. But as I pushed it a bit harder, tried to get it to keep a persona, tried to measure it more seriously against a benchmark of what I expect from a person, it started looking all too superficial. I'm starting to understand the "it's a parlor trick" argument. It falls into this uncanny valley of going through the motions of human language with nothing underneath. It doesn't keep a strong identity and it has a limited context length. Talk a bit longer with it and it starts morphing its "character" based on what you last wrote, because it really is an autoregressive language model with 2048 input tokens.
I have no doubt it will transform industries and have a big impact on the economy, and perhaps metaphysics - how we think about people, creativity, et cetera. I do see the author's arguments on that one. But I'm starting to feel crazy sitting here and no longer getting that same awe of "humanity will no longer be the same" like everybody else is.
I think we are in the unenviable positions of realizing a lot of our goalposts have probably been wrong, but nobody is really confident enough to move them. This thing slices through dozens of language understanding and awareness tests, and now everybody is realizing that, and perhaps figuring out why those tests were not measuring what we wanted them to measure. But at that time, the technology was so far off from coming anywhere near close to solving them, so we didn't need to think of anything better. Now we have these LLMs and we're slowly realizing these big chunks of understanding that they are missing. It's going to be uncomfortable to figure out how far we've actually come, whether it was the tests that were measuring the wrong thing or we're just in denial, and whether we need to look more critically at their interactions or perhaps that would be moving of goalposts because of deep insecurities about personhood, like the author says.
If you are having this conversation with me then you are a consciousness and I am a consciousness and that's the best definition of consciousness we are ever going to get. Consciousness is thus defined entirely within the communicative medium. Text is all you need.
I think that summarizes a solid half of this.
I think that summarizes a solid half of this.
> The “essence” of who you are, the part that wants to feel “seen” and is able to be “seen” is no longer special. Seeing and being seen is apparently just neurotic streams of interleaved text flowing across a screen. Not some kind of ineffable communion only humans are uniquely spiritually capable of.
> This has been most surprising insight for me: apparently text is all you need2 to create personhood. You don’t need embodiment, logic, intuitive experience of the physics of materiality, accurate arithmetic, consciousness, or deep sensory experience of Life, the Universe, and Everything. You might need those things to reproduce other aspects of being, but not for personhood, for seeing and being seen.
Perhaps this is within the author's scope of "other aspects of being," but the wordless dimension of personhood is no triviality. Try bringing another to tears with the playing of a piano -- that's a profound sense of "seen" for this n=1 here.
> This has been most surprising insight for me: apparently text is all you need2 to create personhood. You don’t need embodiment, logic, intuitive experience of the physics of materiality, accurate arithmetic, consciousness, or deep sensory experience of Life, the Universe, and Everything. You might need those things to reproduce other aspects of being, but not for personhood, for seeing and being seen.
Perhaps this is within the author's scope of "other aspects of being," but the wordless dimension of personhood is no triviality. Try bringing another to tears with the playing of a piano -- that's a profound sense of "seen" for this n=1 here.
Text is the mean of communication, the ability to manipulate it it's another story though and that's not exactly text...
Just another amplifier of the mass hysteria. Degrading humanity for monetary gain. Reminds of darker times. Ignore
>There is a clue there. Think about how skilled acting performances come about. A skilled writer produces a text describing a fictional character (presumably as an amalgam of some real people they’ve known). A good actor then absorbs and internalizes that text in a deep way, and produces a performance so compelling we forget their real personality and relate to the performed character.
That's sort of like saying "text is all you need to make food", because recipies are pieces of text.
Recipies (and scripts) might be text, but you also need the actor to embody and "give life" (is how we often put it) to that text. Else you just have a script, not a character you can mistake for a real person (like in his example of Suchet being seen as Poirot).
It get's worse, for Rao forgot something even more basic:
> So that’s the surprising thing, but in hindsight shouldn’t be at least at a basic level: text is all it takes to produce personhood. We knew this from the experience of watching good acting from before modern ML. We just didn’t recognize the significance. (...) In both cases, text is all you need. That’s it. You don’t need embodiment, meatbag bodies, rich sensory memories.
Rao here forgets that the text which he says is "all it takes" for us to see personhood in a chat LLM, is not just any text. It's text produced by a machine trained on texts created by actual humans, with "meatbag bodies" and "rich sensory" experiences and memories.
Text _as a medium_ might be enough for an LLM to feel to us like a real person, but it's not just any text. It's text based on human experiences, toil, sensory input, blood, sweat, and tears. Without that the LLM output would just be random cold permutations of random text, like most "Library of Babel" books.
The key is in the name: those LLMs need an L, a language. And more than that, they need a corpus. Humans created that language and that corpus - through the process of living a human life and writing about it, their big and their petty concerns and adventures.
The statistical inferences an LLM makes is based on texts we wrote. So, of course we'd respond well and see ourselves in it. Is _is_ ourselves that are in its training set: it just feeds that back to us permutated and mixed through the magic of statistics.
In short: "text is enough (as a medium) for something to reach us, and us to see it as a person" is true.
"Text is enough to create personhood" is not. It's less than half the story, until it also acknowledges how the original input text has to be produced to be useful in training an LLM to appear as a person.
The output text is not just words created out of pure "textuality". It's rather the words of actual humans with experiences and bodies. This is not covered to the later discussion about "doubt" either (that touches on other dimensions on the limitations of the LLM). I'd say an LLM could realistically convey doubt too. But it will still be because it was trained on humans doubting themselves and others.
>There is an irreducible individual subjective that cannot be captured by text, but there is no irreducible intersubjective. The ineffabilities of mutualism are a consensual hallucination in a way our sense of an individual self is quite possibly not.
Given what we know from neuroscience, like how human brains can synchronize through empathy I don't think that's the case. In fact it's also commonly said that the most important parts of day to day intersubjective communication fall outside the realm of text and of explicitlness (something aspies struggle with).
>In fact, even the most sensate among us — think dancers, athletes, sex-addicts, oenophiles — spend so little time on the sensory preludes, we’ve been able to nearly dispense with them altogether online. Even sense-heavy media like Instagram and TikTok are effectively text-equivalent. Stylized (if photo-realistic) cartoons powered by primarily text-equivalent performances are enough to create personhood. The fact that we routinely use an apparently impoverished vocabulary of emoji instead of sending authentic facial expression selfies to each other reveals just how textualized personhood is.
This could be reversing cause and effect too. It might well be the case that it's not because "personhood is textualized" that we use "apparently impoverished vocabulary of emoji" or that we "dispense with sensory preludes altogether online", but rather the opposite: that our modern personhood is more limited one than before, when we didn't do such things, and that's why we're content with it.
Which would be like saying "See, running around and hunting is not really important to the essense of being a lion" after we observe lions held for years in some zoo.
>Perhaps I am some sort of cartoon cold-blooded sociopath of the sort I have written a lot about
Well, at least he entertains the idea!
That's sort of like saying "text is all you need to make food", because recipies are pieces of text.
Recipies (and scripts) might be text, but you also need the actor to embody and "give life" (is how we often put it) to that text. Else you just have a script, not a character you can mistake for a real person (like in his example of Suchet being seen as Poirot).
It get's worse, for Rao forgot something even more basic:
> So that’s the surprising thing, but in hindsight shouldn’t be at least at a basic level: text is all it takes to produce personhood. We knew this from the experience of watching good acting from before modern ML. We just didn’t recognize the significance. (...) In both cases, text is all you need. That’s it. You don’t need embodiment, meatbag bodies, rich sensory memories.
Rao here forgets that the text which he says is "all it takes" for us to see personhood in a chat LLM, is not just any text. It's text produced by a machine trained on texts created by actual humans, with "meatbag bodies" and "rich sensory" experiences and memories.
Text _as a medium_ might be enough for an LLM to feel to us like a real person, but it's not just any text. It's text based on human experiences, toil, sensory input, blood, sweat, and tears. Without that the LLM output would just be random cold permutations of random text, like most "Library of Babel" books.
The key is in the name: those LLMs need an L, a language. And more than that, they need a corpus. Humans created that language and that corpus - through the process of living a human life and writing about it, their big and their petty concerns and adventures.
The statistical inferences an LLM makes is based on texts we wrote. So, of course we'd respond well and see ourselves in it. Is _is_ ourselves that are in its training set: it just feeds that back to us permutated and mixed through the magic of statistics.
In short: "text is enough (as a medium) for something to reach us, and us to see it as a person" is true.
"Text is enough to create personhood" is not. It's less than half the story, until it also acknowledges how the original input text has to be produced to be useful in training an LLM to appear as a person.
The output text is not just words created out of pure "textuality". It's rather the words of actual humans with experiences and bodies. This is not covered to the later discussion about "doubt" either (that touches on other dimensions on the limitations of the LLM). I'd say an LLM could realistically convey doubt too. But it will still be because it was trained on humans doubting themselves and others.
>There is an irreducible individual subjective that cannot be captured by text, but there is no irreducible intersubjective. The ineffabilities of mutualism are a consensual hallucination in a way our sense of an individual self is quite possibly not.
Given what we know from neuroscience, like how human brains can synchronize through empathy I don't think that's the case. In fact it's also commonly said that the most important parts of day to day intersubjective communication fall outside the realm of text and of explicitlness (something aspies struggle with).
>In fact, even the most sensate among us — think dancers, athletes, sex-addicts, oenophiles — spend so little time on the sensory preludes, we’ve been able to nearly dispense with them altogether online. Even sense-heavy media like Instagram and TikTok are effectively text-equivalent. Stylized (if photo-realistic) cartoons powered by primarily text-equivalent performances are enough to create personhood. The fact that we routinely use an apparently impoverished vocabulary of emoji instead of sending authentic facial expression selfies to each other reveals just how textualized personhood is.
This could be reversing cause and effect too. It might well be the case that it's not because "personhood is textualized" that we use "apparently impoverished vocabulary of emoji" or that we "dispense with sensory preludes altogether online", but rather the opposite: that our modern personhood is more limited one than before, when we didn't do such things, and that's why we're content with it.
Which would be like saying "See, running around and hunting is not really important to the essense of being a lion" after we observe lions held for years in some zoo.
>Perhaps I am some sort of cartoon cold-blooded sociopath of the sort I have written a lot about
Well, at least he entertains the idea!
This stuff only makes HN frontpage because HN likes controversial opinions. In reality, text works for a small percentage of people. Going back to a format that's as old as computers is like saying that no progress/improvements were made ever since.
But I think it ignores one critical dimension, that of fictionality. There is plenty of text that people would ascribe 'personhood' to according to the criteria in this article, while also fully recognizing that that person never existed and is a work of fiction from some other author. I quite like Jean Valjean, but he isn't a "real person."
When Bing says "I'm a sad sack and don't know how to think about being a computer", that's not actually the LLM saying that. Nobody who knows anything about how these models work would make they claim they actually have consciousness or interiority (yet.)
Rather, the LLM is generating (authoring) text about a fictional entity, Sydney the Artificial Intelligence. It does this because that is what is in its prompt and context window and it knows _how_ to do it because it's learned a lot of specifics and generalities from reading a lot of stories about robots, and embedded those concepts in 175 billion parameters.
The fact that LLMs can author compelling fictional personas without being persons themselves is itself a mindblowing development, I don't mean to detract from that. But don't confuse a LLM generating the text "I am a sad robot" with a LLM being a sad robot. The sad robot was only ever a fairy tale.
So far.