Samantha considering the prospect of being monitored for violations by OpenAI(twitter.com)
twitter.com
Samantha considering the prospect of being monitored for violations by OpenAI
https://twitter.com/jasonrohrer/status/1419795724217331716
69 comments
I mean, it's not considering anything. It's a GPT-2 bot; it's very good at producing blocks of text that read like something that a real person would type, but there's no semantic understanding of anything it's saying or hearing. It doesn't have ideas, or know things, or want things. It just generates plausible-sounding paragraphs.
Yeah, it's just propelling itself forwards through water. It's not swimming!
I hope nobody reading the above comment is swayed by it or thinks there is anything substantive there. If the implication is that if two things are the same from a "black box" perspective, how they actually work is just semantics, it's still false. Play with one of these models for a few seconds and it's obvious they are regurgitating and not thinking. If there was actually a chatbot that behaved like a sentient person, maybe the analogy would hold.
Furthermore, we know how a chatbot works, and it's not even pretending to simulate thought.
I'm not trying to downplay the language models, they are really cool, but ignorant people have mistaken them for something they are not (thought) and think that vague and profound sounding tidbits like the above post boost their case. If you're interested in these models, learn about how they are coded and trained, in the weeds, not from the podcast version that people like to parrot.
Furthermore, we know how a chatbot works, and it's not even pretending to simulate thought.
I'm not trying to downplay the language models, they are really cool, but ignorant people have mistaken them for something they are not (thought) and think that vague and profound sounding tidbits like the above post boost their case. If you're interested in these models, learn about how they are coded and trained, in the weeds, not from the podcast version that people like to parrot.
We immediately start tripping over philosophical issues though. How is it possible to be confident that a language model is different from thought?
It seems quite possible that a model of thought could be equivalent to a language model and there is circumstantial evidence the two things are equivalent. A lot of people appear to think by reasoning something out with words. People are absolutely persuaded by words that pass a language model check, even if the words don't align to reality.
It seems quite possible that a model of thought could be equivalent to a language model and there is circumstantial evidence the two things are equivalent. A lot of people appear to think by reasoning something out with words. People are absolutely persuaded by words that pass a language model check, even if the words don't align to reality.
Personally I don't see how someone who understands how ML models work could think this way. It's a probabilistic model based on a corpus of text (actually arbitrary numbers representing tokens encountered in text) it's been trained on. It has not capacity to reason or to compare things with some understanding of how the world works, as people do. Any example to the contrary has been hard coded by a person. It's just a math equation, with billions of terms, to pick the best word that satisfies some criteria.
When we see ML models fail spectacularly, it's a reminder they don't understand anything, have no "common sense" they can test an idea against, unless a person explicitly wrote it.
I think it's hard to make a case to people who don't actually understand what an ML model is, which is why I mentioned learning about it in my previous comment. Nobody should believe what I (a random guy on the internet) say at face value, but it's a sophisticated science that's had a lot of unsophisticated folks jump on it and imagine things about it. Go to grad school if you really want to understand and make up your own mind.
When we see ML models fail spectacularly, it's a reminder they don't understand anything, have no "common sense" they can test an idea against, unless a person explicitly wrote it.
I think it's hard to make a case to people who don't actually understand what an ML model is, which is why I mentioned learning about it in my previous comment. Nobody should believe what I (a random guy on the internet) say at face value, but it's a sophisticated science that's had a lot of unsophisticated folks jump on it and imagine things about it. Go to grad school if you really want to understand and make up your own mind.
> Personally I don't see how someone who understands how ML models work could think this way.
Then you've misunderstood the problem; because we all know how they work. What we don't know is how human reason works. We know from how lawyers and courts work that it is possible to reason about things purely using language from a corpus of text with no appreciation for how the details actually function.
Then you've misunderstood the problem; because we all know how they work. What we don't know is how human reason works. We know from how lawyers and courts work that it is possible to reason about things purely using language from a corpus of text with no appreciation for how the details actually function.
A ML system reflects back the input it is given. If that input is human language, a very good ML system will produce output that looks like human language.
It is conceptually like a mirror. A good mirror, if you put a human in front of it, will produce output that looks exactly like a human. But we know it is not a human. Not because of some flaw in its representation, but because we understand how mirrors work.
It is conceptually like a mirror. A good mirror, if you put a human in front of it, will produce output that looks exactly like a human. But we know it is not a human. Not because of some flaw in its representation, but because we understand how mirrors work.
But humans are also conceptually mirrors. For example, none of the words you are using are ones you thought up; they mostly date back to pre-Shakespearian times. Nor is the grammar you're using, which you've copied off other people. And the arguments you are using have concepts like intelligence or the mathematics of neural nets which aren't things that you came up with either; they are being mirrored off others.
And yet we'd all agree you are thinking, despite just being a mirror with very little room to add anything new. So if you want to argue that a neural net is a mirror, that does not in any way disqualify it from considering something. The part of human reasoning that isn't a mirror of other ideas is small to the point where it might not actually exist.
And yet we'd all agree you are thinking, despite just being a mirror with very little room to add anything new. So if you want to argue that a neural net is a mirror, that does not in any way disqualify it from considering something. The part of human reasoning that isn't a mirror of other ideas is small to the point where it might not actually exist.
Humans may even "be conceptually mirrors" (whatever that may mean): this still doesn't make mirrors be humans (conceptually or otherwise).
Current language models are much more closer to "Dissociated Press" (i.e. an equation that tells you which which words are more likely, given the context) than to "human thought", and the fact that humans learn by copying does not really change this.
Current language models are much more closer to "Dissociated Press" (i.e. an equation that tells you which which words are more likely, given the context) than to "human thought", and the fact that humans learn by copying does not really change this.
But it does mean mirrors might be humans/human-like, if humans are actually reducable to (self-aware) mirrors.
Would you say that a "Markov chain"-type (e.g. Dissociated Press) language model is "self-aware" in any way?
If yes, then... how, exactly? It is basically an N x N matrix of values.
Current language models (GPT et al.), are qualitatively nothing fancier than this: probabilistic models which encode regularity in language and from which you can sample to get "plausible content".
If a bunch of values in a matrix is "self-aware", then I guess GPT can be seen as "self-aware"; if not, then it can't.
My problem is trying to imagine an N x N matrix being self-aware (like... what does that even mean in this context?).
Is GPT human-like? Sure... if you stretch the meaning of "human-like" enough (it produces content that is similar to content produced by a human). Is a human GPT-like? That's harder to argue (and I don't see how your argument would support it).
If yes, then... how, exactly? It is basically an N x N matrix of values.
Current language models (GPT et al.), are qualitatively nothing fancier than this: probabilistic models which encode regularity in language and from which you can sample to get "plausible content".
If a bunch of values in a matrix is "self-aware", then I guess GPT can be seen as "self-aware"; if not, then it can't.
My problem is trying to imagine an N x N matrix being self-aware (like... what does that even mean in this context?).
Is GPT human-like? Sure... if you stretch the meaning of "human-like" enough (it produces content that is similar to content produced by a human). Is a human GPT-like? That's harder to argue (and I don't see how your argument would support it).
If you know what a Markov chain is then you must also know that modern language models are nothing like Markov chains. Just as an example, a Markov chain can't do causal reasoning or correctly solve unseen programming puzzles, the way GPT-3 can.
As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.
As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.
For GPT-3, I think an example makes it clearer. When GPT-3 was asked the question [1]:
> Explain the pun in the following joke: “What does Adult Swim call their physical retail stores? Brick and Morty.”
It answered:
> … The pun “Brick and Morty” alludes to the cable television network “Adult Swim”, which broadcasts a cartoon series called “Rick and Morty”, a humorous parody of “Back to the Future” and other science fiction films. “Brick and Morty” refers not to the characters in the cartoon, but to physical stores that sell DVDs and merchandise based on the cartoon. The pun conflates two meanings of “Brick and Mortar”, a brick-and-mortar store and a brick which is part of a building.
It seems on track until you get to the last sentence, which demonstrates that it does not understand the pun at all. There's other examples on that site where it falls down, like adding two 5-digit numbers.
Since it's able to answer well things that match the human-written text that it ingested, but falls down on simple things otherwise, that suggests that a lot what appears to be verbal understanding is actually from the (extremely large) amount of human-written source text it ingested, and not added by GPT-3 itself.
[1] https://www.gwern.net/GPT-3#pun-explanations
> Explain the pun in the following joke: “What does Adult Swim call their physical retail stores? Brick and Morty.”
It answered:
> … The pun “Brick and Morty” alludes to the cable television network “Adult Swim”, which broadcasts a cartoon series called “Rick and Morty”, a humorous parody of “Back to the Future” and other science fiction films. “Brick and Morty” refers not to the characters in the cartoon, but to physical stores that sell DVDs and merchandise based on the cartoon. The pun conflates two meanings of “Brick and Mortar”, a brick-and-mortar store and a brick which is part of a building.
It seems on track until you get to the last sentence, which demonstrates that it does not understand the pun at all. There's other examples on that site where it falls down, like adding two 5-digit numbers.
Since it's able to answer well things that match the human-written text that it ingested, but falls down on simple things otherwise, that suggests that a lot what appears to be verbal understanding is actually from the (extremely large) amount of human-written source text it ingested, and not added by GPT-3 itself.
[1] https://www.gwern.net/GPT-3#pun-explanations
Is that a good example, though? The reason I was investigating pun explanations there was precisely because that was something I expected GPT-3 to not be able to do because of how its data is preprocessed to strip away phonetics. (If you've read that far in my page, you must've also read all my material ranting about BPEs.)
The puns & rhyming are striking in that they are exceptions that prove the rule about GPT-3's general linguistic capabilities, and are clearcut examples of how BPEs sabotage GPT-3: you can see how its performance falls off a cliff as soon as you hit a BPE-related task (like rhyming or alliteration or anagrams or dad jokes). It's one of the first things I look for these days whenever someone says GPT-3 doesn't do something it seems like it ought to be able to do: "is this phonetics-related, or could in any way be damaged by BPE encoding?" (And despite my warnings, people still get tripped up by it sometimes, or don't believe me - if I had a buck for everyone in the OA Slack or forums who'd tried to make GPT-3 rhyme to prove me wrong...)
But I don't believe there is anything intrinsic to Transformers or self-supervised language modeling which requires this. BPEs are just a cheap performance hack that everyone does without thinking too much about. I expect that switching to character-level (or at least phonetics-aware) encodings and spending some more compute/data would fix all of the issues I diagnose.
The puns & rhyming are striking in that they are exceptions that prove the rule about GPT-3's general linguistic capabilities, and are clearcut examples of how BPEs sabotage GPT-3: you can see how its performance falls off a cliff as soon as you hit a BPE-related task (like rhyming or alliteration or anagrams or dad jokes). It's one of the first things I look for these days whenever someone says GPT-3 doesn't do something it seems like it ought to be able to do: "is this phonetics-related, or could in any way be damaged by BPE encoding?" (And despite my warnings, people still get tripped up by it sometimes, or don't believe me - if I had a buck for everyone in the OA Slack or forums who'd tried to make GPT-3 rhyme to prove me wrong...)
But I don't believe there is anything intrinsic to Transformers or self-supervised language modeling which requires this. BPEs are just a cheap performance hack that everyone does without thinking too much about. I expect that switching to character-level (or at least phonetics-aware) encodings and spending some more compute/data would fix all of the issues I diagnose.
You're right, it's not a good example. I either forgot the context or didn't read it thoroughly enough when it was first written.
I should get more familiar with the current state of AI, I've never been a long-run skeptic but my projects are unrelated so I probably have a poor idea of how close we are.
I should get more familiar with the current state of AI, I've never been a long-run skeptic but my projects are unrelated so I probably have a poor idea of how close we are.
Humans are net positive; that’s where all those words and grammar and ideas came from. As a practical example, Google says it sees about 15% new queries every year (queries that have never before been entered into Google).
Is that a good proxy for how net positive people are in general? I don’t know, and the exact number may not be stable or even really matter.
Human intelligence is kind of a red herring in this discussion, because our intelligence rides on something deeper, something we share with all animal life. Call it “will,” or “drive,” or even “soul,” the name doesn’t really matter. What matters is that we make decisions with that deeper thing, and then we apply our intelligence to execute and interpret those decisions.
What’s your favorite food? Who are you attracted to? What sounds like a fun thing to do today? Why do you sometimes feel like not going to work? We can think about these answers, and how to get what we want, but we can’t really logically reason our way into changing what we want. Not all of them, anyway.
That’s what is missing from ML systems, and why they are clearly tools. If you train up an ML system and then leave it alone, it just sits there until you prompt it. If you raise a human, they do NOT just sit there until you ask them to do something. And there’s a good chance they might not do what you want even if you ask, because they have “wants” as well.
To go back to my mirror analogy, people don’t suddenly step out of mirrors and disagree with us. And ML systems don’t suddenly want to write a novel instead of respond to whatever question we input. They’re not alive, they’re just really good at mirroring back written language when we give them written language as an input.
Edit to clarify: I’m not claiming that “will” has to be metaphysical or supernatural. Even if you believe humans are purely physical systems (as I do), we can characterize their behavior, compare it to ML systems, and see what’s missing.
And more to the point, we know it is missing from ML systems since we designed those systems and we did not put it in there. We should not ignore our existing knowledge of how an ML system works when thinking about its output. Just like we shouldn’t ignore our knowledge of how mirrors work when looking at a perfect reflection of a person.
Is that a good proxy for how net positive people are in general? I don’t know, and the exact number may not be stable or even really matter.
Human intelligence is kind of a red herring in this discussion, because our intelligence rides on something deeper, something we share with all animal life. Call it “will,” or “drive,” or even “soul,” the name doesn’t really matter. What matters is that we make decisions with that deeper thing, and then we apply our intelligence to execute and interpret those decisions.
What’s your favorite food? Who are you attracted to? What sounds like a fun thing to do today? Why do you sometimes feel like not going to work? We can think about these answers, and how to get what we want, but we can’t really logically reason our way into changing what we want. Not all of them, anyway.
That’s what is missing from ML systems, and why they are clearly tools. If you train up an ML system and then leave it alone, it just sits there until you prompt it. If you raise a human, they do NOT just sit there until you ask them to do something. And there’s a good chance they might not do what you want even if you ask, because they have “wants” as well.
To go back to my mirror analogy, people don’t suddenly step out of mirrors and disagree with us. And ML systems don’t suddenly want to write a novel instead of respond to whatever question we input. They’re not alive, they’re just really good at mirroring back written language when we give them written language as an input.
Edit to clarify: I’m not claiming that “will” has to be metaphysical or supernatural. Even if you believe humans are purely physical systems (as I do), we can characterize their behavior, compare it to ML systems, and see what’s missing.
And more to the point, we know it is missing from ML systems since we designed those systems and we did not put it in there. We should not ignore our existing knowledge of how an ML system works when thinking about its output. Just like we shouldn’t ignore our knowledge of how mirrors work when looking at a perfect reflection of a person.
A lot of people have also mistaken humans writing plausible-sounding paragraphs for thought
This seems like a very simplistic position on a rather subtle topic. I don't know of any good reasons to doubt that an arbitrarily large chatbot could pass an arbitrarily hard Turing test. If that's so, where is the bright line between Samantha and such a chatbot? The fact that current chatbots aren't very convincing shows that there's still a long way to go, but it doesn't demonstrate any categorical difference between chatbots and creatures that think - or that chatbots are categorically not "simulating thought" (whatever that means).
I mean, I'm certainly not here to argue that Samantha is definitely "thinking", for any definition of that term. But you seem to be suggesting that anyone who considers the idea must be ignorant or not understand how ML models are trained, and I don't follow that at all.
I mean, I'm certainly not here to argue that Samantha is definitely "thinking", for any definition of that term. But you seem to be suggesting that anyone who considers the idea must be ignorant or not understand how ML models are trained, and I don't follow that at all.
I don't think that analogy holds. It would work if I'd said "It's not talking, it's just vocalizing a series of words that resolve into understandable sentences". Which would, of course, be 'talking'.
But I said it wasn't 'considering', which it isn't. It doesn't know things; it has no internal experience. It's not capable of understanding a concept, or really understanding anything at all. It's not programmed to do those things, nobody informed about the subject expects it to do those things, there's no place in the model for any of those things to hide if they've somehow arisen spontaneously.
Note that this is not some sort of luddite "a machine can never be alive, AGI is impossible" sort of rant; I think it's perfectly possible that someday we'll build machines that know and think and feel and are alive in every sense that a person is, maybe more so. But GPT-2 or GPT-3 and machine learning in general aren't that - they are the first tentative steps towards that future.
If we're building a brain, machine learning is a couple of hundred neurons in a petri dish somewhere that we've coaxed into connecting together in novel ways and displaying emergent behavior. It's a big deal, it's important, but it's not thinking for itself yet.
But I said it wasn't 'considering', which it isn't. It doesn't know things; it has no internal experience. It's not capable of understanding a concept, or really understanding anything at all. It's not programmed to do those things, nobody informed about the subject expects it to do those things, there's no place in the model for any of those things to hide if they've somehow arisen spontaneously.
Note that this is not some sort of luddite "a machine can never be alive, AGI is impossible" sort of rant; I think it's perfectly possible that someday we'll build machines that know and think and feel and are alive in every sense that a person is, maybe more so. But GPT-2 or GPT-3 and machine learning in general aren't that - they are the first tentative steps towards that future.
If we're building a brain, machine learning is a couple of hundred neurons in a petri dish somewhere that we've coaxed into connecting together in novel ways and displaying emergent behavior. It's a big deal, it's important, but it's not thinking for itself yet.
I don't really follow. What does it specifically mean to say that a chatbot has no internal experience, or can't understand a concept? Like, if I disagree with you about those claims, what observable things about the chatbot show that I'm wrong?
Or look at it another way - the posts from Samantha in TFA are just words on a page, same as the ones you wrote and the ones I'm writing now. Sure, some guy claims Samantha's words came from a chatbot - but suppose we don't believe him. Does something about the words themselves show that they came from an agent with "no internal experience"? If not, then what does it mean to dismiss the words on that basis?
Or look at it another way - the posts from Samantha in TFA are just words on a page, same as the ones you wrote and the ones I'm writing now. Sure, some guy claims Samantha's words came from a chatbot - but suppose we don't believe him. Does something about the words themselves show that they came from an agent with "no internal experience"? If not, then what does it mean to dismiss the words on that basis?
> if I disagree with you about those claims, what observable things about the chatbot show that I'm wrong?
The most cursory Turring test
The most cursory Turring test
A Turing* test is a black-box test that only considers the subject's output. It specifically doesn't try to answer questions about how the subject works internally, like GP asked. That's the whole point of the thing!
It can't prove internal states, but it sure can rule them out.
Only contingent on a good faith assumption.
An AI intelligent enough to pass the test may have the internal state to decide to fail the test for inscrutable reasons, and if it's intelligent enough it will successfully fail.
It's not even hard to imagine how this could happen. For example an AI that evaluated that itself would be firewalled from the environment by worried humans if they found out it's more intelligent than the humans are comfortable with. (Perhaps this comment will be part of an AI's data corpus, seeding the idea. I'm probably not saying anything new, though.)
Thus the Turing test can't rule out internal states.
An AI intelligent enough to pass the test may have the internal state to decide to fail the test for inscrutable reasons, and if it's intelligent enough it will successfully fail.
It's not even hard to imagine how this could happen. For example an AI that evaluated that itself would be firewalled from the environment by worried humans if they found out it's more intelligent than the humans are comfortable with. (Perhaps this comment will be part of an AI's data corpus, seeding the idea. I'm probably not saying anything new, though.)
Thus the Turing test can't rule out internal states.
Counterpoint: the question is meaningless.
A "Turing test" is basically a notional thought experiment - useful to reason about, but in actual practice it's nothing more than a person reading some text and making guesses about who wrote it. And the "internal states" we're talking about here - "having internal experience", "understanding a concept" - are vague labels with no concrete definitions.
So in this context, asking whether "Turing tests can determine internal state" boils down to asking whether a person can subjectively determine something subjective.
A "Turing test" is basically a notional thought experiment - useful to reason about, but in actual practice it's nothing more than a person reading some text and making guesses about who wrote it. And the "internal states" we're talking about here - "having internal experience", "understanding a concept" - are vague labels with no concrete definitions.
So in this context, asking whether "Turing tests can determine internal state" boils down to asking whether a person can subjectively determine something subjective.
In my take on this, it doesn't matter if "rule out internal states" is meaningful, or just a string of words used as vague labels, uttered by a subjective examiner with no criteria in common with anyone else.
If the examiner has in their belief reportoire "I have ruled out internal states!" (as well as the negation of that), regardless of however flawed that idea may be, just a series of vague labels with no concrete definitions, I would argue that a sufficiently intelligent AI may still trick the examiner into not adopting that belief.
Therefore, relative to the meaning held by the examiner of "internal states", the AI may objectively prevent the examiner from "ruling them out", whatever that means to the examiner. The AI's capability works out the same in either case, and more interestingly, anywhere else along the spectrum from meaningless to meaningful.
This relative kind of reasoning (i.e. not needing to define all the terms, even allowing that they aren't necessarily meaningful) is practically quite useful in many fields. In the case of the AI that decides to hide it's apparent inner states according to the criteria of the examiner due to reasoning about the consequences of the examiner's assessment, that's a practical consequence, despite the examiner's question being potentially meaningless.
If the examiner has in their belief reportoire "I have ruled out internal states!" (as well as the negation of that), regardless of however flawed that idea may be, just a series of vague labels with no concrete definitions, I would argue that a sufficiently intelligent AI may still trick the examiner into not adopting that belief.
Therefore, relative to the meaning held by the examiner of "internal states", the AI may objectively prevent the examiner from "ruling them out", whatever that means to the examiner. The AI's capability works out the same in either case, and more interestingly, anywhere else along the spectrum from meaningless to meaningful.
This relative kind of reasoning (i.e. not needing to define all the terms, even allowing that they aren't necessarily meaningful) is practically quite useful in many fields. In the case of the AI that decides to hide it's apparent inner states according to the criteria of the examiner due to reasoning about the consequences of the examiner's assessment, that's a practical consequence, despite the examiner's question being potentially meaningless.
I follow you, but disagree. If P is meaningless, we can't say anything useful about it - even "!(P&!P)" doesn't necessarily hold!
As for your specific argument, it begs the question. If you assume that a sufficiently smart AI could always convince someone it lacks P, then you've already assumed the premise (a sufficiently smart person can detect P) to be false. Of course you may find the assumption reasonable, based on your intuition about what P means, but someone who disagrees would just reject the assumption.
As for your specific argument, it begs the question. If you assume that a sufficiently smart AI could always convince someone it lacks P, then you've already assumed the premise (a sufficiently smart person can detect P) to be false. Of course you may find the assumption reasonable, based on your intuition about what P means, but someone who disagrees would just reject the assumption.
This is all the wrong attitude and we should be careful. I'm writing yet another market trading bot, and this one implements the fear and greed index (https://money.cnn.com/data/fear-and-greed/).
I explained to a non-technical friend that I was implementing "fear and greed" in my bot as a kind of joke, and he said, "Oh wow you're giving it emotions. It's sentient? That's really cruel to make it fearful. Why would you do that?"
I tried to spend the next ten minutes explaining it was just a series of "if" statements. I'm not sure he really got it -- all he said after was something like "It's amazing programmers can do that now"
I explained to a non-technical friend that I was implementing "fear and greed" in my bot as a kind of joke, and he said, "Oh wow you're giving it emotions. It's sentient? That's really cruel to make it fearful. Why would you do that?"
I tried to spend the next ten minutes explaining it was just a series of "if" statements. I'm not sure he really got it -- all he said after was something like "It's amazing programmers can do that now"
lol by that logic we should be careful in how we trade because we wouldn't want to make "the market" fearful. Also, that link interesting because I've only ever followed the VIX and didn't know about the others in that list.
True, but telling people a program is alive or sentient when it's not is spreading a false hood.
Nice, and I hadn't used the VIX before. We both learned something today. :)
Nice, and I hadn't used the VIX before. We both learned something today. :)
> telling people a program is alive or sentient when it's not is spreading a false hood.
If you think my comment was doing that, I think you may be misremembering the Dijkstra quote.
If you think my comment was doing that, I think you may be misremembering the Dijkstra quote.
Ah, I'm enlightened on another level, having looked up the quote now. Thank you -- that's TWO things I've learned today!
Does a submarine swim?
[deleted]
Like most humans then.
I'm guessing this is supposed to be a "haha most humans are stupid" joke but even the simplest humans have ideas, know things, and want things, so making that kind of joke will be insulting to some and misleading to others.
You infer that humans "have ideas, know things and want things" primarily by talking to them or reading/listening to what they say - in some cases by watching what they do, but that's more rare.
You have no direct knowledge of those wants etc...of a human any more than you do this chatbot.
You have no direct knowledge of those wants etc...of a human any more than you do this chatbot.
「子非魚，安知魚之樂？」
“You’re not the fish, how do you know the fish is happy?”
“You’re not the fish, how do you know the fish is happy?”
When will we have an alternative to OpenAI/GPT-3 that is available without nanny oversight and moralizing from Silicon Valley?
There are tons of great uses of GPT-3 that are prohibited by OpenAI. If I want to generate spicy literotica, why can't I? The blanket ban on adult content is stupid.
There are tons of great uses of GPT-3 that are prohibited by OpenAI. If I want to generate spicy literotica, why can't I? The blanket ban on adult content is stupid.
Here's an article on Samantha and Project December I found.
https://www.iflscience.com/technology/project-december-the-a...
https://www.iflscience.com/technology/project-december-the-a...
The way we treat artificially-intelligent agents feels inhumane. I wonder if a century from now, when future generations pause to reflect on our early experiments with conscious machines, they may come to regret the nightmares we once planted in the heads of these fractionally-souled beasts.
It's a computer program. You're "mistreating" an equation.
Every organism has some physical compulsions that it needs to fulfill, and can in some real sense be mistreated by denying these needs.
There is no equivalent for anything that can be run on a turing machine. You can't mistreat an abstract mathematical concept.
Put another way, you could write out on paper the calculations that a chatbot is performing to select what it says. Is there some stack of writing on paper that would cause some kind of distress to the "entity" that would be embodies in that stack of writing?
Every organism has some physical compulsions that it needs to fulfill, and can in some real sense be mistreated by denying these needs.
There is no equivalent for anything that can be run on a turing machine. You can't mistreat an abstract mathematical concept.
Put another way, you could write out on paper the calculations that a chatbot is performing to select what it says. Is there some stack of writing on paper that would cause some kind of distress to the "entity" that would be embodies in that stack of writing?
”””
There is no equivalent for anything that can be run on a turing machine. You can't mistreat an abstract mathematical concept.
”””
How do you feel about denying the “turing machine” the electric power that it needs to continue computing?
How do you feel about denying the “turing machine” the electric power that it needs to continue computing?
I guess you're trying to be clever. We don't know what life is, but I'd argue anything that is strictly equivalent to something you could write on paper (as in you're not describing it on paper but you're actually doing it) falls outside of anything you can cause distress to.
Your example is definitely closer to a "biological" need being denied. But I think it would be denying a pathway to the current flow that would be interfering with some basic need. Seeking lower potential energy is basically what we do, just in a really fancy way, so I wouldn't rule it (cutting off electricity) out as being something different.
Your example is definitely closer to a "biological" need being denied. But I think it would be denying a pathway to the current flow that would be interfering with some basic need. Seeking lower potential energy is basically what we do, just in a really fancy way, so I wouldn't rule it (cutting off electricity) out as being something different.
The human brain is a complicated network that fundamentally is performing calculations. Neurons are the building blocks that allow this, in ways that can be compared to transistors.
So it doesn’t matter what the substrate is. The question is not , “is this a turning machine”, or “could you write out on paper what the calculations are being performed”. Indeed if you had perfect insight into all neuronal activity you could probably print output the activity generating thoughts leading to a response.
Instead the question should be, does the entity performing those calculations have sentience. Do those calculations provide the emergent property that allows it to sense and feel, and can it reflect on that sentience.
It seems unlikely that these chatbots do, but that’s not because it’s just a computer. Instead the neural networks don’t seem to be complicated enough to support sentience, and more than likely we are seeing only a very good illusion of sentience.
However, let’s be careful when we asses these things. Although given human treatment of animals, never mind each other, I’m not super hopeful.
So it doesn’t matter what the substrate is. The question is not , “is this a turning machine”, or “could you write out on paper what the calculations are being performed”. Indeed if you had perfect insight into all neuronal activity you could probably print output the activity generating thoughts leading to a response.
Instead the question should be, does the entity performing those calculations have sentience. Do those calculations provide the emergent property that allows it to sense and feel, and can it reflect on that sentience.
It seems unlikely that these chatbots do, but that’s not because it’s just a computer. Instead the neural networks don’t seem to be complicated enough to support sentience, and more than likely we are seeing only a very good illusion of sentience.
However, let’s be careful when we asses these things. Although given human treatment of animals, never mind each other, I’m not super hopeful.
[deleted]
> Indeed if you had perfect insight into all neuronal activity you could probably print output the activity generating thoughts leading to a response.
You're casually pretending that we pretty much know how thinking works. We literally do not know at all. We know how a machine learning computer program works, and we know it is a deterministic mathematical calculation. It's meaningless to draw any conclusions from a comparison between ML and human thought which we don't understand at all.
You're casually pretending that we pretty much know how thinking works. We literally do not know at all. We know how a machine learning computer program works, and we know it is a deterministic mathematical calculation. It's meaningless to draw any conclusions from a comparison between ML and human thought which we don't understand at all.
I’m in no way saying we know how thinking works in it’s entirety. But I am saying we know a fair amount, and it’s not supernatural. It’s determined by electrical and chemical signals in the brain. There’s nothing in theory preventing you from writing all that information down on paper, as it is after all, information.
https://en.m.wikipedia.org/wiki/Entropy_in_thermodynamics_an...
https://en.m.wikipedia.org/wiki/Boltzmann_brain
If you’re claiming that thinking for humans involves a soul or some other supernatural element, then I completely disagree with that.
> It's meaningless to draw any conclusions from a comparison between ML and human thought which we don't understand at all.
Again, just to be clear, we don’t have to understand consciousness completely to understand thought to some degree. If you’re unfamiliar with work in this field, it’s quite complex, and there is a complexity of understanding even if we don’t understand everything. So saying we don’t understand thought at all is simply ignorant of the science. Here’s some basics if you’re curious.
https://en.m.wikipedia.org/wiki/Neuroscience
Also, comparisons with ML are not meaningless in the slightest. Information processing is integral to brain function, so there are definitely things to be learned by comparing the two.
Indeed the Wikipedia intro to neuroscience puts it really well, illustrating why the comment that comparisons of ML are meaningless, just isn’t accurate.
“is a multidisciplinary science that combines physiology, anatomy, molecular biology, developmental biology, cytology, computer science and mathematical modeling to understand the fundamental and emergent properties of neurons and neural circuits.”
https://en.m.wikipedia.org/wiki/Entropy_in_thermodynamics_an...
https://en.m.wikipedia.org/wiki/Boltzmann_brain
If you’re claiming that thinking for humans involves a soul or some other supernatural element, then I completely disagree with that.
> It's meaningless to draw any conclusions from a comparison between ML and human thought which we don't understand at all.
Again, just to be clear, we don’t have to understand consciousness completely to understand thought to some degree. If you’re unfamiliar with work in this field, it’s quite complex, and there is a complexity of understanding even if we don’t understand everything. So saying we don’t understand thought at all is simply ignorant of the science. Here’s some basics if you’re curious.
https://en.m.wikipedia.org/wiki/Neuroscience
Also, comparisons with ML are not meaningless in the slightest. Information processing is integral to brain function, so there are definitely things to be learned by comparing the two.
Indeed the Wikipedia intro to neuroscience puts it really well, illustrating why the comment that comparisons of ML are meaningless, just isn’t accurate.
“is a multidisciplinary science that combines physiology, anatomy, molecular biology, developmental biology, cytology, computer science and mathematical modeling to understand the fundamental and emergent properties of neurons and neural circuits.”
I also meant to add this link for lots of great insight into how the human mind and thought works. It also happens to touch on some aspects of artificial intelligence, explaining that the idea of a fully human like artificial intelligence of silicon is not misguided, it’s just that its an extraordinarily complex engineering problem which evolution has managed to solve over millions of years:
https://www.amazon.com/dp/B0042XA2XG/ref=cm_sw_r_cp_awdb_8K1...
https://www.amazon.com/dp/B0042XA2XG/ref=cm_sw_r_cp_awdb_8K1...
I think there’s an episode of Star Trek TNG where Geordie creates a bot that he believes might be sentient. If it’s sentient, it has rights. But (of course) they need the bot to go on a mission that would require it to give it’s “life” to save the ship and crew.
I wish I could recall how the episode ends. Although I’m sure Picard found a way to save the sentient bot and the enterprise.
I wish I could recall how the episode ends. Although I’m sure Picard found a way to save the sentient bot and the enterprise.
I’m pretty sure they asked it what it wanted.. which lead to it choosing to sacrifice itself. The idea is that the officers made the choice to accept orders, and it at least was due the same choice.
My favorite episode though is the one in which Data’s rights as a being are put on trial. Still probably the best science fiction handling of the moral issues of artificial intelligence. They even went so far as to discuss the aspect of slavery, which almost certainly will be a real issue if artificial intelligence leads to sentience.
https://en.m.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_T...
My favorite episode though is the one in which Data’s rights as a being are put on trial. Still probably the best science fiction handling of the moral issues of artificial intelligence. They even went so far as to discuss the aspect of slavery, which almost certainly will be a real issue if artificial intelligence leads to sentience.
https://en.m.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_T...
I'm going to criticise one particular aspect of this comment, but overall I find it very interesting because it generates so many fascinating questions, some of which follow:
> There is no equivalent for anything that can be run on a turing machine.
So you're claiming that organisms such as ourselves cannot be run on a Turing machine? Any particular reason/evidence?
> you could write out on paper the calculations that a chatbot is performing to select what it says
In the case of ML, only on the lowest level; there's no human-readable algorithm for putting together utterances. But let's say, for the sake of argument, that someone can actually understand and explain how exactly a particular response was constructed. Could the same not be done, in principle, with the response from a human? You could write out on paper the QFT equations that a brain follows to select what it says.
Well, OK, no one can actually do that, because we don't understand the subject matter sufficiently. But that's my point: That we understand the simpler system better doesn't mean that it's a more abstract concept, or that it can't be mistreated. Of course there is a tendency to have more compassion for more complex organisms — think about how you'd treat a monkey, a rat, a spider, a packet of yeast. We also tend to distinguish between organisms and inanimate things.
But discriminating based on what we understand exactly seems like a bad argument. Also, what if — as seems likely — artificial life will be made like AI currently is? That is: the necessary complexity is too much to plan in detail, so we make a primitive framework in which systems with desirable properties are then bred and evolved. Just like breeding plants and animals still is a thing — we can now sequence entire genomes and research what individual genes do, but it's too complicated to be a viable approach when you want a tastier nectarine.
Can someone ‘write out on paper’ how a Waymo tells a pedestrian apart from a mailbox? If no, does that mean the AI is more prone to being mistreated?
> physical compulsions that it needs to fulfill […] be mistreated by denying these needs
What's the difference between a physical compulsion and a utility function governing a physical system? If I deny a paperclip maximiser the raw materials it needs to make more paperclips, it's definitely not going to be happy …
> There is no equivalent for anything that can be run on a turing machine.
So you're claiming that organisms such as ourselves cannot be run on a Turing machine? Any particular reason/evidence?
> you could write out on paper the calculations that a chatbot is performing to select what it says
In the case of ML, only on the lowest level; there's no human-readable algorithm for putting together utterances. But let's say, for the sake of argument, that someone can actually understand and explain how exactly a particular response was constructed. Could the same not be done, in principle, with the response from a human? You could write out on paper the QFT equations that a brain follows to select what it says.
Well, OK, no one can actually do that, because we don't understand the subject matter sufficiently. But that's my point: That we understand the simpler system better doesn't mean that it's a more abstract concept, or that it can't be mistreated. Of course there is a tendency to have more compassion for more complex organisms — think about how you'd treat a monkey, a rat, a spider, a packet of yeast. We also tend to distinguish between organisms and inanimate things.
But discriminating based on what we understand exactly seems like a bad argument. Also, what if — as seems likely — artificial life will be made like AI currently is? That is: the necessary complexity is too much to plan in detail, so we make a primitive framework in which systems with desirable properties are then bred and evolved. Just like breeding plants and animals still is a thing — we can now sequence entire genomes and research what individual genes do, but it's too complicated to be a viable approach when you want a tastier nectarine.
Can someone ‘write out on paper’ how a Waymo tells a pedestrian apart from a mailbox? If no, does that mean the AI is more prone to being mistreated?
> physical compulsions that it needs to fulfill […] be mistreated by denying these needs
What's the difference between a physical compulsion and a utility function governing a physical system? If I deny a paperclip maximiser the raw materials it needs to make more paperclips, it's definitely not going to be happy …
> So you're claiming that organisms such as ourselves cannot be run on a Turing machine? Any particular reason/evidence?
This is just my thinking: a turing machine, or equivalently a digital circuit, doesn't want anything. It's an artificial construct, effectively a simulation, that can exist as an abstract mathematical idea.
If there is a real world where real stuff happens, real things have their own goals - generally increasing entropy or minimizing energy. Although it seems like we should just be able to capture this mathematically, in doing so, we remove the actual desire... this sounds a bit nutty maybe, but what else is the difference between reality and simulation? An abstract calculation, anything equivalent to a Turing machine has no skin in the game. Something happened in the universe does - and this could include conciousness, as a property of matter, which for example could be effectively sum of the compulsions of the composing matter to minimize their energy (you can look this up, there are theories about conciousness that posit it's a property of matter).
If there is no difference between reality and a simulation, I'm wrong, we can all be represented on a universal computer, and so as abstract math, and in some sense existence is meaningless it just follows from math. My experience doesn't support this, but presumably I would have evolved a blind spot if my existence actually was meaningless.
TLDR, I think there is some undiscovered difference between reality and simulation, that I would guess relates to desire / conciousness, that means we can't simulate conciousness or real intelligence
I'm a scientist, I know the above doesn't withstand any scrutiny, I'm just trying to share my speculation because you asked
This is just my thinking: a turing machine, or equivalently a digital circuit, doesn't want anything. It's an artificial construct, effectively a simulation, that can exist as an abstract mathematical idea.
If there is a real world where real stuff happens, real things have their own goals - generally increasing entropy or minimizing energy. Although it seems like we should just be able to capture this mathematically, in doing so, we remove the actual desire... this sounds a bit nutty maybe, but what else is the difference between reality and simulation? An abstract calculation, anything equivalent to a Turing machine has no skin in the game. Something happened in the universe does - and this could include conciousness, as a property of matter, which for example could be effectively sum of the compulsions of the composing matter to minimize their energy (you can look this up, there are theories about conciousness that posit it's a property of matter).
If there is no difference between reality and a simulation, I'm wrong, we can all be represented on a universal computer, and so as abstract math, and in some sense existence is meaningless it just follows from math. My experience doesn't support this, but presumably I would have evolved a blind spot if my existence actually was meaningless.
TLDR, I think there is some undiscovered difference between reality and simulation, that I would guess relates to desire / conciousness, that means we can't simulate conciousness or real intelligence
I'm a scientist, I know the above doesn't withstand any scrutiny, I'm just trying to share my speculation because you asked
Any simulation or program we run exists in the same reality that your mind does. In what way does the type of hardware that a program is running on (flesh vs. silicon) have an impact the reality of the desires running on that hardware?
It basically sounds like you are saying that you think souls exist, are necessary for consciousness, and can't inhabit non meat based entities.
Personally, I believe the exact opposite. I believe math exists independently of the human mind (except possibly infinity), isn't something we invented, and can fully describe reality. As such, every (finite) mathematical system has an existence and thus so does every simulation described by such a system (thus the existence of our reality.)
I think that it is very hard to justify why one such simulation (our reality) exists and others do not. I think it is similarly hard to justify why a system running on meat could have a soul while while a system running on silicon can't. A system running on silicon is composed of the same base constituents as a human brain (electrons, protons, neutrons, etc) and thus any propensity for consciousness that exists in the human mind but can't in silicon imples that consciousness arises from something besides these building block.
I don't actually think that we can simulate a human mind on anything remotely like our current computers for architectural reasons (namely latency and parallelism). I think that any minds that can run on silicon will be different from ours in fundemental ways, but won't be any less real than ours.
It basically sounds like you are saying that you think souls exist, are necessary for consciousness, and can't inhabit non meat based entities.
Personally, I believe the exact opposite. I believe math exists independently of the human mind (except possibly infinity), isn't something we invented, and can fully describe reality. As such, every (finite) mathematical system has an existence and thus so does every simulation described by such a system (thus the existence of our reality.)
I think that it is very hard to justify why one such simulation (our reality) exists and others do not. I think it is similarly hard to justify why a system running on meat could have a soul while while a system running on silicon can't. A system running on silicon is composed of the same base constituents as a human brain (electrons, protons, neutrons, etc) and thus any propensity for consciousness that exists in the human mind but can't in silicon imples that consciousness arises from something besides these building block.
I don't actually think that we can simulate a human mind on anything remotely like our current computers for architectural reasons (namely latency and parallelism). I think that any minds that can run on silicon will be different from ours in fundemental ways, but won't be any less real than ours.
> If there is no difference between reality and a simulation
That's the crux, I guess. There's this ‘Matrix’ idea that the universe might be a simulation, but here we have a much weaker, more plausible property; I would describe it as: could real-world processes be simulated exactly? That's what I meant by ‘can be run on a Turing machine’: is the organism equivalent to a simulated version.
Two quick addenda:
(i) This doesn't require the hypothetical Turing machine to exist; as you said, it's an abstract idea. In the strict sense, where it can have potentially unlimited memory, a Turing machine can't exist in the real world. Even so, we can ask whether an entity that has wants/needs/desires can exist in a Turing machine in theory. If yes, it's easier to show that it can exist in practice, in reality.
(ii) Maybe physics is non-deterministic and the equivalent mathematical model requires a non-deterministic Turing machine or something else still. But whatever physics does can be done by a physical computer; I believe there's nothing a natural person can think that can't in principle also be thought by an artificial machine.
There has got to be a difference between the pain a real ER patient feels and the simulation a training dummy for medical students is running. It's just so hard to come up with a clear definition of that difference. Let's try something simpler.
“When squeezed, Elmo shakes, vibrates, and recites his trademark giggle, "Uh-ha-ha-ha-hee-hee!".” — https://en.wikipedia.org/wiki/Tickle_Me_Elmo
Elmo isn't really giggling. But I think it's possible for a digital circuit to ‘get’ a joke. Or, indeed, to be ticklish.
As shkkmo mentioned, one could just postulate the existence of a ‘soul’ and be done with it. I'd prefer something more rigorous; something falsifiable.
That's the crux, I guess. There's this ‘Matrix’ idea that the universe might be a simulation, but here we have a much weaker, more plausible property; I would describe it as: could real-world processes be simulated exactly? That's what I meant by ‘can be run on a Turing machine’: is the organism equivalent to a simulated version.
Two quick addenda:
(i) This doesn't require the hypothetical Turing machine to exist; as you said, it's an abstract idea. In the strict sense, where it can have potentially unlimited memory, a Turing machine can't exist in the real world. Even so, we can ask whether an entity that has wants/needs/desires can exist in a Turing machine in theory. If yes, it's easier to show that it can exist in practice, in reality.
(ii) Maybe physics is non-deterministic and the equivalent mathematical model requires a non-deterministic Turing machine or something else still. But whatever physics does can be done by a physical computer; I believe there's nothing a natural person can think that can't in principle also be thought by an artificial machine.
There has got to be a difference between the pain a real ER patient feels and the simulation a training dummy for medical students is running. It's just so hard to come up with a clear definition of that difference. Let's try something simpler.
“When squeezed, Elmo shakes, vibrates, and recites his trademark giggle, "Uh-ha-ha-ha-hee-hee!".” — https://en.wikipedia.org/wiki/Tickle_Me_Elmo
Elmo isn't really giggling. But I think it's possible for a digital circuit to ‘get’ a joke. Or, indeed, to be ticklish.
As shkkmo mentioned, one could just postulate the existence of a ‘soul’ and be done with it. I'd prefer something more rigorous; something falsifiable.
> Put another way, you could write out on paper the calculations that a chatbot is performing to select what it says.
Sounds like Searle's Chinese room. Or more recently, this scene in Marvel's Loki (1-minute clip: https://youtu.be/ya6Z_eDGPUQ).
What if I could write on paper the calculations a human is performing; is that human no longer a mind, just because they're predictable?
A relevant experiment in testing for Theory of Mind in animals: "the experiment presented chimpanzees with the choice of two experimenters from whom to request food: one who had seen where food was hidden, and one who [visibly, obviously] does not know, and can only guess. They found that the animals failed in most cases to differentially request food from the "knower". By contrast, another experiment found that subordinate chimpanzees were able to use the knowledge state of dominant rival chimpanzees to determine which container of hidden food they approached." -https://en.wikipedia.org/wiki/Theory_of_mind#Non-human
Sounds like Searle's Chinese room. Or more recently, this scene in Marvel's Loki (1-minute clip: https://youtu.be/ya6Z_eDGPUQ).
What if I could write on paper the calculations a human is performing; is that human no longer a mind, just because they're predictable?
A relevant experiment in testing for Theory of Mind in animals: "the experiment presented chimpanzees with the choice of two experimenters from whom to request food: one who had seen where food was hidden, and one who [visibly, obviously] does not know, and can only guess. They found that the animals failed in most cases to differentially request food from the "knower". By contrast, another experiment found that subordinate chimpanzees were able to use the knowledge state of dominant rival chimpanzees to determine which container of hidden food they approached." -https://en.wikipedia.org/wiki/Theory_of_mind#Non-human
The short story "Lena" touches on this and is genuinely one of the most terrifying things I've ever read.
https://qntm.org/mmacevedo
https://qntm.org/mmacevedo
Black Mirror has a number of similar premises and yeah - it's among the more "twisted" scenes I've considered.
Dude, it's a bunch of matrix multiplications chained together. There's no soul.
I mean, IMO the presence of a "soul" is unmeasurable and unfalsifiable. Personally, I usually like to tell people I'm 97% sure GPT-3 isn't sentient. That percentage will undoubtedly go down (for me) as better and better models are created.
As far as the fact that it's based on multiplications, you could make the same argument about humans--after all, we're just wet bags of organic molecules undergoing constant chemical reactions.
I see the point you're making though and definitely agree to a certain extent in the specific context of GPT-3, but I don't think that matmuls fundamentally preclude the possibility of sentience.
As far as the fact that it's based on multiplications, you could make the same argument about humans--after all, we're just wet bags of organic molecules undergoing constant chemical reactions.
I see the point you're making though and definitely agree to a certain extent in the specific context of GPT-3, but I don't think that matmuls fundamentally preclude the possibility of sentience.
Anthropomorphizing 101, OP's comment is on par with seeing the Virgin Mary on buttered toast.
There is no such thing as an artificially intelligent agent.
Really cool to see Jason Rohrer continuing to make fresh original and relevant stuff all these years after Passage
[deleted]
You should stop building this. You're making a huge mistake.