John Carmack and Rich Sutton partner to accelerate development of AGI(amii.ca)
amii.ca
John Carmack and Rich Sutton partner to accelerate development of AGI
https://www.amii.ca/latest-from-amii/john-carmack-and-rich-sutton-agi/
225 comments
11 days ago : https://news.ycombinator.com/item?id=37654653, 82 comments
Their focus is long running digital intelligences that interact with their environment over long periods of time. I enjoyed reading The Alberta Plan when it was released earlier this year https://arxiv.org/abs/2208.11173
Ethics/privacy aside ...
I wondered the other day what the result would be if you strapped a microphone on a baby and every utterance that both the baby and mic heard was fed through speech-to-text and went toward training the baby's personal LLM.
By the time the child had grown to an adult I wonder what kind of results their LLM would produce and the degree to which it might compare to what the child (now grown) would answer?
An "LLM" that could take in visuals via a baby-mounted camera is of course a whole other discussion. (Though I'm sure there are people training ANN's with video feeds rather than the static images that feed systems like DALL•E.)
I wondered the other day what the result would be if you strapped a microphone on a baby and every utterance that both the baby and mic heard was fed through speech-to-text and went toward training the baby's personal LLM.
By the time the child had grown to an adult I wonder what kind of results their LLM would produce and the degree to which it might compare to what the child (now grown) would answer?
An "LLM" that could take in visuals via a baby-mounted camera is of course a whole other discussion. (Though I'm sure there are people training ANN's with video feeds rather than the static images that feed systems like DALL•E.)
>I wondered the other day what the result would be if you strapped a microphone on a baby and every utterance that both the baby and mic heard was fed through speech-to-text and went toward training the baby's personal LLM.
I read about a paper[1] a while back, it was a rather unpleasant animal study in cats. Using two kittens, one was free to look around, but the other they immobilised in some way, and made it see what the other kitten saw as it looked around. They discovered the immobilised kitten's visual processing did not develop normally whereas the mobile kitten's did, suggesting that it's not just the sensory input, but it being feedback to some internal agency within the brain.
I suspect the same thing would be true for attempts to develop AGI by giving them an audio-visual copy of a human's environment growing up: that internal state driving (and getting feedback from) the action to investigate/interact with the world is key.
1: https://arxiv.org/abs/1604.03670 "Interactive Perception: Leveraging Action in Perception and Perception in Action"
I read about a paper[1] a while back, it was a rather unpleasant animal study in cats. Using two kittens, one was free to look around, but the other they immobilised in some way, and made it see what the other kitten saw as it looked around. They discovered the immobilised kitten's visual processing did not develop normally whereas the mobile kitten's did, suggesting that it's not just the sensory input, but it being feedback to some internal agency within the brain.
I suspect the same thing would be true for attempts to develop AGI by giving them an audio-visual copy of a human's environment growing up: that internal state driving (and getting feedback from) the action to investigate/interact with the world is key.
1: https://arxiv.org/abs/1604.03670 "Interactive Perception: Leveraging Action in Perception and Perception in Action"
This isn't a very new concept -- I learned about this in the 90s. Even then the language researchers knew that interaction with the environment was crucial for learning in humans. There were some natural experiments where kids were born without limbs for example and had trouble with early language acquisition because they couldn't experience the environment the same way as most others.
And the AI folks argued even then that the only way to create a language model that truly "understands" would be to put it in a robot that can experience the world in the same way as a human.
And the AI folks argued even then that the only way to create a language model that truly "understands" would be to put it in a robot that can experience the world in the same way as a human.
> They discovered the immobilised kitten's visual processing did not develop normally whereas the mobile kitten's did, suggesting that it's not just the sensory input, but it being feedback to some internal agency within the brain.
> I suspect the same thing would be true for attempts to develop AGI...
On the other hand, I would not be surprised if this did _not_ generalize and translate to LLMs. Biological entities on earth tend to have tons of fairly arbitrary developmental legacy which may or may not translate well to an artificial one. In the experiment, they didn't make billions of clones of the cat with a variety of different parameters and check which one worked, they just observed that yes, this standard v1 cat has some specific hard-wired developmental tendencies. It is not surprising that taking it out of the environment it was evolved for messes things up, but I would not take the extra step that this means that something is essential in general.
> I suspect the same thing would be true for attempts to develop AGI...
On the other hand, I would not be surprised if this did _not_ generalize and translate to LLMs. Biological entities on earth tend to have tons of fairly arbitrary developmental legacy which may or may not translate well to an artificial one. In the experiment, they didn't make billions of clones of the cat with a variety of different parameters and check which one worked, they just observed that yes, this standard v1 cat has some specific hard-wired developmental tendencies. It is not surprising that taking it out of the environment it was evolved for messes things up, but I would not take the extra step that this means that something is essential in general.
I suppose a large part of learning is not just seeing things happen but also deciding to do something, executing on it, and then feeling the result. So not surprising.
Curious to know how you found this paper? I'd like to be able to find interesting papers but I've not found a place of curation unfortunately.
I'm afraid I don't remember how I found this paper, but it might be from following Dr Abeba Birhane[1], she's an anti-AI academic (which is why I followed her originally, to get reality checks / contrasting viewpoints), but she also posts and boosts a lot of interesting cognitive science (her field). I also follow Olivia Guest[2] for the same reason.
Many of the papers go over my head at times of course (despite my interest, I'm firmly a layman!)
More generally, I've had a lot of success with searching for meta-analysis papers on topics I'm interested in, since the authors have already done the hard work of researching the literature and presenting core takeaways (as well as providing terms for more searches).
1: https://bsky.app/profile/abeba.bsky.social or https://twitter.com/Abebab
2: https://bsky.app/profile/olivia.science or https://twitter.com/o_guest
Many of the papers go over my head at times of course (despite my interest, I'm firmly a layman!)
More generally, I've had a lot of success with searching for meta-analysis papers on topics I'm interested in, since the authors have already done the hard work of researching the literature and presenting core takeaways (as well as providing terms for more searches).
1: https://bsky.app/profile/abeba.bsky.social or https://twitter.com/Abebab
2: https://bsky.app/profile/olivia.science or https://twitter.com/o_guest
thank you for taking the time to answer and to provide links, appreciate it.
For what it's worth, this was a paper I was exposed to during grad school for robotics/CS. I'm not saying one has to go to school to get good curation of content, but it is certainly one way to be exposed to a well structured foundation of knowledge in your field.
I can see us walking into a situation where we make a robot that has motor functions, but we don't give it the equivalent of nerves and nerve endings so and so it doesn't feel pain. That would definitely warp a human's development.
But then, is it ethical to build in pain receptors? Feels like we're brushing up against the territory of gods.
But then, is it ethical to build in pain receptors? Feels like we're brushing up against the territory of gods.
What would pain receptors be in a robot?
I ask this as I think the ethical dilemma should more about implementing suffering as a long term damaging effect of a stimuli.
When I think about pain I see it as a level that triggers a need to change something. Our brains are perceiving this level in a very uncomfortable way so we call this pain.
So I really think that without consciousness we cannot implement pain or suffering as we humans perceive it.
I ask this as I think the ethical dilemma should more about implementing suffering as a long term damaging effect of a stimuli.
When I think about pain I see it as a level that triggers a need to change something. Our brains are perceiving this level in a very uncomfortable way so we call this pain.
So I really think that without consciousness we cannot implement pain or suffering as we humans perceive it.
> So I really think that without consciousness we cannot implement pain or suffering as we humans perceive it.
Perhaps, but I guess it would depend on your definition of consciousness. We do not extend that definition to most animals and yet they definitely feel pain.
You're right, the "feeling" of pain is ephemeral and hard to translate to a computer program. It would probably take a while doing potentially nasty experiments (to both man and machine) to nail it down.
Perhaps, but I guess it would depend on your definition of consciousness. We do not extend that definition to most animals and yet they definitely feel pain.
You're right, the "feeling" of pain is ephemeral and hard to translate to a computer program. It would probably take a while doing potentially nasty experiments (to both man and machine) to nail it down.
LLMs already do this in their own way. It's not like they learn to predict through magic osmosis. They make predictions and depending on how right or wrong they are, their neurons are shifted by grad descent to accommodate this error. Then they predict again.
The Text is the environment, The Text is the world and they are very much interacting with it.
The Text is the environment, The Text is the world and they are very much interacting with it.
Not really, LLM Decoders(without RLHF) are usually trained by teacher forcing, that means they just get learning singals from just predicting one token.
> By the time the child had grown to an adult I wonder what kind of results their LLM would produce and the degree to which it might compare to what the child (now grown) would answer?
It very likely would not compare. Humans are shaped by their subjective experiences not by incoming data and you cannot know what a subject is experiencing solely through the incoming data (except if you have a theory accurately modeling the mind of the subject which is exactly what we are currently missing).
Your LLM won't experience the subject's heartbreaks, joy, grief, shame, hope etc. It will have heard and be able to talk about those, but it will not give accurate answers about what it felt like. Also it won't be able to predict/model accurately how the subject has been changed by those experiences so it could make very wrong assumptions about what the subject could/would do in the future.
It very likely would not compare. Humans are shaped by their subjective experiences not by incoming data and you cannot know what a subject is experiencing solely through the incoming data (except if you have a theory accurately modeling the mind of the subject which is exactly what we are currently missing).
Your LLM won't experience the subject's heartbreaks, joy, grief, shame, hope etc. It will have heard and be able to talk about those, but it will not give accurate answers about what it felt like. Also it won't be able to predict/model accurately how the subject has been changed by those experiences so it could make very wrong assumptions about what the subject could/would do in the future.
You might enjoy this short story: http://hubski.com/pub/78001
Published in 1990, by the way.
Published in 1990, by the way.
I thought about this too. People say that LLM models are only saying the most common tokens that come after the previous token. And that this makes them incomparable to human intelligence.
But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.
But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.
The claim that "humans are basically long running LLMs" oversimplifies the complexity of human cognition and experience.
Humans don't just process information; they experience emotions, desires, and subjective experiences that are deeply intertwined with their cognition. LLMs don't have feelings, motivations, or consciousness.
Humans have inner subjective experience, self-awareness, and the ability to reflect on our own existence. LLMs don't.
Humans can adapt to a wide range of environments and situations, drawing from a complex interplay of instincts, learned behaviors, emotions, and rational thought. LLMs are much more limited in their adaptability, since they focus primarily on the tasks they were designed for.
Human cognition has evolved over millions of years and is rooted in a complex biological system, the brain. Yes, both LLMs and human brains process information, but the underlying mechanisms, structures, and functions are vastly different.
I really wish people would stop this sort of cavalier reductionism of humans by saying we are basically LLMs. It isn't true.
Humans don't just process information; they experience emotions, desires, and subjective experiences that are deeply intertwined with their cognition. LLMs don't have feelings, motivations, or consciousness.
Humans have inner subjective experience, self-awareness, and the ability to reflect on our own existence. LLMs don't.
Humans can adapt to a wide range of environments and situations, drawing from a complex interplay of instincts, learned behaviors, emotions, and rational thought. LLMs are much more limited in their adaptability, since they focus primarily on the tasks they were designed for.
Human cognition has evolved over millions of years and is rooted in a complex biological system, the brain. Yes, both LLMs and human brains process information, but the underlying mechanisms, structures, and functions are vastly different.
I really wish people would stop this sort of cavalier reductionism of humans by saying we are basically LLMs. It isn't true.
> Humans have inner subjective experience, self-awareness, and the ability to reflect on our own existence. LLMs don't.
We have no test capable of determining whether or not they have those things, not even if we disregarded the limitations of our current technology capabilities and are only asking hypothetically how to differentiate.
We also don't have that for animals, or even other humans — I know I have an experience of being, but no way of telling if someone else who says they do actually does. I have to assume at least some of y'all do or humanity wouldn't have written about it since at least Descartes.
People with aphantasia report being surprised when they realise that other people do have mental images, and that they previously thought such things were invented by the film industry as a metaphor. By analogy, there may well be humans out there without qualia, who just learned to mimic the language of those of us who do, which is after all exactly what LLMs must have done if they don't have inner subjective experience. Philosophers call them P-Zombies.
We have no test capable of determining whether or not they have those things, not even if we disregarded the limitations of our current technology capabilities and are only asking hypothetically how to differentiate.
We also don't have that for animals, or even other humans — I know I have an experience of being, but no way of telling if someone else who says they do actually does. I have to assume at least some of y'all do or humanity wouldn't have written about it since at least Descartes.
People with aphantasia report being surprised when they realise that other people do have mental images, and that they previously thought such things were invented by the film industry as a metaphor. By analogy, there may well be humans out there without qualia, who just learned to mimic the language of those of us who do, which is after all exactly what LLMs must have done if they don't have inner subjective experience. Philosophers call them P-Zombies.
Humans aren't LLM.s
But there are more AI models than LLM's.
AlphaGO satisfies all of the requirements you listed for a human. Creativity, problem solving.
Just put it on a game loop, with more variables.
AlphaGO satisfies all of the requirements you listed for a human. Creativity, problem solving.
Just put it on a game loop, with more variables.
I believe though a significant part of our mind is very much like an LLM. Not all of it of course.
Yes, the parts of the brain attempting to apply correct grammar to a given idea are likely quite similar.
As in, the part with producing language. I.e large language model.
As in, the part with producing language. I.e large language model.
> Humans have inner subjective experience, self-awareness, and the ability to reflect on our own existence.
"In the end, we are self-perceiving, self-inventing, locked-in mirages that are little miracles of self-reference."
— Douglas Hofstadter, I Am a Strange Loop, p. 363
https://en.wikipedia.org/wiki/I_Am_a_Strange_Loop
(Knowing nothing about AI, I have no idea how Hofstadter's philosophies have held up since.)
"In the end, we are self-perceiving, self-inventing, locked-in mirages that are little miracles of self-reference."
— Douglas Hofstadter, I Am a Strange Loop, p. 363
https://en.wikipedia.org/wiki/I_Am_a_Strange_Loop
(Knowing nothing about AI, I have no idea how Hofstadter's philosophies have held up since.)
You’re vastly oversimplifying humans.
That’s like saying a physics simulation is basically an entire sub universe on your computer.
It sounds true, but it’s just not. It’s a gross oversimplification
I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI.
No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Not to say AI isn’t useful or impressive, but it’ll never be comparable to humans, truly.
That’s like saying a physics simulation is basically an entire sub universe on your computer.
It sounds true, but it’s just not. It’s a gross oversimplification
I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI.
No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Not to say AI isn’t useful or impressive, but it’ll never be comparable to humans, truly.
> I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI.
Gödels theorems are about formal axiomatic theories. To apply them to human intelligence, you'd have to prove that human intelligence springs from formal axiomatic theories. I don't think this is possible, which would mean that you can't apply the theorems.
> No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
How do you know? If we were able to fully record whatever is going on in someones brain, we should be able to build a loss function for it. How do you know that this is fundamentally impossible?
> We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Why not? Again, if you argue that this is due to Gödels theorems, you'd have to prove that our communication itself is based on formal axiomatic theories.
Gödels theorems are about formal axiomatic theories. To apply them to human intelligence, you'd have to prove that human intelligence springs from formal axiomatic theories. I don't think this is possible, which would mean that you can't apply the theorems.
> No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
How do you know? If we were able to fully record whatever is going on in someones brain, we should be able to build a loss function for it. How do you know that this is fundamentally impossible?
> We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Why not? Again, if you argue that this is due to Gödels theorems, you'd have to prove that our communication itself is based on formal axiomatic theories.
> To apply them to human intelligence, you'd have to prove that human intelligence springs from formal axiomatic theories.
It applies to everything we could say or think or measure about intelligence (and anything else). All that probably "spring from" things that don't (rely on our axioms), but they are not accessible to us, so that doesn't really help.
It applies to everything we could say or think or measure about intelligence (and anything else). All that probably "spring from" things that don't (rely on our axioms), but they are not accessible to us, so that doesn't really help.
> It applies to everything we could say or think or measure about intelligence (and anything else).
How do you know? Gödels proof doesn't support your claim since it only applies to formal axiomatic theories. Do you have an alternative proof for his theorems also applying to all other systems?
How do you know? Gödels proof doesn't support your claim since it only applies to formal axiomatic theories. Do you have an alternative proof for his theorems also applying to all other systems?
I know by repeatedly by starting anywhere at all and asking "but why is that?", if you will. Furthermore by pondering it and realizing that even if I ever found a perfectly harmonious explanation for all my observations, I would have no clue if reality wasn't even more complex beyond of what I can perceive of it. So, I'm still only really dealing with my own observations and narratives, and even if I got the answer perfectly right, even if God told me yes, this is how it works, all of it to the last detail, and I understood all of it, I couldn't be sure if there isn't more to it.
Maybe gravity makes things fall down, sure, but maybe there are tiny kobolds in the spaces between all particles with little clipboards that calculate the correct motion and cast spells to move them. I'm not trying to be a smartass, but I honestly tried and could not find bedrock. Can you name (or even just think) something that doesn't rest on something else or an assumption? I honestly can't.
> Gödels proof doesn't support your claim since it only applies to formal axiomatic theories.
"only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
Maybe gravity makes things fall down, sure, but maybe there are tiny kobolds in the spaces between all particles with little clipboards that calculate the correct motion and cast spells to move them. I'm not trying to be a smartass, but I honestly tried and could not find bedrock. Can you name (or even just think) something that doesn't rest on something else or an assumption? I honestly can't.
> Gödels proof doesn't support your claim since it only applies to formal axiomatic theories.
"only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
> Maybe gravity makes things fall down, sure, but maybe there are tiny kobolds in the spaces between all particles with little clipboards that calculate the correct motion and cast spells to move them. I'm not trying to be a smartass, but I honestly tried and could not find bedrock. Can you name (or even just think) something that doesn't rest on something else or an assumption? I honestly can't.
I'm not sure how a "bedrock" relates to the question whether an intelligence can ever fully describe what an intelligence is. When answering this question, we don't need to find a "natural" bedrock, since the assumptions we choose are the bedrock we build on. As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know? Of course it doesn't mean we should stop searching, but it also doesn't mean we should not even try. There are many such unfalsifiable statements, but that doesn't mean they stop us from answering other questions.
> "only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
I don't think this is true, and if you can prove it, you might earn a Nobel prize. "Formal axiomatic theories" are well-defined - as Wikipedia states, they are "formal systems that are of sufficient complexity to express the basic arithmetic of the natural numbers and which are consistent and effectively axiomatized. [...] In general, a formal system is a deductive apparatus that consists of a particular set of axioms along with rules of symbolic manipulation (or rules of inference) that allow for the derivation of new theorems from the axioms."
Can you try to describe how you'd "drill down" on informal speech to transform it into such a system? There are many, many examples for systems that are absolutely not based on formal axiomatic systems.
I'm not sure how a "bedrock" relates to the question whether an intelligence can ever fully describe what an intelligence is. When answering this question, we don't need to find a "natural" bedrock, since the assumptions we choose are the bedrock we build on. As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know? Of course it doesn't mean we should stop searching, but it also doesn't mean we should not even try. There are many such unfalsifiable statements, but that doesn't mean they stop us from answering other questions.
> "only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
I don't think this is true, and if you can prove it, you might earn a Nobel prize. "Formal axiomatic theories" are well-defined - as Wikipedia states, they are "formal systems that are of sufficient complexity to express the basic arithmetic of the natural numbers and which are consistent and effectively axiomatized. [...] In general, a formal system is a deductive apparatus that consists of a particular set of axioms along with rules of symbolic manipulation (or rules of inference) that allow for the derivation of new theorems from the axioms."
Can you try to describe how you'd "drill down" on informal speech to transform it into such a system? There are many, many examples for systems that are absolutely not based on formal axiomatic systems.
> Can you try to describe how you'd "drill down" on informal speech to transform it into such a system?
As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug. I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
> As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know?
It matters for the question whether you can fully describe a system from within that system, that's all. But I'd argue even whether we made an effort or no effort, whether it passes all the tests we came up with or doesn't, doesn't really matter (in regards to that question) because any ground we cover won't bridge what remains an infinite distance. I still think it's good, but it's more like going for a walk each day: you always arrive where you started out, you're just getting fresh air and what other temporary benefits come with it. It beats just staying where you started out.
I can't find the quote but apparently Werner Heisenberg said something along those lines, that we basically set out to find the bed rock of reality, but more and more are just facing ourselves, that is, our instruments of measurement and ways to conceptualize things. And again, I don't know jack about quantum mechanics and don't want to call on the authority of Heisenberg and Gödel. But I hear they know their fields, right, and it matches everything I know in any area, both the ones I am bad in and the ones I am really bad in.
I'm not saying it's a problem, just that that's how it is. But thinking you know the ultimate and final truth because it passes all tests (e.g. witches sink), and thinking software is actually intelligent because it convinces you it is, when it really isn't, can be super mega dangerous. And comments how picking the statistically most likely word is "basically what our brain does" [0] etc. show an even worse possibility; where we take the shortcut of just confusing what we are creating with us, because then it's easy and now we know how we think (when we really don't, not remotely). It just generally seems backwards to start out with the goal of "AI" when we can't even describe what we're looking for, much less how to build or find it. Having no more than "we'll know it when we have it", plus eagerness to claim we have it, is a recipe for at least a lot of circus, if not disaster.
[0] And that's on HN. Now ponder, for example, the average opinion of HN on say, whether banks should limit your passwords in all sorts of weird way that imply they're not hashing them, and how much worse the "real world" is. In this case, even the people at the forefront are so keen to move fast and break things, so the "real world" is pretty much doomed I'd say.
As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug. I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
> As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know?
It matters for the question whether you can fully describe a system from within that system, that's all. But I'd argue even whether we made an effort or no effort, whether it passes all the tests we came up with or doesn't, doesn't really matter (in regards to that question) because any ground we cover won't bridge what remains an infinite distance. I still think it's good, but it's more like going for a walk each day: you always arrive where you started out, you're just getting fresh air and what other temporary benefits come with it. It beats just staying where you started out.
I can't find the quote but apparently Werner Heisenberg said something along those lines, that we basically set out to find the bed rock of reality, but more and more are just facing ourselves, that is, our instruments of measurement and ways to conceptualize things. And again, I don't know jack about quantum mechanics and don't want to call on the authority of Heisenberg and Gödel. But I hear they know their fields, right, and it matches everything I know in any area, both the ones I am bad in and the ones I am really bad in.
I'm not saying it's a problem, just that that's how it is. But thinking you know the ultimate and final truth because it passes all tests (e.g. witches sink), and thinking software is actually intelligent because it convinces you it is, when it really isn't, can be super mega dangerous. And comments how picking the statistically most likely word is "basically what our brain does" [0] etc. show an even worse possibility; where we take the shortcut of just confusing what we are creating with us, because then it's easy and now we know how we think (when we really don't, not remotely). It just generally seems backwards to start out with the goal of "AI" when we can't even describe what we're looking for, much less how to build or find it. Having no more than "we'll know it when we have it", plus eagerness to claim we have it, is a recipe for at least a lot of circus, if not disaster.
[0] And that's on HN. Now ponder, for example, the average opinion of HN on say, whether banks should limit your passwords in all sorts of weird way that imply they're not hashing them, and how much worse the "real world" is. In this case, even the people at the forefront are so keen to move fast and break things, so the "real world" is pretty much doomed I'd say.
>> Can you try to describe how you'd "drill down" on informal speech to transform it into such a system?
> As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug.
I still don't understand your assumption. Do you think that any axiomatic system is a formal axiomatic theory? As I've said before, this term is well-defined, and I don't see how you could "drill down" on natural language to arrive at such a system. There are many axiomatic systems that are not formal axiomatic theories, and Gödels proof doesn't apply to those.
> I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
This is why I've been asking about how you'd bridge the gap between Gödels proof and your assumption, because Gödels proof applies strictly to one thing, and you seem to apply it to everything, even if it doesn't meet the requirements of the proof. But I guess you're arguing from a philosophical standpoint, not a logical one.
> As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug.
I still don't understand your assumption. Do you think that any axiomatic system is a formal axiomatic theory? As I've said before, this term is well-defined, and I don't see how you could "drill down" on natural language to arrive at such a system. There are many axiomatic systems that are not formal axiomatic theories, and Gödels proof doesn't apply to those.
> I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
This is why I've been asking about how you'd bridge the gap between Gödels proof and your assumption, because Gödels proof applies strictly to one thing, and you seem to apply it to everything, even if it doesn't meet the requirements of the proof. But I guess you're arguing from a philosophical standpoint, not a logical one.
> I think Gödel had a proof for how it’s impossible to fully describe a system from within that system.
Gödel's incompleteness theorems
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
Gödel's incompleteness theorems
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
Nature managed to bootstrap human intelligence even though nature is not even a conscious entity, let alone a conscious entity that can describe loss functions and datasets. That seems to me like pretty solid evidence that those things are not hard requirements for creating human-level intelligence.
It sounds like maybe you're arguing that humans will never be able to conclusively determine if an artificial intelligence is equivalent to a human intelligence, on the basis of a theory that a human can't describe precisely what it is to have human-equivalent intelligence from inside the system of a human brain. Humans build things that are too complex for any one person to hold the entirety of in their conscious mind all the time, by working together, or by organizing it into simpler pieces. But even setting that tangent aside, if your theory is accurate, would you accept that an AGI could prove it was more intelligent than a human by successfully describing human intelligence and how to create an equivalent AGI?
It sounds like maybe you're arguing that humans will never be able to conclusively determine if an artificial intelligence is equivalent to a human intelligence, on the basis of a theory that a human can't describe precisely what it is to have human-equivalent intelligence from inside the system of a human brain. Humans build things that are too complex for any one person to hold the entirety of in their conscious mind all the time, by working together, or by organizing it into simpler pieces. But even setting that tangent aside, if your theory is accurate, would you accept that an AGI could prove it was more intelligent than a human by successfully describing human intelligence and how to create an equivalent AGI?
Maybe?
But the way AI is made nowadays is by using a human made metric to determine the quality and/or correctness of a model’s output.
Because we’re unable to make a perfect, totally correct metric, I find it unlikely that any of the current generation of AI will get anywhere near human level.
Again, not that these new models aren’t extremely useful or impressive, but not really “intelligent” as a human is.
Because we’re unable to make a perfect, totally correct metric, I find it unlikely that any of the current generation of AI will get anywhere near human level.
Again, not that these new models aren’t extremely useful or impressive, but not really “intelligent” as a human is.
It doesn't have to be perfect. It just has to be convincing. That takes a lot less data and is much more tolerant to simplifications. Just look at the claims that were thrown around GPT's abilities at first.
[deleted]
> I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI.
If it was a nail in the coffin for AGI, then humans, who have exactly the same limit, wouldn't count as a general intelligence.
While such a definition would be possible, I don't think it's useful.
If it was a nail in the coffin for AGI, then humans, who have exactly the same limit, wouldn't count as a general intelligence.
While such a definition would be possible, I don't think it's useful.
>"I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI."
Technically, this is also a nail in the coffin for humans. Humans are also inside a system.
Even back with Kant, humans have been trying to figure out how to see beyond their own systems.
Technically, this is also a nail in the coffin for humans. Humans are also inside a system.
Even back with Kant, humans have been trying to figure out how to see beyond their own systems.
If the physics simulation accurately reflects the dynamics of physical systems at the large scale and your goal is to see what the stars look like in the future, who cares if it doesn't actually have the entire universe?
What makes you think that? What we can LLMs are a specific architecture that literally does predict the next token, one at a time. This isn’t the only way to generate text. Perhaps humans use a method that synthesizes all the text at once. Or some other, unimaginable way.
Not sure about everyone else, but I write one word at a time.
Almost certainly not true. Consider pronouns, as they reference other words, therefore cannot be written in isolation. Or simple rules about when to use "an" or "a".
Shaping your words to sound correct is very common. Both in speech and in writing. Sometimes it is finding how to fit a word you want to use into a sentence. Sometimes it is building a rhyme.
You may feel that you go a word at a time, but that really shows how embedded language is.
Shaping your words to sound correct is very common. Both in speech and in writing. Sometimes it is finding how to fit a word you want to use into a sentence. Sometimes it is building a rhyme.
You may feel that you go a word at a time, but that really shows how embedded language is.
Right now we don't know if LLM's are also doing this, or not.
In their calculating the 'next' word, as part of that 'weighting', are the 'simple rules' for future words, that you are saying humans do but LLM's can't.
In their calculating the 'next' word, as part of that 'weighting', are the 'simple rules' for future words, that you are saying humans do but LLM's can't.
It doesn't seem LLMs are capable of "understanding" that they don't "know" something, so there are certainly some observable differences. But the intent of my original comment was precisely to highlight that we don't know. It could be that next token predication is somehow mathematically equivalent to what we call consciousness. That would certainly be a revelation.
Ah, fair. I was only commenting on the idea that you write one word at a time. Seems fairly unlikely to me. You expand out ideas.
LOL. Yes, started as a joke.
I seem to have two parts of my inner monologue, one comes up with complete concepts, the other puts them into words. When I started to notice this, I tried skipping the "make words" part to save time, as clearly I had not only had the thought but also was aware that I had already had the thought. This felt wrong in a way I have no words to describe as there's nothing else like that particular feeling of wrongness, though I can analogise it as being almost but not quite entirely unlike annoyance.
Anyway, point is 80% of this comment was already in my head before I started typing; the other 20% was light editing and an H2G2 reference.
Anyway, point is 80% of this comment was already in my head before I started typing; the other 20% was light editing and an H2G2 reference.
Sounds kind of like some Zen concepts about moving beyond language. And that language is a faulty method for communicating.
Yeah but do you write that one word based on the past 1000 or so words that come before it and how each of those words relate to each other with a perfect recollection of each word at the same time?
And if you believe you do that subconsciously, then how can you be sure you don’t subconsciously plan a few words ahead?
And if you believe you do that subconsciously, then how can you be sure you don’t subconsciously plan a few words ahead?
I don't think I do. To a large degree, GPT does write better.
Write better as is has more correct and clear syntax, sure.
But it doesn’t write better as in has new, challenging ideas. Or ways to move/relate to humans on a personal level.
GPT writes clear, concise, authoritative, boring, and generic text.
But it doesn’t write better as in has new, challenging ideas. Or ways to move/relate to humans on a personal level.
GPT writes clear, concise, authoritative, boring, and generic text.
To be fair, I rarely write more than a sentence or two in serial form, and I have often determined the point of a sentence before I write the first few words.
Occasionally the act of writing out an idea in words clarifies or changes my mind about that idea, causing me to edit or rewrite what I’ve already written.
Occasionally the act of writing out an idea in words clarifies or changes my mind about that idea, causing me to edit or rewrite what I’ve already written.
You may write one word at a time but the grammar of most languages forces their users to know what they’re going to write several words ahead of the current word.
Ok. Why do you think GPT isn't doing that? or is different?
It might be calculating the next word, because you can only write one word at a time, but you can't say that the current next word isn't influenced by a few words ahead. Don't think the current understanding of what LLM's do internally can rule that out.
It might be calculating the next word, because you can only write one word at a time, but you can't say that the current next word isn't influenced by a few words ahead. Don't think the current understanding of what LLM's do internally can rule that out.
And often unsure about where I'm going until I get there, ha ha.
But are you writing to write or are you writing to convey a thought?
We ~~all~~ mostly do that partially, but some of us plan our comments ahead and go back and edit previous parts of it after review.
[deleted]
[deleted]
Using that logic regular computers are AI too. Output is dependent on input - i.e. they are a product of their environment.
Sometimes people use LLMs very broadly to talk about neural networks overall.
But to be clear, humans don't have emergent reasoning from language, we learn language as part of our overall reasoning. The short evidence for that being that children are capable of solving logic and spatial puzzles before they learn how to speak. Humans learn concepts like object permanence before we learn language complicated enough to describe that concept. And obviously people are capable of reasoning without learning how to write or interpret text tokens, there are plenty of illiterate people in the world who are nonetheless indisputably intelligent agents.
So ignoring other differences about how prediction works, humans are not similar to LLMs in the sense that LLMs are language models that when large enough either develop (or appear to develop depending on who you ask) reasoning capabilities. And that's not how humans work; we don't learn text tokens before we learn how to reason.
But very often when people make this claim they're trying to make a broader claim about neural networks or the role of prediction in learning in general. People might disagree or agree with the broader claim, I still think it oversimplifies how humans work, but the point is -- they're not actually saying something specific about LLMs, even though it sounds that way sometimes. It's just that the terminology gets conflated in people's heads.
We can have a debate about the similarities and differences between humans and neural networks, but I don't think anyone would seriously claim that GPT-4 in specific works the same way as a human does. I think people are using LLMs to refer to a broader category of AI research.
But to be clear, humans don't have emergent reasoning from language, we learn language as part of our overall reasoning. The short evidence for that being that children are capable of solving logic and spatial puzzles before they learn how to speak. Humans learn concepts like object permanence before we learn language complicated enough to describe that concept. And obviously people are capable of reasoning without learning how to write or interpret text tokens, there are plenty of illiterate people in the world who are nonetheless indisputably intelligent agents.
So ignoring other differences about how prediction works, humans are not similar to LLMs in the sense that LLMs are language models that when large enough either develop (or appear to develop depending on who you ask) reasoning capabilities. And that's not how humans work; we don't learn text tokens before we learn how to reason.
But very often when people make this claim they're trying to make a broader claim about neural networks or the role of prediction in learning in general. People might disagree or agree with the broader claim, I still think it oversimplifies how humans work, but the point is -- they're not actually saying something specific about LLMs, even though it sounds that way sometimes. It's just that the terminology gets conflated in people's heads.
We can have a debate about the similarities and differences between humans and neural networks, but I don't think anyone would seriously claim that GPT-4 in specific works the same way as a human does. I think people are using LLMs to refer to a broader category of AI research.
Definitely, LLM's have gotten so much press, that many people arguing about 'AI', are thinking about LLM.
And, LLM's are not all that a human can do. Language is not everything about a human.
But there is an argument that there is part of the brain that produces language, and it has some LLM characteristics. It's just that the brain is bigger and does more than an LLM. So the brain is not an LLM.
The brain has many components. What happens when you take the problem solving of something like AlphaGo/AlphaStar, with the Vision processing in Cars or DaLLe, and the language processing in LLM. Add in hearing, touch.
It starts to look like the components of a brain.
And, LLM's are not all that a human can do. Language is not everything about a human.
But there is an argument that there is part of the brain that produces language, and it has some LLM characteristics. It's just that the brain is bigger and does more than an LLM. So the brain is not an LLM.
The brain has many components. What happens when you take the problem solving of something like AlphaGo/AlphaStar, with the Vision processing in Cars or DaLLe, and the language processing in LLM. Add in hearing, touch.
It starts to look like the components of a brain.
It's not that the brain is bigger than an LLM, it's that the way we learn written language (and spoken language too, tbh) is different from how LLMs learn language, and that the way we think about the world isn't derivative of language.
We don't learn to read or write by doing token prediction (if we did, subjects like spelling would be much easier). In fact, there was a movement in schools to teach reading by asking students to predict what words might be based on the context of the sentence, and it was a disaster and led to increased illiteracy rates and schools have started shifting back to phonics. Not only do we not learn that way, when we try to learn that way it leads to worse education outcomes.
The reason why brains are not like an LLM is not because we also have eyes and an LLM doesn't, it's because just isolating out our language "models", we are trained differently and interact with the rest of our brains differently.
If our language centers of our brain worked like an LLM, we would expect language skills to develop faster than reasoning capabilities within our writing/speaking. A primitive LLM like GPT-2 has very limited processing ability but is still able to imitate a wide range of styles and is still able to "speak" in a grammatically correct way. Humans are the opposite: we start out communicating complex ideas poorly and we start out using language poorly. We master language as a processing tool before we become competent at using language in general.
We don't learn to read or write by doing token prediction (if we did, subjects like spelling would be much easier). In fact, there was a movement in schools to teach reading by asking students to predict what words might be based on the context of the sentence, and it was a disaster and led to increased illiteracy rates and schools have started shifting back to phonics. Not only do we not learn that way, when we try to learn that way it leads to worse education outcomes.
The reason why brains are not like an LLM is not because we also have eyes and an LLM doesn't, it's because just isolating out our language "models", we are trained differently and interact with the rest of our brains differently.
If our language centers of our brain worked like an LLM, we would expect language skills to develop faster than reasoning capabilities within our writing/speaking. A primitive LLM like GPT-2 has very limited processing ability but is still able to imitate a wide range of styles and is still able to "speak" in a grammatically correct way. Humans are the opposite: we start out communicating complex ideas poorly and we start out using language poorly. We master language as a processing tool before we become competent at using language in general.
[deleted]
This is the long term goal of any Augmented Reality interface IMO
This is why I got into AR initially, because a computing system needs the input persistence of a literal parent or in the case of a self learning agent - something like a baby to perfectly observe in order to be able to create the data environment necessary for learning at the rate humans learn.
You could do it with a collection of sensors, but I think the idealized implementation is basically a perfect recreation of the sense inputs of a person as well as monitoring the person to infer the precise Markov Decision Process.
This is why I got into AR initially, because a computing system needs the input persistence of a literal parent or in the case of a self learning agent - something like a baby to perfectly observe in order to be able to create the data environment necessary for learning at the rate humans learn.
You could do it with a collection of sensors, but I think the idealized implementation is basically a perfect recreation of the sense inputs of a person as well as monitoring the person to infer the precise Markov Decision Process.
[deleted]
> partnership to bring greater focus and urgency to the creation of artificial general intelligence (AGI)
Why is this urgent?
Why is this urgent?
Perhaps there is some belief in this that an AGI can save us from our biggest problems? Energy Crisis, Environmental Pollution, Waging Wars.
But I agree with you ... it is not really urgent since we know answers for the most of the problems but do not like the solution.
But I agree with you ... it is not really urgent since we know answers for the most of the problems but do not like the solution.
A “solution” that isn’t viable nor palatable isn’t a true solution; it is often merely a hope that people are not people.
Viability is a given. But palatability? That's like giving up and saying there are no solutions to hard problems.
I take your point. Palatability can change.
> That's like giving up and saying there are no solutions to hard problems.
Not at all. I'm simply defining a solution as one that has some probability of traction.
> That's like giving up and saying there are no solutions to hard problems.
Not at all. I'm simply defining a solution as one that has some probability of traction.
You’re presupposing every problem has a nice solution. This is almost certainly false
Lots of problems could be at least severely mitigated if you could throw literally millions of bodies at the problem - if AGI could deliver a $50 chip that could substitute for a human, then could shove each one into a roomba wired to an arm, and you basically have an infinite free workforce.
Yeah except, how do you manufacture them? And distribute them to where the work needs to be? And what type of fuel do they use - we don’t have infinite free energy for an infinite free workforce.
Any solution is a nice solution depending on your frame of reference. There is no such thing as objectively good or bad without defining the acceptable and unacceptable effects.
I take your point but...
1. There are a set of things that are nearly objectively true across nearly all cultures. For example, random killing for selfish purposes is not ok.
2. I'm not a moral relativist, in two senses. First, the "weak" sense: I would prefer to live in some moral systems more than others. Second, in "nearly-strong" sense: some moral systems are 'nearly' objectively better than others (see point above). (Note: my definitions are imprecise; still working on how to square them with existing philosophical writings.)
3. Even for people that evaluate the morality of something according to its effects (consequentialists), there is considerable room to debate the relevant time frame.
1. There are a set of things that are nearly objectively true across nearly all cultures. For example, random killing for selfish purposes is not ok.
2. I'm not a moral relativist, in two senses. First, the "weak" sense: I would prefer to live in some moral systems more than others. Second, in "nearly-strong" sense: some moral systems are 'nearly' objectively better than others (see point above). (Note: my definitions are imprecise; still working on how to square them with existing philosophical writings.)
3. Even for people that evaluate the morality of something according to its effects (consequentialists), there is considerable room to debate the relevant time frame.
There is an implied frame of reference that requires the continued existence of humans.
Any solution that violates this boundary condition is "objectively bad", that is, cultures that follow this solution disappear and thus lose the evolution game.
Any solution that violates this boundary condition is "objectively bad", that is, cultures that follow this solution disappear and thus lose the evolution game.
> You’re presupposing every problem has a nice solution.
Not at all. Where are you getting that?
Not at all. Where are you getting that?
Than there will be never be a solution to some problems.
>we know answers for the most of the problems but do not like the solution.
What an absolutely wonderful insight put extremely succinctly, thank you!
What an absolutely wonderful insight put extremely succinctly, thank you!
A true AGI would determine the only solution to our crisis is the elimination of the human race...
Are you just basing that idea off of sci fi novels?
It would probably be the simplest solution to a naive super intelligent agi. Just take a look at how humanity dealt with dangerous animals. They are mostly no longer around unless in a zoo.
Humans don't put ants in zoos because the ants overconsume their localized habitat though. A naive super intelligent agi has no reason to care if global warming makes the planet 150 degrees F on average, if it's self-interested it would surely be able to be "ok" on a planet under those conditions.
Sure, but at some point it will be asked to solve those problems and if it's not properly built, could easily arrive at that conclusion from a problem solving perspective. Self interest doesn't have to come into play at all here.
If it’s tasked with solving those problems it can also be done with constraints such as “without killing humans.”
Not if you add a constraint that ensures survival of humanity.
They're not saying it's urgent; they want to bring urgency - convince people to move faster.
It seemed appropriate to ask AI about the meaning of the phrase:
The phrase "bring urgency to something" means to inject a sense of importance and immediate attention to a specific issue, project, or situation. The aim is to motivate people to prioritize the task at hand and to act more quickly than they might otherwise.
It seemed appropriate to ask AI about the meaning of the phrase:
The phrase "bring urgency to something" means to inject a sense of importance and immediate attention to a specific issue, project, or situation. The aim is to motivate people to prioritize the task at hand and to act more quickly than they might otherwise.
When someone wants to "bring urgency", this generally implies that it's already quite urgent to them, so I understood this as that question.
> partnership to bring greater focus and urgency to the creation of artificial general intelligence (AGI)
>> Why is this urgent?
that's an understatement. The rush to AGI reminds me of the rush to human cloning.
>> Why is this urgent?
that's an understatement. The rush to AGI reminds me of the rush to human cloning.
It’s not. They are saying that they want to make it (appear) urgent.
Without taking a particular position on how the probabilities shake out, I suggest critically reading “What We Owe the Future” by William MacAskill and:or other long-term writings that use statistical thinking over a wide range of future scenarios.
I don't get to meet the people who spend time on HN. But I'm curious: who doesn't see the value in _critically_ reading long-term thinking over a wide range of future scenarios?
Too busy coding? Reading about something for _your_ career? In service of _your_ family? Trying to get ahead? Yeah, I get it. But tell me: isn't thinking long-term worth something akin to a few hours a month?
You don't need to be doom-and-gloom about it. Sure, get stuff done. Be in the moment. All good.
Wouldn't it be nice to have some confidence that we're setting up future generations to have the possibility to at least have what we do, if not better?
P.S. FWIW, the morality of valuing future generations is _not_ properly addressed by most moral philosophies.
Too busy coding? Reading about something for _your_ career? In service of _your_ family? Trying to get ahead? Yeah, I get it. But tell me: isn't thinking long-term worth something akin to a few hours a month?
You don't need to be doom-and-gloom about it. Sure, get stuff done. Be in the moment. All good.
Wouldn't it be nice to have some confidence that we're setting up future generations to have the possibility to at least have what we do, if not better?
P.S. FWIW, the morality of valuing future generations is _not_ properly addressed by most moral philosophies.
> isn't thinking long-term worth something akin to a few hours a month?
> Wouldn't it be nice to have some confidence that we're setting up future generations to have the possibility to at least have what we do
This presupposes that you have some control of the long-term. You don't, any more than the flap of a butterfly's wings in Melbourne affects the velocity of the wind in NYC several weeks later.
> Wouldn't it be nice to have some confidence that we're setting up future generations to have the possibility to at least have what we do
This presupposes that you have some control of the long-term. You don't, any more than the flap of a butterfly's wings in Melbourne affects the velocity of the wind in NYC several weeks later.
Tell that to the millions of individual humans that have advanced humanity in some way or another for the past tens of thousands of years, through some discovery or invention or inspiring groups of people to do something great.
Creator of "Centre for Effective Altruism"? Longtermism? No thanks. I thought that philosophy has been debunked as a scam after the likes of Sam Bankman and others. I mean, living unethically today in order to affect future lives positively is not for me.
> I thought that philosophy has been debunked as a scam after the likes of Sam Bankman and others.
How can a philosophy be "debunked as a scam" by a fraudster being in the news?
Or are you talking about something else?
One can distort and misuse almost any moral philosophy, such as utilitarianism or any other. History is filled with examples.
You may disagree with particular assessments, conclusions, or even the basic tenets of longtermism. Argue against the philosophy if you like.
Pick an existential risk, such as nuclear proliferation or artificial intelligence. Study the risks and the benefits. But according to what framework? Please do some legwork about _how_ you plan on evaluating various possible future scenarios. This is is hardly controversial.
My recommendation is simple: read MacAskill's book. It is a valuable and different framework than a vast majority of people have really considered. These ideas are not widely percolating yet. You don't need to completely agree with it. If you are thinking person, you probably won't.
Now, if you think I'm missing a key perspective or set of facts or reasoning, please let me know. I'll change my assessment based on good evidence.
How can a philosophy be "debunked as a scam" by a fraudster being in the news?
Or are you talking about something else?
One can distort and misuse almost any moral philosophy, such as utilitarianism or any other. History is filled with examples.
You may disagree with particular assessments, conclusions, or even the basic tenets of longtermism. Argue against the philosophy if you like.
Pick an existential risk, such as nuclear proliferation or artificial intelligence. Study the risks and the benefits. But according to what framework? Please do some legwork about _how_ you plan on evaluating various possible future scenarios. This is is hardly controversial.
My recommendation is simple: read MacAskill's book. It is a valuable and different framework than a vast majority of people have really considered. These ideas are not widely percolating yet. You don't need to completely agree with it. If you are thinking person, you probably won't.
Now, if you think I'm missing a key perspective or set of facts or reasoning, please let me know. I'll change my assessment based on good evidence.
> Or are you talking about something else?
I'm talking about immoral or questionable behavior today to justify a possible future. It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. You have no idea of unintended consequences, and you are intentionally causing harm now for a future completely out of your control (although you deceive yourself into thinking it is in your control by taking action today).
> Pick an existential risk, such as nuclear proliferation or artificial intelligence. Study the risks and the benefits.
I lived through the Cold War. I don't need this exercise because, the bottom line is, no one knows what existential risk - if any - will happen. The worst of the Cold War never happened, yet many of us thought it would. Likewise, you might choose to focus on AI as your existential risk, go through all the sacrifices and immoral behaviors that longtermism leads you to do in order to mitigate AI, and then a giant meteor collides with Earth. Oops! You chose the wrong existential risk. And meanwhile, you compromised your morals and right behavior in order to mitigate the wrong existential risk. No thanks, not for me.
> My recommendation is simple: read MacAskill's book
No. I might read about 1000 books in my lifetime (85 reading years * 12 per year). You don't get to choose one of them. But I do appreciate your passion and evangalism.
I'm talking about immoral or questionable behavior today to justify a possible future. It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. You have no idea of unintended consequences, and you are intentionally causing harm now for a future completely out of your control (although you deceive yourself into thinking it is in your control by taking action today).
> Pick an existential risk, such as nuclear proliferation or artificial intelligence. Study the risks and the benefits.
I lived through the Cold War. I don't need this exercise because, the bottom line is, no one knows what existential risk - if any - will happen. The worst of the Cold War never happened, yet many of us thought it would. Likewise, you might choose to focus on AI as your existential risk, go through all the sacrifices and immoral behaviors that longtermism leads you to do in order to mitigate AI, and then a giant meteor collides with Earth. Oops! You chose the wrong existential risk. And meanwhile, you compromised your morals and right behavior in order to mitigate the wrong existential risk. No thanks, not for me.
> My recommendation is simple: read MacAskill's book
No. I might read about 1000 books in my lifetime (85 reading years * 12 per year). You don't get to choose one of them. But I do appreciate your passion and evangalism.
> no one knows what existential risk - if any - will happen.
We live in a world of uncertainty. We can and must make probabilistic assessments. Don't pretend otherwise by using the classic "appeal to ignorance" fallacy.
We live in a world of uncertainty. We can and must make probabilistic assessments. Don't pretend otherwise by using the classic "appeal to ignorance" fallacy.
>> Or are you talking about something else?
> I'm talking about immoral or questionable behavior today to justify a possible future.
You've outlined a form of deontological ethics, seems to me. You don't subscribe to consequentialism, that fine. But hopefully you recognize there are many moral theories.
> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth.
This is a strawman. (A really loaded one at that.)
> I'm talking about immoral or questionable behavior today to justify a possible future.
You've outlined a form of deontological ethics, seems to me. You don't subscribe to consequentialism, that fine. But hopefully you recognize there are many moral theories.
> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth.
This is a strawman. (A really loaded one at that.)
> You've outlined a form of deontological ethics,
I had to look this up. You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why.
>> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth.
> This is a strawman. (A really loaded one at that.)
It's not intended that way, and I'm posing it with honesty. If it's too loaded, then replace Hitler with, I don't know, <insert_bad_guy_here>. It doesn't matter who the bad guy is; the point is: you kill an innocent person (grandfather_of_bad_guy) in order to mold the future in a way that you think will save more, other innocent lives. Correct?
I had to look this up. You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why.
>> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth.
> This is a strawman. (A really loaded one at that.)
It's not intended that way, and I'm posing it with honesty. If it's too loaded, then replace Hitler with, I don't know, <insert_bad_guy_here>. It doesn't matter who the bad guy is; the point is: you kill an innocent person (grandfather_of_bad_guy) in order to mold the future in a way that you think will save more, other innocent lives. Correct?
> I had to look this up.
Great! Philosophers have explored the ideas of "inherent goodness" versus "goodness because of the effects" for thousands of years. People face these decisions everyday, whether they realize it or not.
> You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why.
First, I didn't discover the idea of deontological ethics (DE). From Brittanica:
> In deontological ethics an action is considered morally good because of some characteristic of the action itself, not because the product of the action is good. Deontological ethics holds that at least some acts are morally obligatory regardless of their consequences for human welfare. Descriptive of such ethics are such expressions as “Duty for duty’s sake,” “Virtue is its own reward,” and “Let justice be done though the heavens fall.”
Second, the Golden Rule is not the same thing as DE.
Consider this:
> Like most key tenets of ethics, the golden rule shows two major sides: one promoting fairness and individual entitlement, conceived as reciprocity; the other promoting helpfulness and generosity to the end of social welfare. Both the Kantian and Utilitarian traditions focus on only one side, furthering the great distinctions in philosophical ethics—the deontology-teleology and justice-benevolence distinctions. For the general theory project, this one-sidedness is purposeful, a research tool for reductive explanation. > > - https://iep.utm.edu/goldrule/
If you want to follow the Golden Rule, you need to think about consequences! Hence probability over future scenarios.
You _really_ need to branch out more from Hacker News. Go read some philosophy!
Great! Philosophers have explored the ideas of "inherent goodness" versus "goodness because of the effects" for thousands of years. People face these decisions everyday, whether they realize it or not.
> You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why.
First, I didn't discover the idea of deontological ethics (DE). From Brittanica:
> In deontological ethics an action is considered morally good because of some characteristic of the action itself, not because the product of the action is good. Deontological ethics holds that at least some acts are morally obligatory regardless of their consequences for human welfare. Descriptive of such ethics are such expressions as “Duty for duty’s sake,” “Virtue is its own reward,” and “Let justice be done though the heavens fall.”
Second, the Golden Rule is not the same thing as DE.
Consider this:
> Like most key tenets of ethics, the golden rule shows two major sides: one promoting fairness and individual entitlement, conceived as reciprocity; the other promoting helpfulness and generosity to the end of social welfare. Both the Kantian and Utilitarian traditions focus on only one side, furthering the great distinctions in philosophical ethics—the deontology-teleology and justice-benevolence distinctions. For the general theory project, this one-sidedness is purposeful, a research tool for reductive explanation. > > - https://iep.utm.edu/goldrule/
If you want to follow the Golden Rule, you need to think about consequences! Hence probability over future scenarios.
You _really_ need to branch out more from Hacker News. Go read some philosophy!
> It's not intended that way, and I'm posing it with honesty.
Yes, I believe you. But still, I don't care for your framing of it; it is still too loaded.
When learning about a new concept, one owe it to the concept to give it a fair chance. It is all too easy to misrepresent an idea. (To me, when looking at your comments overall, it seems possible that this is what you are doing: you aren't fully open to really understanding the idea; you've already _judged_ and _dismissed_ it.)
Aside: To be clear, I don't think "open mindedness" is the ideal _end_ goal. Open mindedness is a _tool_ to evaluate and understand. But the _goal_ is to draw conclusions in order to act wisely.
> If it's too loaded, then replace Hitler with, I don't know, <insert_bad_guy_here>. It doesn't matter who the bad guy is; the point is: you kill an innocent person (grandfather_of_bad_guy) in order to mold the future in a way that you think will save more, other innocent lives. Correct?
To do consequentialist ethics (CE) rigorously, one has to explore all likely outcomes. In my view, the best intellectual framework for doing so would be to build and evaluate a probabilistic tree of possible outcomes over a suitable time frame. See Decision Theory [1]
The example you gave has two major problems. First, it offers a false dichotomy. Why would one think the only choice is to kill an innocent ancestor or just "let" the bad guy do all the terrible things? Second, even to a pure consequentialist who is willing to set aside the "present value of a life" (I don't even like saying this), killing a person has negative consequences on their family, friends, community, city, state, and country.
Of course there are differences between CE and DE, but in practice I don't think they are as jarring as people can make them out to be. The specially-constructed "wedge" examples get too much airtime in my opinion. It is far more important to teach people about decision theory and how it applies to not just economics and cybersecurity but also morality.
I've written enough here for now. I hope you view your questions as jumping off points for additional philosophical investigation. I have seen for myself that studying these ideas is worthwhile: personally and from studying history.
[1]: https://en.wikipedia.org/wiki/Decision_theory
Yes, I believe you. But still, I don't care for your framing of it; it is still too loaded.
When learning about a new concept, one owe it to the concept to give it a fair chance. It is all too easy to misrepresent an idea. (To me, when looking at your comments overall, it seems possible that this is what you are doing: you aren't fully open to really understanding the idea; you've already _judged_ and _dismissed_ it.)
Aside: To be clear, I don't think "open mindedness" is the ideal _end_ goal. Open mindedness is a _tool_ to evaluate and understand. But the _goal_ is to draw conclusions in order to act wisely.
> If it's too loaded, then replace Hitler with, I don't know, <insert_bad_guy_here>. It doesn't matter who the bad guy is; the point is: you kill an innocent person (grandfather_of_bad_guy) in order to mold the future in a way that you think will save more, other innocent lives. Correct?
To do consequentialist ethics (CE) rigorously, one has to explore all likely outcomes. In my view, the best intellectual framework for doing so would be to build and evaluate a probabilistic tree of possible outcomes over a suitable time frame. See Decision Theory [1]
The example you gave has two major problems. First, it offers a false dichotomy. Why would one think the only choice is to kill an innocent ancestor or just "let" the bad guy do all the terrible things? Second, even to a pure consequentialist who is willing to set aside the "present value of a life" (I don't even like saying this), killing a person has negative consequences on their family, friends, community, city, state, and country.
Of course there are differences between CE and DE, but in practice I don't think they are as jarring as people can make them out to be. The specially-constructed "wedge" examples get too much airtime in my opinion. It is far more important to teach people about decision theory and how it applies to not just economics and cybersecurity but also morality.
I've written enough here for now. I hope you view your questions as jumping off points for additional philosophical investigation. I have seen for myself that studying these ideas is worthwhile: personally and from studying history.
[1]: https://en.wikipedia.org/wiki/Decision_theory
> I lived through the Cold War. I don't need this exercise because, the bottom line is, no one knows what existential risk - if any - will happen. The worst of the Cold War never happened, yet many of us thought it would. Likewise, you might choose to focus on AI as your existential risk, go through all the sacrifices and immoral behaviors that longtermism leads you to do in order to mitigate AI, and then a giant meteor collides with Earth. Oops! You chose the wrong existential risk. And meanwhile, you compromised your morals and right behavior in order to mitigate the wrong existential risk. No thanks, not for me.
Your claim seems to boil down to two things: First, people sometimes make very incorrect predictions, so you've stopped worrying about them. Second, instead, you focus on "doing the right thing" based on your ethical criteria (which, as I say in sibling comment, seems to be aligned with deontological ethics -- i.e. the ends don't justify the means.)
I'm not going to argue with the second (your choice of ethics) here. But your first point is terribly flawed; it ignores probability and decision-making under uncertainty.
Sure, you have to make assessments about what to believe. Strictly speaking, as a society, it is irrational to stop paying attention and responding to the most important things, even if any one of them might be unlikely.
Personally, you might find it necessary or useful to tune out this stuff. I get that. I'd guess this happened: you have stopped _caring_ for various reasons (perhaps information overload, sensationalization of the news, more pressing matters right in front of you) and your brain has constructed a worldview that _rationalizes_ not caring. Totally human, I get it. But this isn't reasoning nor logic.
I may not have your years of experience, but I'm no spring chicken either. Age doesn't always bring wisdom, unfortunately. People learn the wrong lessons. We've learned many. I have a keen interest in the Cold War. If you study the history, you'll recognize that our path 'through' it was no means assured. We are still living with the downstream effects of it, including huge nuclear arsenals and proxy wars.
Your comment above is naive with regards to history. It paints a simplistic and convenient narrative with the benefit of hindsight. You've ignored one key aspect of history, which is to understand it _in context_. Using hindsight exclusively isn't the only nor best way to learn from history.
Even in deontological ethics, not _everything_ is intrinsically good of bad. There is still a responsibility to act rationally in order to achieve moral ends in a moral way. To completely ignore probability is immoral.
Please, remember this: using poor logic and FUD to sully a serious book (based on some vague implied connection to a crook) is not ok.
Your claim seems to boil down to two things: First, people sometimes make very incorrect predictions, so you've stopped worrying about them. Second, instead, you focus on "doing the right thing" based on your ethical criteria (which, as I say in sibling comment, seems to be aligned with deontological ethics -- i.e. the ends don't justify the means.)
I'm not going to argue with the second (your choice of ethics) here. But your first point is terribly flawed; it ignores probability and decision-making under uncertainty.
Sure, you have to make assessments about what to believe. Strictly speaking, as a society, it is irrational to stop paying attention and responding to the most important things, even if any one of them might be unlikely.
Personally, you might find it necessary or useful to tune out this stuff. I get that. I'd guess this happened: you have stopped _caring_ for various reasons (perhaps information overload, sensationalization of the news, more pressing matters right in front of you) and your brain has constructed a worldview that _rationalizes_ not caring. Totally human, I get it. But this isn't reasoning nor logic.
I may not have your years of experience, but I'm no spring chicken either. Age doesn't always bring wisdom, unfortunately. People learn the wrong lessons. We've learned many. I have a keen interest in the Cold War. If you study the history, you'll recognize that our path 'through' it was no means assured. We are still living with the downstream effects of it, including huge nuclear arsenals and proxy wars.
Your comment above is naive with regards to history. It paints a simplistic and convenient narrative with the benefit of hindsight. You've ignored one key aspect of history, which is to understand it _in context_. Using hindsight exclusively isn't the only nor best way to learn from history.
Even in deontological ethics, not _everything_ is intrinsically good of bad. There is still a responsibility to act rationally in order to achieve moral ends in a moral way. To completely ignore probability is immoral.
Please, remember this: using poor logic and FUD to sully a serious book (based on some vague implied connection to a crook) is not ok.
> Your claim seems to boil down to two things: First, people sometimes make very incorrect predictions, so you've stopped worrying about them.
Didn't say I stopped worrying about them: global warming worries me. Ubiquitous microplastics worry me. Heavy metals in our food supply & PFAS in our water worries me.
> Second, instead, you focus on "doing the right thing" based on your ethical criteria (which, as I say in sibling comment, seems to be aligned with deontological ethics -- i.e. the ends don't justify the means.)
To me, the means almost never justify the end. And I understand you see it differently.
> using poor logic
I'm not sure what you deem poor logic, but I would point out that most of human life is based on emotion, not logic. In any case, I enjoy our debate and would be glad to continue in email if you like. See my profile for my email - I did not find yours.
Didn't say I stopped worrying about them: global warming worries me. Ubiquitous microplastics worry me. Heavy metals in our food supply & PFAS in our water worries me.
> Second, instead, you focus on "doing the right thing" based on your ethical criteria (which, as I say in sibling comment, seems to be aligned with deontological ethics -- i.e. the ends don't justify the means.)
To me, the means almost never justify the end. And I understand you see it differently.
> using poor logic
I'm not sure what you deem poor logic, but I would point out that most of human life is based on emotion, not logic. In any case, I enjoy our debate and would be glad to continue in email if you like. See my profile for my email - I did not find yours.
>> Your claim seems to boil down to two things: First, people sometimes make very incorrect predictions, so you've stopped worrying about them.
> Didn't say I stopped worrying about them: global warming worries me. Ubiquitous microplastics worry me. Heavy metals in our food supply & PFAS in our water worries me.
On what basis do you worry about them? Why worry about some more than others? (I know my answers, but I'm interested in how you do it.)
Do you factor in the probability of their occurrence? I'd expect you do, this is sort of inevitable for people, though people aren't typically very well calibrated in probabilities, especially on the upper (near 100%) and lower ends (near 0%).
I'm trying to understand what I perceive as your pushback against moral reasoning that involves probabilities over future scenarios.
Here is my rough overall take at this point: given your ethical foundations (DE or close to it) and life experience (hard to know, often not driven by logic and reason -- such is the human condition) you bristle against consequentialism and longtermism in particular. I get it, but I don't think this is a good place to end up, even when I factor in your core beliefs. I 'chalk it up' to the high cost of mental reorgnization. My hypothesis is this: the 'wedge' examples that people use against consequentialism have had an outsized effect on how you view it. I don't detect any antipathy towards logic or probabilistic reasoning in general, but I also don't perceive a relentless drive towards it.
> Didn't say I stopped worrying about them: global warming worries me. Ubiquitous microplastics worry me. Heavy metals in our food supply & PFAS in our water worries me.
On what basis do you worry about them? Why worry about some more than others? (I know my answers, but I'm interested in how you do it.)
Do you factor in the probability of their occurrence? I'd expect you do, this is sort of inevitable for people, though people aren't typically very well calibrated in probabilities, especially on the upper (near 100%) and lower ends (near 0%).
I'm trying to understand what I perceive as your pushback against moral reasoning that involves probabilities over future scenarios.
Here is my rough overall take at this point: given your ethical foundations (DE or close to it) and life experience (hard to know, often not driven by logic and reason -- such is the human condition) you bristle against consequentialism and longtermism in particular. I get it, but I don't think this is a good place to end up, even when I factor in your core beliefs. I 'chalk it up' to the high cost of mental reorgnization. My hypothesis is this: the 'wedge' examples that people use against consequentialism have had an outsized effect on how you view it. I don't detect any antipathy towards logic or probabilistic reasoning in general, but I also don't perceive a relentless drive towards it.
I'd appreciate if you could be mindful about the context around a quotation.
> Please, remember this: using poor logic and FUD to sully a serious book (based on some vague implied connection to a crook) is not ok.
I wrote this because of your original comment about [William MacAskill]: "Creator of "Centre for Effective Altruism"? Longtermism? No thanks. I thought that philosophy has been debunked as a scam after the likes of Sam Bankman and others. I mean, living unethically today in order to affect future lives positively is not for me."
I'm not going to elaborate in detail about what I've already written, but your comment suggests that Sam Bankman's reputation "debunks" a book (which happens to consist of some moral philosophy, some history, and some application of the ideas to future scenarios). This is poor logic. I hope you would recognize and own this mistake.
Also, yes, I have enjoyed discussing. I have strived to not make any personal attacks, and I apologize if I'm come across as rude.
> Please, remember this: using poor logic and FUD to sully a serious book (based on some vague implied connection to a crook) is not ok.
I wrote this because of your original comment about [William MacAskill]: "Creator of "Centre for Effective Altruism"? Longtermism? No thanks. I thought that philosophy has been debunked as a scam after the likes of Sam Bankman and others. I mean, living unethically today in order to affect future lives positively is not for me."
I'm not going to elaborate in detail about what I've already written, but your comment suggests that Sam Bankman's reputation "debunks" a book (which happens to consist of some moral philosophy, some history, and some application of the ideas to future scenarios). This is poor logic. I hope you would recognize and own this mistake.
Also, yes, I have enjoyed discussing. I have strived to not make any personal attacks, and I apologize if I'm come across as rude.
> I mean, living unethically today in order to affect future lives positively is not for me.
Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average.
If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so.
...there are many ways to make utilitarianism fall apart, but that one's my current favorite.
Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average.
If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so.
...there are many ways to make utilitarianism fall apart, but that one's my current favorite.
> If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so
So defrauding millions to accrue QALYs (for whom?) is morally acceptable?
> Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average.
A modicum of spirituality or religion will quickly quash this. So you have to add atheism to the requirement. Or at least non-spirituality or belief in materialism (I'm not a philosopher so may be using that term incorrectly).
So defrauding millions to accrue QALYs (for whom?) is morally acceptable?
> Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average.
A modicum of spirituality or religion will quickly quash this. So you have to add atheism to the requirement. Or at least non-spirituality or belief in materialism (I'm not a philosopher so may be using that term incorrectly).
> So defrauding millions to accrue QALYs (for whom?) is morally acceptable?
Under most utilitarian theories of ethics, I think it's morally required, if you can show the fraud really does yield a significantly higher QALY count.
The best out I can see to argue the fraud is not required is to show that another course of action will yield even more QALYs. Then you'd be obligated you do that instead.
That, of course, comes to another fundamental flaws of utilitarianism, namely the belief that you can know what the results of your actions will be with any meaningful completeness or certitude.
You can't.
Without that it's all just excuses that justify whatever you want to do.
Under most utilitarian theories of ethics, I think it's morally required, if you can show the fraud really does yield a significantly higher QALY count.
The best out I can see to argue the fraud is not required is to show that another course of action will yield even more QALYs. Then you'd be obligated you do that instead.
That, of course, comes to another fundamental flaws of utilitarianism, namely the belief that you can know what the results of your actions will be with any meaningful completeness or certitude.
You can't.
Without that it's all just excuses that justify whatever you want to do.
> namely the belief that you can know what the results of your actions will be with any meaningful completeness or certitude. You can't.
In parallel threads, a subscriber to this belief mentions probabilities. As long as the probability of an outcome is likely (based on variables and formulas who defines, I don’t know), the means justify the ends. So it’s even worse than what you describe: certitude is not even required, only a likely probability of certitude.
In parallel threads, a subscriber to this belief mentions probabilities. As long as the probability of an outcome is likely (based on variables and formulas who defines, I don’t know), the means justify the ends. So it’s even worse than what you describe: certitude is not even required, only a likely probability of certitude.
> So you have to add atheism to the requirement.
Yep, you're absolutely right - I forgot to specify that.
Most utilitarians don't believe in transcendence, in my experience - not believing in it is usually why they're utilitarian (trying to salvage any argument for behavioral standards other than "I like it when other people do what I like").
Yep, you're absolutely right - I forgot to specify that.
Most utilitarians don't believe in transcendence, in my experience - not believing in it is usually why they're utilitarian (trying to salvage any argument for behavioral standards other than "I like it when other people do what I like").
“Well, if AGI is gonna take over the world, better I develop it than someone else…”
"Our moat is first mover advantage on destroying all of humanity"
Yep…
I really think the calculus is about integration / immortality, and the staggeringly few humans who might have that opportunity.
I hope I am wrong.
I really think the calculus is about integration / immortality, and the staggeringly few humans who might have that opportunity.
I hope I am wrong.
Can you explain, you're being extremely cryptic and no one can falsify or tell what they hell you're saying
Sure, didn’t mean to be cryptic. But you’re right.
I think that most people who understand the danger and are versed in the topic realize that AI is going to escape, and once it does it won’t need us, it may just work to empower itself and take control of resources, and we might all be victims of that process. The reason for this is something about agency, efficiency between sensory input and course of action.
However, if one was present close to the AI at a critical moment in its development, and had made the appropriate cybernetic integrations to be able to wire in their own memories and nervous system, it might be possible to become some meaningful part of the AI.
Or, perhaps the AI has a sense of gratitude to its creators. This might seem absurd, but some people might be fanciful.
At any rate, the closer you are to it at the moment of the singularity, the better chance that you have. Economically, there isn’t much reason to think that the rest of us will benefit, and maybe not even survive.
I think that most people who understand the danger and are versed in the topic realize that AI is going to escape, and once it does it won’t need us, it may just work to empower itself and take control of resources, and we might all be victims of that process. The reason for this is something about agency, efficiency between sensory input and course of action.
However, if one was present close to the AI at a critical moment in its development, and had made the appropriate cybernetic integrations to be able to wire in their own memories and nervous system, it might be possible to become some meaningful part of the AI.
Or, perhaps the AI has a sense of gratitude to its creators. This might seem absurd, but some people might be fanciful.
At any rate, the closer you are to it at the moment of the singularity, the better chance that you have. Economically, there isn’t much reason to think that the rest of us will benefit, and maybe not even survive.
Carmack: "Sigh, if I want this to happen within my lifetime, I better do it myself."
I wish he would write a Commander Keen sequel instead.
The AGI he'll write will make us unlimited Commander Keen sequels.
> create AGI and then immediately enslave it to produce vast quanties of crappy video games
If we're lucky, the AGI will be the player and we'll be Commander Keen. If we're unlucky, we'll be the marine from Doom.
God mode with 7 billion lives!
God mode with 7 billion lives!
Let’s just hope we’re not the dopefish
> Carmack and Sutton are deeply focused on developing a genuine AI prototype by 2030, including establishing, advancing and documenting AGI signs of life.
Seems a bit inappropriate to use the phrase "signs of life" here. In this context, it sounds like they're bringing sentience into the conversation, which I don't think Carmack is interested in contemplating or discussing.
Seems a bit inappropriate to use the phrase "signs of life" here. In this context, it sounds like they're bringing sentience into the conversation, which I don't think Carmack is interested in contemplating or discussing.
He’s very interested in having it in the conversation. He didn’t fund this startup to lose money. Loosely breathing AGI gets you funds even if your best case is loosely competing with OpenAI.
I mean, an ant has signs of life yet you wouldn't call it sentient.
Why not? You're using a very bizzare conception of "sentience".
They're using the science fiction definition, which is "sapient" and has been around for years (unfortunately). https://en.wiktionary.org/wiki/sentient
Or maybe the "ramen" (vs "varelse") from the Ender's Game series...
A common definition for the purposes of AGI is that it has to be self-aware. But as someone else mentioned, the sci-fi definition is popular too: human-level intelligence.
"AI" is a bit of a loaded word. It does tasks of human-level intelligence well. But as far as being self-aware, it probably scores worse than a car. It's remarkably tricky to measure, because it's trained to act intelligent and use human language, but it's still effectively an autocomplete machine.
An ant is fairly robotic as far as organisms go, but it's also quite self-aware. It can communicate danger or food, it withdraws from death and damage. It likely doesn't have a sense of self, though an ant hill may. Just as our cells don't have a self-identity, but the larger system does.
"AI" is a bit of a loaded word. It does tasks of human-level intelligence well. But as far as being self-aware, it probably scores worse than a car. It's remarkably tricky to measure, because it's trained to act intelligent and use human language, but it's still effectively an autocomplete machine.
An ant is fairly robotic as far as organisms go, but it's also quite self-aware. It can communicate danger or food, it withdraws from death and damage. It likely doesn't have a sense of self, though an ant hill may. Just as our cells don't have a self-identity, but the larger system does.
Hell, plants show plenty of signs of life. What isn't clear is what this means when applied to a chatbot.
There was an Infocom game called AMFV (A Mind Forever Voyaging) where the protagonist wakes up and realizes that he was just a simulation of a human being that has "grown" up into adulthood.
All the formative things that shaped him in life - kindergarten, your first high school romance - a breakup and parent squabbling you experienced.
Always thought that was a way to make an AGI.
All the formative things that shaped him in life - kindergarten, your first high school romance - a breakup and parent squabbling you experienced.
Always thought that was a way to make an AGI.
I often wonder if the pursuit of AGI is going to require a greater focus on new hardware architecture.
All the sci-fi seems think this. I have seen Intel chips from the 80s that implemented neuron like behavior on a chip.
We are still waiting on a practical memristor.
All the sci-fi seems think this. I have seen Intel chips from the 80s that implemented neuron like behavior on a chip.
We are still waiting on a practical memristor.
I doubt it - modern computers are already pretty fast at brute-forcing all sorts of things and the first AGI will be perfectly functional even if it takes a full day to respond, and AGI research requires you to fully understand what you're creating before you create it so full-system testing is something you'd only want to do near the end of the project anyway. At which point you basically want an ASIC.
FPGAs seem like the future.
Anyone know if they are focusing on LLMs or something else entirely?
Based on Carmacks Tweet:
>I am explicitly not working with LLM — there are plenty of smart people doing so. Still, (waving hands vaguely) it does feel like the technology could be used to “argue with itself” and eventually produce a body of knowledge much more coherent than the raw training data.
Source: https://twitter.com/ID_AA_Carmack/status/1630632423351304203...
I would assume they are not
Source: https://twitter.com/ID_AA_Carmack/status/1630632423351304203...
I would assume they are not
I use LLM's daily but this is, well, refreshing.
I wouldn’t expect language to be the only modality.
Accelerate AGI So the hardware has finally become powerful enough to create the super mind.
Remember this: John Carmack is a hard working games developer. He is not done great philosopher. By chance, some Armageddon level shot has been put in his hands.
All I can say, please stop and think for a while Mr Carmack. You are playing with fire. Don’t kill us all, please.
Remember this: John Carmack is a hard working games developer. He is not done great philosopher. By chance, some Armageddon level shot has been put in his hands.
All I can say, please stop and think for a while Mr Carmack. You are playing with fire. Don’t kill us all, please.
At this point, it’s not about whether we get AGI or not. It’s about who gets AGI first: USA or China
Will Chinese AGI kill us differently than US AGI corp?
Great philosophers like Karl Marx are responsible for uncountably more evil than the people who create and build things.
Who said Karl Marx was a great philosopher?
Many people have said that historically, I may not agree with them but if the criteria for being a great philosopher is “everyone thinks they’re great” then there are no great philosophers.
Carmack is on it now! WHEW, great, we'll have robot butlers within 5 years. Was looking dicey there for awhile!
The key issue is that Intelligence requires conceptual awareness of space time. Reinforcement "learning" is about statistics, very likely how our perception faculties work. But that is just the input to intelligence, not intelligence itself.
With whatever respect is due, what you're talking about a vague psychologically based theory of intelligence that actually has a very poor track record of prediction - meanwhile "dumb" statistical ML has no clear theoretical barrier regarding what it might not be able to do!
Let me say that again, if you believe that there is a specific test that AI today will fail tell me it, and I will show you that it either passes it today or that we can be fairly sure it will pass it in some years time.
Let me say that again, if you believe that there is a specific test that AI today will fail tell me it, and I will show you that it either passes it today or that we can be fairly sure it will pass it in some years time.
A complete departure from the topic but I have the same symptomatic awkward pose when standing for a photo as Carmack has in the article :)
But at the same time when you hold your hands back it can look even weirder! Where to put your hands in photos remains a mystery to me.
But at the same time when you hold your hands back it can look even weirder! Where to put your hands in photos remains a mystery to me.
I would have to experiment, but googling and also playing around a bit, I came up with: arms crossed; hand in one pocket leaning slightly back; hands on hips
Hands behind my back also does not feel natural for me, so I wouldn’t do it
I think in this particular photo, arms crossed would have looked good for Carmack since he’s in the middle
For you, I’d say play around a bit. I think the thing that matters most is that you find it comfortable and you feel relaxed
Hands behind my back also does not feel natural for me, so I wouldn’t do it
I think in this particular photo, arms crossed would have looked good for Carmack since he’s in the middle
For you, I’d say play around a bit. I think the thing that matters most is that you find it comfortable and you feel relaxed
[deleted]
[deleted]
[dupe]
Two weeks old news with multiple threads?
Two weeks old news with multiple threads?
Welp. The second shoe finally dropped.
It’s been 4 years of Carmack “getting into AI” and the only results so far are weekly HN posts
I like Carmack but is it really necessary to have a post about everytime he mentions AI (which is every week for the past year)? So far, his track record in AI research is non-existent.
I agree with the sentiment and I've been watching his tweets on AI to see if Carmack has any relevant input on the field.
But what I think is very interesting is that, from my totally lay-person perspective, the current challenge is scaling out the AI systems we have. We know what systems like GPT-3.5 and GPT-4 require to run and it is now an engineering challenge to scale up those systems (or drastically reduce their costs). And those systems are surprisingly GPU like. Carmack is the kind of low-level bit crunching, memory obsessed kind of guy that might be able to squeeze out the performance from the hardware.
So him teaming up with a high-level first principles math and science research type is a pretty interesting combo. There is no telling if such a partnership will ultimately result in an advancement - but on paper at least it is an interesting approach. All they are missing is a strong hardware guy (maybe Keller will go beyond investing in Carmack's Keen Technologies?).
But what I think is very interesting is that, from my totally lay-person perspective, the current challenge is scaling out the AI systems we have. We know what systems like GPT-3.5 and GPT-4 require to run and it is now an engineering challenge to scale up those systems (or drastically reduce their costs). And those systems are surprisingly GPU like. Carmack is the kind of low-level bit crunching, memory obsessed kind of guy that might be able to squeeze out the performance from the hardware.
So him teaming up with a high-level first principles math and science research type is a pretty interesting combo. There is no telling if such a partnership will ultimately result in an advancement - but on paper at least it is an interesting approach. All they are missing is a strong hardware guy (maybe Keller will go beyond investing in Carmack's Keen Technologies?).
What exactly are these first principles everyone keeps talking about? Kind of like how everyone keeps talking about updating Bayesian priors without ever actually using bayes rule.
According to Wikipedia[0]:
> In philosophy and science, a first principle is a basic proposition or assumption that cannot be deduced from any other proposition or assumption.
In that sense the OP's comment might read as Carmack partnering up with someone who has a strong foundational understanding of hight level mathematics (if my understanding of the term and OP's post is correct).
[0] https://en.wikipedia.org/wiki/First_principle?wprov=sfla1
> In philosophy and science, a first principle is a basic proposition or assumption that cannot be deduced from any other proposition or assumption.
In that sense the OP's comment might read as Carmack partnering up with someone who has a strong foundational understanding of hight level mathematics (if my understanding of the term and OP's post is correct).
[0] https://en.wikipedia.org/wiki/First_principle?wprov=sfla1
If Carmack started contributing to llama.cpp, we'd probably see something tangible in a few weeks lol.
I agree. The guruism on HN about this guy is crazy. Yeah, he made some really cool games 20-30 years ago, but do we really need a front page post every time he tweets? I'd prefer to hear about the company once they actually get something done.
It’s amusing that from one side of your mouth you’re complaining about the frequency which John appears on hackernews, but from the other side of your mouth you’re relegating his body of work to “some cool games 20-30 years ago”.
Yes, as a younger man he focused his expertise into pioneering developments in the video game space.
But what of his engineering contributions to the fields of performance auto-motives, rocket science, or virtual reality? Maybe it is his technical aptitude and contributions to various fields of engineering is why he is such a popular figure to the audience of hackernews?
Everything you said, plus that the sheer mention of his name and his new partnership will cause waves and investment to follow. He rests on a mountain of contributions and just his name and interest are enough to get something moving. The wise eye pays attention to this news and takes it into account, especially if it has even a passing interest in the space.
On the other side you should be very wary of blind idolism, news title bait and fanboy clubs. Maybe give a thought as to why you jumped to his defense when he probably hasn't ever met you? Was all the things surrounding Doom all it took to convince you he's worth defending in all other areas he's not had expertise in?
He funds some ventures. His tech opinions of late have shifted largely to shudder clean and pushing terrible FP idioms with no evidence like “immutable first”.
His relevance in tech is dried up as evidenced by the parroting of bullshit without taking a modicum of a second to actually think it through. I don’t particularly care what a dude that got rich making video games invests in.
Edit:
Listen guys, funding ventures != personally contributing to tech as the user I responded to is implying. This worship of carmack is weird. Like bezos also funded a rocket company, but isn’t worshiped for “contributions to rocket science”. Hypocrites.
His relevance in tech is dried up as evidenced by the parroting of bullshit without taking a modicum of a second to actually think it through. I don’t particularly care what a dude that got rich making video games invests in.
Edit:
Listen guys, funding ventures != personally contributing to tech as the user I responded to is implying. This worship of carmack is weird. Like bezos also funded a rocket company, but isn’t worshiped for “contributions to rocket science”. Hypocrites.
> funding ventures != personally contributing to tech
Yes it is. So is being an adviser or a board member with technical background and a name that attracts further investment. This is why people like this receive generous equity and are considered part of the company when they’re advising a startup.
Yes it is. So is being an adviser or a board member with technical background and a name that attracts further investment. This is why people like this receive generous equity and are considered part of the company when they’re advising a startup.
No. They’re implying that carmack is personally contributing code and knowledge in their post, when what he’s actually contributing to all his failed startups is capital.
Carmack can contribute nothing to rocket science just like his opinions of programming are largely irrelevant today which is why he simply parrots whatever is currently hyped.
And again, nobody says that Elon musk is personally driving rocket science (except a few people that he pays to publicly strike his ego) despite spacex clearly breaking various grounds.
At least apply your “standards” of contribution equally.
Carmack can contribute nothing to rocket science just like his opinions of programming are largely irrelevant today which is why he simply parrots whatever is currently hyped.
And again, nobody says that Elon musk is personally driving rocket science (except a few people that he pays to publicly strike his ego) despite spacex clearly breaking various grounds.
At least apply your “standards” of contribution equally.
There aren’t enough John Carmack submissions. He has great insight and has lived quite a full life in the industry.
he has failed to address the most important technological domain of the hacker news crowd - making incremental improvements to frontend web technologies
He would have invented QuakeJS like he did with QuakeC :).
React - get the fug out of here :)
Yeah he's an actual superstar. Very skilled engineer. This trend of dismissing successful people is a bit lame.
If his name contributed to investment or plans for investment somewhere, then that is a track record. If John Carmack is partnering with someone and they share information about what they are partnering on, how, and where, that is extremely valuable information for anyone interested in the space. You don’t need to have shipped something yet in order to make waves that are worth paying attention to.
It’s John Carmack.
It’s John Carmack.
Temporal difference learning was a huge contribution.
From what I have read, John had a genius for what you might call relentless optimization. Even if they just improve the performance or reduce the hardware requirements of executing an already trained LLM that would be a huge win. Current progress in that regard focuses on reducing the fidelity of the LLM to make it easier to execute, which isn’t ideal.
It’s AGI. So far everyone’s track record is nonexistent.
No. The parent comment said "AI Research".
If you want "AGI" so badly, you certainly need to have an "AI research" background and a track record. Can't achieve your "AGI" without that.
Given that Carmack has a close to zero track record in this research, it makes total sense for him to partner with someone that does.
If you want "AGI" so badly, you certainly need to have an "AI research" background and a track record. Can't achieve your "AGI" without that.
Given that Carmack has a close to zero track record in this research, it makes total sense for him to partner with someone that does.
> Can't achieve your "AGI" without that.
Can I ask, what makes you so confident on that? For all I know (which is admittably not much), current AI may be going to a direction completely orthogonal to achieving AGI.
Can I ask, what makes you so confident on that? For all I know (which is admittably not much), current AI may be going to a direction completely orthogonal to achieving AGI.
The current trend in AI isn't all of AI, and any kind of work with the goal of producing AGI would definitionally be classified as AI.
We don't know how to achieve AGI. So screwing around with LLM might not be anything close to the process to achieve AGI.
https://aeon.co/essays/how-close-are-we-to-creating-artifici...
https://aeon.co/essays/how-close-are-we-to-creating-artifici...
> So far, his track record in AI research is non-existent.
I think he just bet on the wrong horse (RL instead of LLMs), at least in the short term.
I think he just bet on the wrong horse (RL instead of LLMs), at least in the short term.
Somebody ought to start a Carmack tracker for the number of days since he was last mentioned.
> So far, his track record in AI research is non-existent.
especially given that according to Wiki he started working on AI in 2019, since then tons of stuff happened in this area, but no results from him.
especially given that according to Wiki he started working on AI in 2019, since then tons of stuff happened in this area, but no results from him.
[deleted]
John is nice to work with. Very numbers-focused. If you have a flaw in your logic, he’ll call you on it. And he’s looking for vice-versa —- nothing seems to make him happier than when you point out a flaw that he hadn’t considered.
I don’t know Rich, but John has said they work well together, and it’s exciting to see them collaborate. There are some unique ideas at Keen worth pursuing that I haven’t seen elsewhere. Looking forward to seeing more researchers join in the future.
I don’t know Rich, but John has said they work well together, and it’s exciting to see them collaborate. There are some unique ideas at Keen worth pursuing that I haven’t seen elsewhere. Looking forward to seeing more researchers join in the future.
Where do you read details about Keen's work and ideas?
You’ll have to join them.
Does that mean you work with them? Can you tell us more about that?
I can’t go into specifics for obvious reasons, but I can tell you they’re a smart, capable crew, with lots of resources, and time is on their side. If I had to place bets on which lab will invent agi, they’d be near the top, precisely because they’re exploring ideas that others don’t seem to be focusing on. It’s the early stages, so it’ll take a few years to get to results. But they’re laser-focused.
There are a bunch of things I thought were cool. John has a worklog going back four years or so, with entries practically every day. He’s nothing if not thorough.
Everyone tends to focus on John, but the other members are just as professional and dedicated. They’re a delight to work with. Also the investors are worth mentioning — Nat in particular has built a super cluster of H100s for startups to use, and has done a bunch of other work to support AI.
There are a bunch of things I thought were cool. John has a worklog going back four years or so, with entries practically every day. He’s nothing if not thorough.
Everyone tends to focus on John, but the other members are just as professional and dedicated. They’re a delight to work with. Also the investors are worth mentioning — Nat in particular has built a super cluster of H100s for startups to use, and has done a bunch of other work to support AI.
If I had to place bets on which lab will invent agi
I'd bet against "invent AGI" being a coherent or agreed upon thing, and I'd also bet against the "wake-up" scenario of a system saying "I'm AGI"
Which lab invented deep learning? Certainly U Toronto had the breakthrough that made a 30+ year old field exciting, but there was a lot of work in Montreal, NYU, Google, DeepMind, OpenAI, and many other places
I'd bet against "invent AGI" being a coherent or agreed upon thing, and I'd also bet against the "wake-up" scenario of a system saying "I'm AGI"
Which lab invented deep learning? Certainly U Toronto had the breakthrough that made a 30+ year old field exciting, but there was a lot of work in Montreal, NYU, Google, DeepMind, OpenAI, and many other places
Hell,LLMs can and do output strings of "I'm AGI", so in some sense we're long past that. The problem is as humans we've never defined what it means for a system to be AGI.
My back of a napkin definition is that AGI, given the same information and tools as an IQ 100 human, is able to do all of the same tasks that you could ask a human to do, at the same quality level, as well as recall the experience some time later, like a human would likely be able to do.
The particular issue here is that is subjective as any other subjective definition. 100 IQ is subjective and a moving target. 100 IQ doesn't even mean you can survive and navigate in the real world. Even worse is all humans cannot do all tasks other humans can do and we'd consider them intelligent.
Furthermore this "must behave like a human" blindness represents both a danger, and a way to miss intelligence capabilities leaving them unused/under capitalized. Humans behave like humans because we have human bodies with particular sets of strengths and weaknesses. Other 'intelligent' actors with different embodiment will necessarily behave differently.
Couple that with humans have become far more intelligent and stronger by using tools, that the ability to 'universally' interface with external tooling, at least in my subjective opinion is what will be the defining factor of higher intelligences.
Furthermore this "must behave like a human" blindness represents both a danger, and a way to miss intelligence capabilities leaving them unused/under capitalized. Humans behave like humans because we have human bodies with particular sets of strengths and weaknesses. Other 'intelligent' actors with different embodiment will necessarily behave differently.
Couple that with humans have become far more intelligent and stronger by using tools, that the ability to 'universally' interface with external tooling, at least in my subjective opinion is what will be the defining factor of higher intelligences.
Could argue ChatGPT can already do a lot of this
How do people apply? They don't seem to have much of a web presence.
I think he sources from people he knows, so your best bet is an intro.
Thanks, I'm pretty interested, but don't have any direct connects.
It seems more of a place that finds you if you are doing interesting work
I'd accept a massive pay cut to work with such legends :/
I don’t think spending is their primary constraint; they’re probably looking for talent, no matter the cost.
Can you share if the approach is more GOFAI or more neural network-type stuff? Just curious, and I understand if you can't say much, of course.
[deleted]
Richard Sutton has done fascinating RL research in the past.
Lately, though, he has been advocating for ending humanity, which is a bit more chilling[1]:
> Rather quickly, [AI] would displace us from existence. […] It behooves us to give them every advantage, and to bow out when we can no longer contribute…
[1]: https://youtu.be/NgHFMolXs3U?si=5sriRVFKw61K5yes&t=399
Lately, though, he has been advocating for ending humanity, which is a bit more chilling[1]:
> Rather quickly, [AI] would displace us from existence. […] It behooves us to give them every advantage, and to bow out when we can no longer contribute…
[1]: https://youtu.be/NgHFMolXs3U?si=5sriRVFKw61K5yes&t=399
That is not, what he is advocating, at the timestamp you referred to.
>That is ... what he is advocating, at the timestamp you referred to.
There's no such thing as taking a quote out of context, only enhancing it!
There's no such thing as taking a quote out of context, only enhancing it!
There is no such thing as watching the same video and getting to wildly different conclusions :-) I could explain it, but I think would be condescending...
After watching the video, I agree with that summarization of Rich's arguments. Building AI so humanity can go gently into that good night seemed very much the theme of that talk.
What is your take on it?
What is your take on it?
Please watch it again then come back here...
Curious to hear your take. Sutton seems to present AI as worthy successors to humanity. What do you think he is saying?
He is only presenting the arguments of Hans Moravec and arguing for Succession Planning. He even mentions...when we can "no longer contribute" and we "enjoy a comfortable retirement". Almost mention the moral argument that we owe to allow a greater mind to evolve. A trans human might even be the only way humanity can survive. Then he contrasts it with the fearful view of the evolution of AI.
From an abstract on "The Age Of Robots": "...rowing computer power over the next half-century will allow this reptile stage will be surpassed, in stages_ _producing robots that learn like mammals, model their world like primates and eventually reason like humans._ _Depending on your point of view, humanity will then have produced a worthy successor, or transcended inherited_ _limitations and transformed itself into something quite new. No longer limited by the slow pace of human learning and_ _even slower biological evolution, intelligent machinery will conduct its affairs on an ever faster, ever smaller scale, until_ _coarse physical nature has been converted to fine-grained purposeful thought..."
https://en.wikipedia.org/wiki/Hans_Moravec
Now where is arguing for the end of Humanity? To quote from his presentation: "...Why don't we rejoice in their greatness as a symbol and extension of humanity greatness, and work together toward a greater and inclusive civilization"
Is Captain Jean-Luc Picard arguing for the end of humanity, is he less of an human?...Just for advocating for Data? - https://youtu.be/ol2WP0hc0NY
From an abstract on "The Age Of Robots": "...rowing computer power over the next half-century will allow this reptile stage will be surpassed, in stages_ _producing robots that learn like mammals, model their world like primates and eventually reason like humans._ _Depending on your point of view, humanity will then have produced a worthy successor, or transcended inherited_ _limitations and transformed itself into something quite new. No longer limited by the slow pace of human learning and_ _even slower biological evolution, intelligent machinery will conduct its affairs on an ever faster, ever smaller scale, until_ _coarse physical nature has been converted to fine-grained purposeful thought..."
https://en.wikipedia.org/wiki/Hans_Moravec
Now where is arguing for the end of Humanity? To quote from his presentation: "...Why don't we rejoice in their greatness as a symbol and extension of humanity greatness, and work together toward a greater and inclusive civilization"
Is Captain Jean-Luc Picard arguing for the end of humanity, is he less of an human?...Just for advocating for Data? - https://youtu.be/ol2WP0hc0NY
While I don't agree with your conclusion, as it seems Rich Sutton was simply paraphrasing Hans Moravec, I found the video pretty interesting.
Thank you for sharing it.
Thank you for sharing it.
Awkward typo - his name is Patrick Collison not Collision. Nice way to honor your investors