The Myth of a Superhuman AI(wired.com)
wired.com
The Myth of a Superhuman AI
https://www.wired.com/2017/04/the-myth-of-a-superhuman-ai/
91 comments
You're making an argument that is already in TFA. It's in addition -and only partially related to- the main point which is: The arguments that are commonly used to demonstrate the possibility or threat of "ASI" are made from claims presented _without evidence_.
This will invalidate fantastical accelerationist woo until such evidence is presented, and -in the current state of the art research within relevant fields- it's not even certain we can know what would constitute such evidence _in principle_.
This will invalidate fantastical accelerationist woo until such evidence is presented, and -in the current state of the art research within relevant fields- it's not even certain we can know what would constitute such evidence _in principle_.
I assume you're referring to this part:
If the expectation of a superhuman AI takeover is built on five key assumptions that have no basis in evidence, then this idea is more akin to a religious belief — a myth
My problem is, even this is wrong. I do not find the "five key assumptions" he cites to be the foundation of beliefs that a dangerous "Superhuman AI" could emerge. So to me, this article simply fails right from the jump. The thing he is arguing for and/or against just don't, collectively, constitute a well-developed argument for the overall thesis. Not in my opinion anyway.
To go through them one by one:
Artificial intelligence is already getting smarter than us, at an exponential rate.
I don't know anybody making this claim, nor is it apparent that belief in this is necessary to think that dangerous "Superhuman AI" could be a problem in the future. I guess it comes down to what you think Kelly meant by "soon" in the preceding sentence.
We’ll make AIs into a general purpose intelligence, like our own.
It's the "like our own" part of this that I have a problem with. Yes, I think many people believe that we will create some manner of "general intelligence", but I also believe that a lot of researchers acknowledge that it may not be 100% "human like". But this may be my own bias showing, as I personally harp a lot on the importance of the distinction between "human level" intelligence and "human like" intelligence.
We can make human intelligence in silicon.
This is probably the most sound one of all, but here's the thing... saying "there is no evidence for this" is simply wrong. To hold to this is to completely ignore all progress made building AI in silicon to date. And while we don't yet have "general intelligence", I don't see how one can just ignore incremental progress completely and say "there's nothing there". I would argue that the null hypothesis should probably be that general intelligence on silicon is possible that that the burden of proof is on the people arguing the opposite.
Also, not everybody thinking about / working on AI is focused on the "in silicon" approach. The majority? Yes. But I'm sure there are a handful of people working on ideas related to DNA computing, chemical computing, growing synthetic brain matter, etc.
Intelligence can be expanded without limit.
To me, this may or not not be true, but it's completely irrelevant to the thesis at hand. The question isn't "can machine intelligence be unbounded?", but rather "can machine intelligence exceed human intelligence?"
Once we have exploding superintelligence it can solve most of our problems.
Again, this strikes me as mostly irrelevant to the question at hand.
So yes, this article raises some interesting points, but I don't see it accomplishing what it seems to purport to do.
If the expectation of a superhuman AI takeover is built on five key assumptions that have no basis in evidence, then this idea is more akin to a religious belief — a myth
My problem is, even this is wrong. I do not find the "five key assumptions" he cites to be the foundation of beliefs that a dangerous "Superhuman AI" could emerge. So to me, this article simply fails right from the jump. The thing he is arguing for and/or against just don't, collectively, constitute a well-developed argument for the overall thesis. Not in my opinion anyway.
To go through them one by one:
Artificial intelligence is already getting smarter than us, at an exponential rate.
I don't know anybody making this claim, nor is it apparent that belief in this is necessary to think that dangerous "Superhuman AI" could be a problem in the future. I guess it comes down to what you think Kelly meant by "soon" in the preceding sentence.
We’ll make AIs into a general purpose intelligence, like our own.
It's the "like our own" part of this that I have a problem with. Yes, I think many people believe that we will create some manner of "general intelligence", but I also believe that a lot of researchers acknowledge that it may not be 100% "human like". But this may be my own bias showing, as I personally harp a lot on the importance of the distinction between "human level" intelligence and "human like" intelligence.
We can make human intelligence in silicon.
This is probably the most sound one of all, but here's the thing... saying "there is no evidence for this" is simply wrong. To hold to this is to completely ignore all progress made building AI in silicon to date. And while we don't yet have "general intelligence", I don't see how one can just ignore incremental progress completely and say "there's nothing there". I would argue that the null hypothesis should probably be that general intelligence on silicon is possible that that the burden of proof is on the people arguing the opposite.
Also, not everybody thinking about / working on AI is focused on the "in silicon" approach. The majority? Yes. But I'm sure there are a handful of people working on ideas related to DNA computing, chemical computing, growing synthetic brain matter, etc.
Intelligence can be expanded without limit.
To me, this may or not not be true, but it's completely irrelevant to the thesis at hand. The question isn't "can machine intelligence be unbounded?", but rather "can machine intelligence exceed human intelligence?"
Once we have exploding superintelligence it can solve most of our problems.
Again, this strikes me as mostly irrelevant to the question at hand.
So yes, this article raises some interesting points, but I don't see it accomplishing what it seems to purport to do.
Yeah. All you need is an intelligence that's human level but faster, and you have something that is, to some extent, superhuman. If you can get to human intelligence levels you can just crank up the clock speed a bit and give it a better working memory, and now you have something like a super-Einstein in a box.
It could be that there's no way to do that in silica without outstripping plausible bounds on hardware size or power use, I don't know...
It could be that there's no way to do that in silica without outstripping plausible bounds on hardware size or power use, I don't know...
But Einstein was no ordinary human, he was a genius. So you would need the AGI to be more than just your average human. Would the average Joe sped up be some form of superintelligence? They would be able to perform tasks faster, but that doesn't make them smarter.
"Som fanden leser bibelen"
- Norwegian proverb, lit.: "Like the devil reads the bible"
I sense that your arguments are in concordance with TFA, broadly speaking. Also, I suspect you read it rather uncharitably for reasons having to do with tone or stated thesis. Either that, or my reading comprehension is failing me. Maybe I want it to be more convincing that it really is, as i already laid bare my sentiments towards accelerationist woo.
I sense that your arguments are in concordance with TFA, broadly speaking. Also, I suspect you read it rather uncharitably for reasons having to do with tone or stated thesis. Either that, or my reading comprehension is failing me. Maybe I want it to be more convincing that it really is, as i already laid bare my sentiments towards accelerationist woo.
Norwegian proverb, lit.: "Like the devil reads the bible"
I'll take that as a compliment!
All joking aside, I'm not saying every word of the article is wrong or anything. Maybe I'm just being too literal or pedantic or something. It just seems to me that the article sets out to make one very specific claim, and I just don't think it succeeds in making a strong case for that specific claim. Is there some interesting discussion there? Yes. Are there things that Kelly says that I agree with? Yes, absolutely.
So maybe we're not too far apart on this after all.
I'll take that as a compliment!
All joking aside, I'm not saying every word of the article is wrong or anything. Maybe I'm just being too literal or pedantic or something. It just seems to me that the article sets out to make one very specific claim, and I just don't think it succeeds in making a strong case for that specific claim. Is there some interesting discussion there? Yes. Are there things that Kelly says that I agree with? Yes, absolutely.
So maybe we're not too far apart on this after all.
Maybe the article was AI generated? /s
Humanity is an existence proof for human level AI, and it seems arrogant to assume that our brains are at the limit of what is possible.
Reproductive fitness requires many attributes which obviously compete with intelligence, and evolution is a blind hill-climbing search which cannot easily escape deep local minima.
To me it's obvious that the development of superhuman AI is inevitable assuming we don't wipe ourselves out before we have time to develop it.
Reproductive fitness requires many attributes which obviously compete with intelligence, and evolution is a blind hill-climbing search which cannot easily escape deep local minima.
To me it's obvious that the development of superhuman AI is inevitable assuming we don't wipe ourselves out before we have time to develop it.
But Google can read, injest, and search for information far faster than I can.
Stockfish players chess far better than I can.
AlphaGo plays Go far better than I can.
Random Sudoku bot built on top of minizinc / z3 / whatever SAT solver can solve NP-complete problems better than any human.
--------
Here's the thing: what is "strong AI" ?? As far as I can tell, we're already at superhuman AI in most intelligence tasks (collecting data, collating it together, analyzing it, etc. etc.).
Its called a computer. What if "superhuman AI" is just the stuff we have today at our fingertips? It wasnt very long ago when reading documents and organizing them was considered AI. But today, we call that Google.
Stockfish players chess far better than I can.
AlphaGo plays Go far better than I can.
Random Sudoku bot built on top of minizinc / z3 / whatever SAT solver can solve NP-complete problems better than any human.
--------
Here's the thing: what is "strong AI" ?? As far as I can tell, we're already at superhuman AI in most intelligence tasks (collecting data, collating it together, analyzing it, etc. etc.).
Its called a computer. What if "superhuman AI" is just the stuff we have today at our fingertips? It wasnt very long ago when reading documents and organizing them was considered AI. But today, we call that Google.
Stockfish and AlphaGo are also available for humans to use. A human using Stockfish will be just as good at chess as a strong AI using Stockfish, and likewise for any future improvements to chess engines that either humans or strong AIs develop.
There are no superhuman craftsman AI.
Something that could in one package:
- hold a conversation about what it is that needs to be done
- clean a room
- fix a sink
- make a chair or table out of wood
- dig a hole
We don't know how to make AI that will learn to play even simple game like montezuma's revenge in some reasonable play time like 30 min.
We are currently closer to months or years of gameplay required.
Something that could in one package:
- hold a conversation about what it is that needs to be done
- clean a room
- fix a sink
- make a chair or table out of wood
- dig a hole
We don't know how to make AI that will learn to play even simple game like montezuma's revenge in some reasonable play time like 30 min.
We are currently closer to months or years of gameplay required.
> We don't know how to make AI that will learn to play even simple game like montezuma's revenge in some reasonable play time like 30 min.
Explain to me why a Tool-assisted Speedrun doesn't count as "AI". Its certainly not a human who is playing that (https://www.youtube.com/watch?v=pFB0kuPx5mQ).
> Something that could in one package:
I'm not sure if that sort of thing is worth the investment.
But if we have "Fly the F35 airframe" (which is incredibly unstable), then we suddenly reach "only computers can fly this thing" levels. Just the assumption of super-human control systems is baked into modern products. https://www.youtube.com/watch?v=cyN-CRNrb3E
Our cell phones are largely manufactured by machines these days. A few tidbits here and there remain in human hands for cheapness... but a lot of today's stuff is basically an AI. https://www.youtube.com/watch?v=SRu02F6AOmg
Explain to me why a Tool-assisted Speedrun doesn't count as "AI". Its certainly not a human who is playing that (https://www.youtube.com/watch?v=pFB0kuPx5mQ).
> Something that could in one package:
I'm not sure if that sort of thing is worth the investment.
But if we have "Fly the F35 airframe" (which is incredibly unstable), then we suddenly reach "only computers can fly this thing" levels. Just the assumption of super-human control systems is baked into modern products. https://www.youtube.com/watch?v=cyN-CRNrb3E
Our cell phones are largely manufactured by machines these days. A few tidbits here and there remain in human hands for cheapness... but a lot of today's stuff is basically an AI. https://www.youtube.com/watch?v=SRu02F6AOmg
> Explain to me why a Tool-assisted Speedrun doesn't count as "AI". Its certainly not a human who is playing that (https://www.youtube.com/watch?v=pFB0kuPx5mQ).
Because TAS are made by human speedrunners, who go through the painstaking process of specifying inputs for each frame. I'm not sure what you're trying to say here, TAS are basically the exact opposite of AI.
> I'm not sure if that sort of thing...
> But if we have "Fly the F35 airframe"...
> Our cell phones are largely manufactured by machines...
That's a red herring. There's plenty of evidence we can make very useful narrow AI in many fields, and nobody is arguing against that.
The point is, they are, well, narrow. What your parent is saying is that there's no evidence that the statistical techniques behind the current wave of AI are capable of anything else.
To paraphrase Norvig, it could well be that we're trying to get to the moon by climbing a tree, reporting steady progress all the way to the top of the tree.
Because TAS are made by human speedrunners, who go through the painstaking process of specifying inputs for each frame. I'm not sure what you're trying to say here, TAS are basically the exact opposite of AI.
> I'm not sure if that sort of thing...
> But if we have "Fly the F35 airframe"...
> Our cell phones are largely manufactured by machines...
That's a red herring. There's plenty of evidence we can make very useful narrow AI in many fields, and nobody is arguing against that.
The point is, they are, well, narrow. What your parent is saying is that there's no evidence that the statistical techniques behind the current wave of AI are capable of anything else.
To paraphrase Norvig, it could well be that we're trying to get to the moon by climbing a tree, reporting steady progress all the way to the top of the tree.
When I was involved in physics I thought that the range of problems that were solvable were like a continent (linear systems) surrounded by islands (integrable systems and a domain in which perturbation theory works around them.)
I thought that the problems solved by Bethe, Feynman and their cohort were the easy problems and the current generation is struggling with problems that were much harder, possibly impossible to solve.
One example is the problem of chaotic dynamics with six degrees of freedom as opposed to four degrees of freedom. With four degrees of freedom you have unbroken KAM tori that partition the phase space into areas that it will stay in forever. In 6 DOF or more, some tori are still unbroken but it's possible for the system to go around them, so even in a phase space region that has a lot of intact tori (e.g. the motion of the planets around the Sun) you can't prove anything for the long term (e.g. the Earth doesn't get ejected from the solar system.)
Look also at examples such as the halting problem, Godel's theorems, etc. I'm not sure if a "higher intelligence" could actually solve that many problems we can't because the problems are intractable.
I thought that the problems solved by Bethe, Feynman and their cohort were the easy problems and the current generation is struggling with problems that were much harder, possibly impossible to solve.
One example is the problem of chaotic dynamics with six degrees of freedom as opposed to four degrees of freedom. With four degrees of freedom you have unbroken KAM tori that partition the phase space into areas that it will stay in forever. In 6 DOF or more, some tori are still unbroken but it's possible for the system to go around them, so even in a phase space region that has a lot of intact tori (e.g. the motion of the planets around the Sun) you can't prove anything for the long term (e.g. the Earth doesn't get ejected from the solar system.)
Look also at examples such as the halting problem, Godel's theorems, etc. I'm not sure if a "higher intelligence" could actually solve that many problems we can't because the problems are intractable.
The work that most (even highly skilled) people do is more like playing chess or writing a novel than proving theorems.
Not pushing the boundaires of human knowledge, but requiring skills which are not easily learned without experience and within which there is a huge variation in human ability.
Can you really say that the existence of an agent (or a million) matching or exceeding the 100th percentile in many different skills would not be a massive change?
Not pushing the boundaires of human knowledge, but requiring skills which are not easily learned without experience and within which there is a huge variation in human ability.
Can you really say that the existence of an agent (or a million) matching or exceeding the 100th percentile in many different skills would not be a massive change?
Superhuman chess playing A.I. is a reality today.
I don't think it changes the human condition.
If any kind of "Terminator" threatens the human race I think it is something that can understand individual psychology and seduce anyone.
I don't think it changes the human condition.
If any kind of "Terminator" threatens the human race I think it is something that can understand individual psychology and seduce anyone.
> Humanity is an existence proof for human level AI, and it seems arrogant to assume that our brains are at the limit of what is possible.
Why? Is it arrogant to assume that Turing machines are at the limit of what is possible to compute? Perhaps, but we also have good reasons to believe that to be the case. If you're just talking about things like computational speed differences or being able to control powerful machinery and weaponry, then those are already available for humans to use, and can already be used by humans to do bad things much more effectively. It seems to me that any future improvements in computational speed, memory capacity, etc. will also continue to be available to humans as well as any strong AIs we might develop.
Why? Is it arrogant to assume that Turing machines are at the limit of what is possible to compute? Perhaps, but we also have good reasons to believe that to be the case. If you're just talking about things like computational speed differences or being able to control powerful machinery and weaponry, then those are already available for humans to use, and can already be used by humans to do bad things much more effectively. It seems to me that any future improvements in computational speed, memory capacity, etc. will also continue to be available to humans as well as any strong AIs we might develop.
>This is a pretty bad article.
Agreed.
Dunno why he thinks this: "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale."
The notion of self-improvement is both powerful and not assumed to be 'infinite'.
I figure it goes like this:
Q. Is AI potentially dangerous? A. Yes.
Q. Will it be human-like? A. Probably not.
People tend to couch these discussions as a series of anthropomorphisms.
Agreed.
Dunno why he thinks this: "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale."
The notion of self-improvement is both powerful and not assumed to be 'infinite'.
I figure it goes like this:
Q. Is AI potentially dangerous? A. Yes.
Q. Will it be human-like? A. Probably not.
People tend to couch these discussions as a series of anthropomorphisms.
The article is a response to claims about the singularity and subsequent worries about concerning super AI.
> Intelligence is not a single dimension, so “smarter than humans” is a meaningless concept.
That's not the point. We can map any multi-dimensional form of intelligence into a single dimension with any suitable norm. It just matters that for any dimension we can quantify intelligence in the sense that something more intelligent is imaginable.
> Humans do not have general purpose minds, and neither will AIs.
That does not matter at all. Then a superhuman intelligence is not "general", so what? It's still wastly more capable then any human.
> Emulation of human thinking in other media will be constrained by cost.
But what is the cost constraints? Human brains are constrained by mechanical considerations. How efficient could a computerized AI become?
> Dimensions of intelligence are not infinite.
Again, what's the argument? The number of atoms in the universe is not infinite, either (probably).
> Intelligences are only one factor in progress.
That's not the problem of the super AI.
That's not the point. We can map any multi-dimensional form of intelligence into a single dimension with any suitable norm. It just matters that for any dimension we can quantify intelligence in the sense that something more intelligent is imaginable.
> Humans do not have general purpose minds, and neither will AIs.
That does not matter at all. Then a superhuman intelligence is not "general", so what? It's still wastly more capable then any human.
> Emulation of human thinking in other media will be constrained by cost.
But what is the cost constraints? Human brains are constrained by mechanical considerations. How efficient could a computerized AI become?
> Dimensions of intelligence are not infinite.
Again, what's the argument? The number of atoms in the universe is not infinite, either (probably).
> Intelligences are only one factor in progress.
That's not the problem of the super AI.
FWIW, I agree with your sentiments pretty much 100%. I was going to say more or less the same thing(s) but didn't feel motivated enough to do more of a "point by point" response. Glad to see somebody chipping in with very similar viewpoints.
Reminds me of the brilliant "On the Impossibility of Supersized Machines" [1].
We show that it is not only implausible that machines will ever exceed human size, but in fact impossible.
[...]
The term “supersized machine” implies a machine that has crossed some threshold, which is often denoted “human-level largeness.” However, it is not clear what “human-level largeness” could refer to. Has a machine achieved human-level largeness if it has the same height as the average human? If it has the same volume? The same weight? [...]
When one begins to consider these questions, one quickly concludes that there are an infinite number of metrics that could be used to measure largeness, and that people who speak of “supersized machines” do not have a particular metric in mind. Surely, then, any future machine will be larger than humans on some metrics and smaller than humans on others, just as they are today.
[1] https://arxiv.org/abs/1703.10987
We show that it is not only implausible that machines will ever exceed human size, but in fact impossible.
[...]
The term “supersized machine” implies a machine that has crossed some threshold, which is often denoted “human-level largeness.” However, it is not clear what “human-level largeness” could refer to. Has a machine achieved human-level largeness if it has the same height as the average human? If it has the same volume? The same weight? [...]
When one begins to consider these questions, one quickly concludes that there are an infinite number of metrics that could be used to measure largeness, and that people who speak of “supersized machines” do not have a particular metric in mind. Surely, then, any future machine will be larger than humans on some metrics and smaller than humans on others, just as they are today.
[1] https://arxiv.org/abs/1703.10987
[deleted]
Yeah, I stopped reading after making roughly the same assessment. Article is just one giant straw man. And I'm not a particularly staunch AGI proponent.
There is no reason we know that brain emulation is not possible. Even if one second takes 100 years in processing now, there is no reason to think that there is fundamental limit in how many processor you could connect. And for superhuman it just needs to do one thing well(which computers already does) than human and match humans in everything. I am not saying it would be done but I haven't seen any argument on why this exact argument is "impossible".
> There is no reason we know that brain emulation is not possible.
Well, neither is there any reason we know that it's possible.
You're just demonstrating your apriori biases ("the brain is just a computer"), nothing more.
Well, neither is there any reason we know that it's possible.
You're just demonstrating your apriori biases ("the brain is just a computer"), nothing more.
We have reason to believe it's possible because we know we can simulate a single neuron. It's just the sheer complexity of full brain we can't scan and efficiently simulate.
There's absolutely no reason to believe that the brain can be reduced to a collection of neurons. It's what we want to believe, because it's simpler and makes us feel in control; but in real nature there's intelligence without neurons, and the brain needs much more than neurons to do its thing.
P.S. If we presume that human intelligence is a random processes, then there's no reason to think that this random process can create another that has more information complexity than the original. It makes no sense mathematically.
P.S. If we presume that human intelligence is a random processes, then there's no reason to think that this random process can create another that has more information complexity than the original. It makes no sense mathematically.
It's not impossible in principle, but:
- we don't understand enough about the brain to emulate it.
- we don't know how to model the brain.
- even assuming we had the required scientific knowledge, there would likely be major engineering problems.
So, in other words, it's not impossible in the same sense FTL travel is not impossible, our scientific understanding does not categorically exclude it, but making predictions as to when it will happen is kind of silly, because it will likely require several fundamentally unpredictable scientific breakthroughs.
- we don't understand enough about the brain to emulate it.
- we don't know how to model the brain.
- even assuming we had the required scientific knowledge, there would likely be major engineering problems.
So, in other words, it's not impossible in the same sense FTL travel is not impossible, our scientific understanding does not categorically exclude it, but making predictions as to when it will happen is kind of silly, because it will likely require several fundamentally unpredictable scientific breakthroughs.
As the article points out, the nervous systems is larger than the brain. There are millions of neurons in your gut. Your sensory organs have neurons that prepare the signal sent to the brain from the sensory noise. No brain lives independent of a body. That's not how organisms evolved.
> there is no reason to think that there is fundamental limit in how many processor you could connect
Square-cube law. As the number of processors grow how do you hook them together? If you want to minimize latency you end up with a solid packed sphere. O(cube root) latency. In that situation, heat dissipation only scales with the surface area of that sphere. So there is a limit to the scale. If you choose a topology that permits cooling, then you accept ever increasing inter-processor latency which eventually bottlenecks your computation.
Computers are not magic. They exist in a physical reality and have to deal with thermodynamics and the speed of light. Just because we feel like we have a lot of them doesn't mean we have enough for "magical super AI" to work.
Square-cube law. As the number of processors grow how do you hook them together? If you want to minimize latency you end up with a solid packed sphere. O(cube root) latency. In that situation, heat dissipation only scales with the surface area of that sphere. So there is a limit to the scale. If you choose a topology that permits cooling, then you accept ever increasing inter-processor latency which eventually bottlenecks your computation.
Computers are not magic. They exist in a physical reality and have to deal with thermodynamics and the speed of light. Just because we feel like we have a lot of them doesn't mean we have enough for "magical super AI" to work.
Bold claim.
At the moment we don't fully understand the brain and intelligence. Even what makes the difference a living and a non living accumulation of atoms is unknown.
But this doesn't not mean we couldn't create a superhuman AI in the future or by accident.
I think we don't have the capabilities to do so but this isn't a 100% certaincy more like 99%
But this doesn't not mean we couldn't create a superhuman AI in the future or by accident.
I think we don't have the capabilities to do so but this isn't a 100% certaincy more like 99%
"the problem with the world is that the intelligent are full of doubt and the idiots cocksure". Applies directly to the bold title. Doesn't matter if we are all annihilated of course nobody will remember.
Interesting discussions. Perhaps it may be easier to look at it from another angle.
Current blend of ML/DL algorithms are not AI. I think the tendency of giving aspirational names to software algorithms have created a lot of confusion. Perhaps AI will be one day possible but if so, it's probably safe to say that it won't be anything based on the current ML/DL paradigm.
Fun example, Google DeepMind is yet able to learn math:
https://www.zdnet.com/article/ai-aint-no-a-student-deepmind-...
Current blend of ML/DL algorithms are not AI. I think the tendency of giving aspirational names to software algorithms have created a lot of confusion. Perhaps AI will be one day possible but if so, it's probably safe to say that it won't be anything based on the current ML/DL paradigm.
Fun example, Google DeepMind is yet able to learn math:
https://www.zdnet.com/article/ai-aint-no-a-student-deepmind-...
We keep making glorified optic nerves and spinal reflexes and calling it a brain.
Only skimmed the bullet points, but one that’s missing and seems important to me is:
* The marginal utility of intelligence may not be so great. If you are already really smart, is there any point to being smarter?
Most interesting problems are NP-complete. If you are really smart you can solve a big NP-complete problem. But how much smarter do you need to be to solve a slightly bigger (and therefore more interesting) problem? Well, twice as smart or something. Being just a little bit smarter gets you nothing. So (in this case) the marginal utility of intelligence approaches zero as your AI gets more intelligent.
* The marginal utility of intelligence may not be so great. If you are already really smart, is there any point to being smarter?
Most interesting problems are NP-complete. If you are really smart you can solve a big NP-complete problem. But how much smarter do you need to be to solve a slightly bigger (and therefore more interesting) problem? Well, twice as smart or something. Being just a little bit smarter gets you nothing. So (in this case) the marginal utility of intelligence approaches zero as your AI gets more intelligent.
Being 10% smarter puts you just marginally ahead of other people; and a competitive marginal advantage is often major.
So I'd say there's great utility in an AI being smarter than any human... and then the returns diminish.
So I'd say there's great utility in an AI being smarter than any human... and then the returns diminish.
Old article. Previous discussion: https://news.ycombinator.com/item?id=14205042
Looks like you're being ignored, or at least no one (until me) has replied. I guess people would rather hear themselves think than look at what other people said a few years ago.
But thanks for bringing up this link!
But thanks for bringing up this link!
Other commenters have mentioned that these particular points have been refuted elsewhere, so I won't rehash that. But I do think it's interesting that the author frames the "orthodox" position as the one taking AI alignment seriously. That's pretty big progress and I think the folks who have been sounding the alarm[0] about AGI should give themselves a pat on the back that the zeitgeist has shifted so much.
The arguments from the anti-alarmism camp have been so consistently weak, I think the best explanation for their motivation is still something like "All this AGI stuff sounds so sci-fi. It can't be a real concern". It just goes to show how strongly we bias our expectations of the future on our what we've personally experienced. You really have to try hard to escape that default mode of thinking.
[0] By sounding the alarm I mean saying things like "This is at least a possible scenario with huge error bars on when it might happen, maybe at least some people should be thinking hard about this, rather than nobody"
The arguments from the anti-alarmism camp have been so consistently weak, I think the best explanation for their motivation is still something like "All this AGI stuff sounds so sci-fi. It can't be a real concern". It just goes to show how strongly we bias our expectations of the future on our what we've personally experienced. You really have to try hard to escape that default mode of thinking.
[0] By sounding the alarm I mean saying things like "This is at least a possible scenario with huge error bars on when it might happen, maybe at least some people should be thinking hard about this, rather than nobody"
It doesn't really even sound sci-fi, or rather, it would make for some bad sci-fi. What we are at risk of, is having deterministic model artifacts trigger complex unintended consequences from being fetishized as decision-making agents. Considering how few CS and AI people I've met who even know that e.g. pragmatics are a critical component of sentence meaning, I sense the most proximate AI dystopia is a lot closer to this latter scenario.
Conceptually the possibility of an AGI does present a real challenge (& one worth grappling); in practice, we haven't made any progress really toward realizing that problem. We've become good at feeding "large" datasets into algorithms that are many orders of magnitude less advanced than a lizard brain. Then we get to the issue that our large datasets are only large when thinking of classical statistical sample sizes. They nonetheless are lacking in redundancy, interactivity, stakes or any centered perspective such that there is no chance of anything resembling intelligence, agency, or plain old intentions arising.
Conceptually the possibility of an AGI does present a real challenge (& one worth grappling); in practice, we haven't made any progress really toward realizing that problem. We've become good at feeding "large" datasets into algorithms that are many orders of magnitude less advanced than a lizard brain. Then we get to the issue that our large datasets are only large when thinking of classical statistical sample sizes. They nonetheless are lacking in redundancy, interactivity, stakes or any centered perspective such that there is no chance of anything resembling intelligence, agency, or plain old intentions arising.
It's all fun and games until you lose your job to a robot. Likewise superhuman AI already exists in some specific tasks. Machines, for instance, have been superior to humans in facial recognition and matching for 15 years now, if not longer. The speed at which things are improving in other tasks would be considered ludicrous in any other field. Yet people still persist in thinking that it has to be some sentient robot in order to have a huge societal impact.
We are on the precipice of many labor intensive but relatively low skill professions going away entirely, and yet people still stress out about synthetic intelligence they can have a conversation with, because it sounds "scarier" to the readership of Wired, which never was anywhere near these imperiled jobs and whose livelihood does not depend on them. But a program that can write crappy-yet-plausible Python code? Now you got their clicks and attention.
We are on the precipice of many labor intensive but relatively low skill professions going away entirely, and yet people still stress out about synthetic intelligence they can have a conversation with, because it sounds "scarier" to the readership of Wired, which never was anywhere near these imperiled jobs and whose livelihood does not depend on them. But a program that can write crappy-yet-plausible Python code? Now you got their clicks and attention.
Someone must have written at some point "the myth of a pandemic". Its incomprehensibly stupid to dismiss any kind of threat as a myth unless they prove that it's impossible by laws of physics. And still then there would be room for error. Sad thst a respectable magazine posts such naive articles. I dismiss it from the title before even reading it.
nobody who is sane would have made that claim about a pandemic because we have empirical evidence for the existence of pandemics. In fact in contrast to superintelligence we have a pretty clear model and definition of what a pandemic entails.
The notion that we ought to take everything seriously that is 'not impossible by the laws of physics' is entirely stupid. Because virtually anything is possible, the question is how likely it is, and what evidence we have for it that is quantifiable and meaningful. Or are you currently prepping for an alien invasion?
Myth is the absolutely correct label for 'superhuman AI' fantasies, because the difference between a real threat and a myth is that the former is grounded in empiricism while the latter is a product of psychology and storytelling.
And superhuman AI is a very old story, it's simply the Golem myth. The religious idea that humans, in their hubris, create a tool beyond their control. It's Goethe's The Sorcerer's Apprentice and it's The Terminator.
Also, Kevin Kelly the author of the article is the founder of the magazine.
The notion that we ought to take everything seriously that is 'not impossible by the laws of physics' is entirely stupid. Because virtually anything is possible, the question is how likely it is, and what evidence we have for it that is quantifiable and meaningful. Or are you currently prepping for an alien invasion?
Myth is the absolutely correct label for 'superhuman AI' fantasies, because the difference between a real threat and a myth is that the former is grounded in empiricism while the latter is a product of psychology and storytelling.
And superhuman AI is a very old story, it's simply the Golem myth. The religious idea that humans, in their hubris, create a tool beyond their control. It's Goethe's The Sorcerer's Apprentice and it's The Terminator.
Also, Kevin Kelly the author of the article is the founder of the magazine.
Even worse that Kelly wrote it. Have you even read the statistics that majority of AI experts believe it is possible? Are you an AI expert? Shouldn't we listen to the scientists on the subject instead of doing our own research like in Covid? The question was rhetorical. Yes we should listen to the scientists and not dismiss what they say is a probable scenario that could lead to annihilation. I also suggest to you and anyone that hasn't read the book "Superintelligence" to read it because the arguments there are solid.
It’s really frightening that “AI risk isn’t real” is rapidly congealing into a tribe-signaling _political_ stance that generates reliable terrible articles like this one and the one in the Washington Post a few weeks ago. You’d think with Covid still killing thousands of people a day the media would more able to recognize that fault mode and not repeat the “Covid isn’t a problem, because saying that it’s a problem would be racist” error of February 2020, but apparently not.
[deleted]
tl;dr: Humans have souls which machines will never have. The core of the article is this quote "I will extend that further to claim that the only way to get a very human-like thought process is to run the computation on very human-like wet tissue. That also means that very big, complex artificial intelligences run on dry silicon will produce big, complex, unhuman-like minds." which is just the boring old argument against replicating human thinking.
Similarly, there is an extreme amount of pro-human arrogance in the article "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale. I find no evidence for this.... So the question is, where is the limit of intelligence? We tend to believe that the limit is way beyond us, way “above” us, as we are “above” an ant. Setting aside the recurring problem of a single dimension, what evidence do we have that the limit is not us? Why can’t we be at the maximum?"
Humans are just a bunch of atoms that randomly learned to think over a few billion years. Given how fast we can improve tech via directed means, we should be able to replicate what nature has done in thousands, hundreds, or tens of years.
Similarly, there is an extreme amount of pro-human arrogance in the article "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale. I find no evidence for this.... So the question is, where is the limit of intelligence? We tend to believe that the limit is way beyond us, way “above” us, as we are “above” an ant. Setting aside the recurring problem of a single dimension, what evidence do we have that the limit is not us? Why can’t we be at the maximum?"
Humans are just a bunch of atoms that randomly learned to think over a few billion years. Given how fast we can improve tech via directed means, we should be able to replicate what nature has done in thousands, hundreds, or tens of years.
The word "soul" does not appear once in the article
> That also means that very big, complex artificial intelligences run on dry silicon will produce big, complex, unhuman-like minds.
Incidentally, this is exactly what the AI superintelligence people are terrified of, and what the paperclip problem is meant to illustrate. https://www.decisionproblem.com/paperclips/index2.html
Incidentally, this is exactly what the AI superintelligence people are terrified of, and what the paperclip problem is meant to illustrate. https://www.decisionproblem.com/paperclips/index2.html
Who is the author Kevin Kelly and why does he think he has a greater hold on this issue than luminaries like Ray Kurzweil?
>luminaries like Ray Kurzweil
This is satire, right?
This is satire, right?
To the first half of your question:
https://en.wikipedia.org/wiki/Kevin_Kelly_(editor)
To the second half:
Dunno. Good question.
https://en.wikipedia.org/wiki/Kevin_Kelly_(editor)
To the second half:
Dunno. Good question.
Appeals to authority are not convincing
This isn't a formal debate, and authority is useful especially as a "first pass" filter, or in terms of establishing Bayesian priors. The key, IMO, is not to assume that the mere existence of authority guarantees that someone is right. But in either case, it's certainly valid to ask the question "why should I take Kevin Kelly seriously on this topic?"
I don't agree in this context. The author wrote an entire article about his position, and dismissing it entirely because some "luminary" disagrees is not an interesting response.
OP didn't even bother to point out how or why Kurzweil purportedly disagrees.
OP didn't even bother to point out how or why Kurzweil purportedly disagrees.
I thought it was obvious. Clearly it was not. Ray Kurzweil is a futurist who has spent multiple decades researching this topic and has written multiple books on it. His bio speaks for itself.
https://en.wikipedia.org/wiki/Ray_Kurzweil
https://en.wikipedia.org/wiki/Ray_Kurzweil
I guess I interpreted tech-historian's post a little differently than you.
Likewise, I don't support dismissing Kelly's article out-of-hand, based on Kurzweil's position. But I do think it's fair to raise the issue in terms of "why should one take Kevin Kelly seriously on this?"
Likewise, I don't support dismissing Kelly's article out-of-hand, based on Kurzweil's position. But I do think it's fair to raise the issue in terms of "why should one take Kevin Kelly seriously on this?"
Why is Ray Kurzweil considered a luminary in this context?
Sure, he talks a lot. He makes grand statements. Is there any reason to take him seriously in this context?
He sure seems to believe, and to want to believe. But actual evidence that he's going to be right any time soon? That seems to be missing.
Sure, he talks a lot. He makes grand statements. Is there any reason to take him seriously in this context?
He sure seems to believe, and to want to believe. But actual evidence that he's going to be right any time soon? That seems to be missing.
Everybody wants to be the Fox Mulder of their story.
It's clear you're not familiar with Kurzweil's work in AI (or his background in general). He's written multiple books on the topic over multiple decades, including the seminal "The Singularity is Near". He's devoted his life to the study of this field.
https://en.wikipedia.org/wiki/Ray_Kurzweil
https://en.wikipedia.org/wiki/The_Singularity_Is_Near
https://en.wikipedia.org/wiki/Ray_Kurzweil
https://en.wikipedia.org/wiki/The_Singularity_Is_Near
Thank you for enlightening us with the new knowledge of this wonderful luminary. I have learned so much from these URLs. Do you have any more recommendations on what we should read next? I find myself deeply enthralled by his fascinating claims and I want to seek out more of his teachings.
He's also written books on how much of your diet should be fat, and similar topics. I've never been able to tell if he's worth taking seriously. My instincts say no.
https://en.wikipedia.org/wiki/The_10%25_Solution_for_a_Healt...
https://en.wikipedia.org/wiki/Fantastic_Voyage:_Live_Long_En...
https://en.wikipedia.org/wiki/The_10%25_Solution_for_a_Healt...
https://en.wikipedia.org/wiki/Fantastic_Voyage:_Live_Long_En...
> It's clear you're not familiar with Kurzweil's work in AI (or his background in general).
Because it's impossible that I could be familiar with his work and still think that he's full of it? (At least on AI and the Singularity.) That sounds like you've had a bit too much Kurzweil Koolaid...
Because it's impossible that I could be familiar with his work and still think that he's full of it? (At least on AI and the Singularity.) That sounds like you've had a bit too much Kurzweil Koolaid...
Have you read Ray's books? I recommend 2005's The Singularity is Near. https://en.wikipedia.org/wiki/The_Singularity_Is_Near
It's been 16 years. Is the singularity any nearer? Is there any evidence that the singularity is any nearer?
So, how "near" was the singularity in 2005? How accurate was Kurzweil?
So, how "near" was the singularity in 2005? How accurate was Kurzweil?
His prediction is that a machine will pass the turing test in 2029.
And yes, there's evidence of incredible machine learning discoveries all around us. You'd have to be willfully blind not to see those. AlphaZero in Go & Chess, the protein folding miracles of Deepmind, all the amazing work being done today. The field is racing ahead. Human minds are good at seeing linear progression. Our brains aren't particularly good at exponential growth, which is what is happening in this field.
And yes, there's evidence of incredible machine learning discoveries all around us. You'd have to be willfully blind not to see those. AlphaZero in Go & Chess, the protein folding miracles of Deepmind, all the amazing work being done today. The field is racing ahead. Human minds are good at seeing linear progression. Our brains aren't particularly good at exponential growth, which is what is happening in this field.
He has timeliness on the optimistic side but people were skeptical of his claims in the 80s that a computer will win at Chess by 2000 or that the internet will become ubiquitous. He did think a US company will reach 1 trillion $ by 2010 and that speech recognition will be more widely used but even in cases like that he wasn't massively off.
As far as I can tell for decades he's been consistently more right than his critics and often on issues that are a lot harder to get right than simple coin tosses.
As far as I can tell for decades he's been consistently more right than his critics and often on issues that are a lot harder to get right than simple coin tosses.
He's been right (approximately) on how fast the tech improves. That doesn't tell you anything about how likely he is to be right about the singularity, though, unless you think he actually has a handle on how much tech would have to improve to get us there, which means that he would have a handle on where "there" actually is, in technical terms. I don't see evidence that he (or anyone) has that.
I've read Ray's books. He's a big talker that writes speculative science fiction, but he doesn't do anything.
As the parent poster said, there is zero reason to take him seriously.
As the parent poster said, there is zero reason to take him seriously.
He was inventor of:
the first CCD flat-bed scanner
first omni-font optical character recognition
the first print-to-speech reading machine for the blind
the first text-to-speech synthesizer
the first music synthesizer capable of recreating a grand piano
the first CCD flat-bed scanner
first omni-font optical character recognition
the first print-to-speech reading machine for the blind
the first text-to-speech synthesizer
the first music synthesizer capable of recreating a grand piano
I'm aware of his track record prior to becoming a science fiction author.
But what has he invented after he started writing about the singularity?
But what has he invented after he started writing about the singularity?
> Hyper-intelligent algorithms are not going to take over the world for these five reasons.
I agree.
Just like a nuclear power plant will never meltdown, because they are extremely well designed with large safety buffers.
And just like a pandemic will never be started by lab experiments, because they use extreme safety measures when they work with viruses.
We can always just unplug the AI or disconnect it from the Internet, or just not give it a survival instinct, right?
I agree.
Just like a nuclear power plant will never meltdown, because they are extremely well designed with large safety buffers.
And just like a pandemic will never be started by lab experiments, because they use extreme safety measures when they work with viruses.
We can always just unplug the AI or disconnect it from the Internet, or just not give it a survival instinct, right?
Pretty sure this is sarcasm, as in, the lies are so transparent they're humorous, and this was on purpose.
For those who didn't notice, nuclear plants have, in fact, melted down, more than once. Biosafety level 4 labs have, in fact, had leaks of pathogens before, more than once. An agent AI will tend to develop a survival instinct by default, even if we don't give it one, because it is a convergent instrumental goal. Thus, it would be motivated to not simply let you yank it, would fight you if you tried, and may even take preemptive precautions.
For those who didn't notice, nuclear plants have, in fact, melted down, more than once. Biosafety level 4 labs have, in fact, had leaks of pathogens before, more than once. An agent AI will tend to develop a survival instinct by default, even if we don't give it one, because it is a convergent instrumental goal. Thus, it would be motivated to not simply let you yank it, would fight you if you tried, and may even take preemptive precautions.
The bottleneck for intelligence is experience. Running on silicon doesn't allow an intelligence to engage with reality significantly faster than a human can. There are some benefits due to access to data and processing power, but the data isn't useful without the context of experience and all processing power can do it overfit model without experience.
Even if we put AI into an army of robots running around and experiencing things, there are still scaling limits to encoding and communicating knowledge and understanding. Human organizations are a great example of the scaling limits of intelligence.
Even if we put AI into an army of robots running around and experiencing things, there are still scaling limits to encoding and communicating knowledge and understanding. Human organizations are a great example of the scaling limits of intelligence.
Actually no, there is almost no chance that it will be harder to transfer skills between machines than it is to raise and educate a child.
Something very weird would have to happen for it to be that hard.
Something very weird would have to happen for it to be that hard.
Is it?
Humans have been optimizing for information transmission between individuals for a long time. Not just our hardware and neural nets have been optimized, but the structure of the information itself.
It is quite possible that you need such a large set of compex interactions to develop intelligence that intelligence can only develop in reality. It is possible that the process for doing this inherently inolves a chaotic system that makes duplicating or paralleling that development impossible. I don't see any reason why these artificial intelligences with divergent chaotically emergent capabilities would have to be able to teach each other things as easily as we do.
I've long been of the opinion culture / "social software" is vital for producing knowledge for humans and that this social software has been fine tuned and iterated upon for a very long time (and possibly co-developed with the evolution of our brain hardware so that both are co-specialized for each other). This software is likely very specialized to our brain structures and might not generalize well to other intelligence substrates. Thus even if we do develop neural network structres that can run on silicone and match human level intelligence, those entities would still need a lot of interaction with reality to develop their own culture that would allow them understand the world at the level we do. Edit: Is it even possible to efficiently run the necessary theories of human mind on non-human brains to operate effectively in our epistemic landscape? Communication requires huge amounts of inference using theories of mind to resolve ambiguity and add implicit information.
It is possible that we can "recompile" human knowledge to run on other types of intelligences, but I suspect this will be far more difficult that commonly expected. Just look at how hard (nigh impossible) it is today to even boostrap a new general purpose computer without depending on the output other general purpose computers.
We simply do not understand cognition or semantics well enough to know the answers to these questions adequately or with any degree of certainty.
Humans have been optimizing for information transmission between individuals for a long time. Not just our hardware and neural nets have been optimized, but the structure of the information itself.
It is quite possible that you need such a large set of compex interactions to develop intelligence that intelligence can only develop in reality. It is possible that the process for doing this inherently inolves a chaotic system that makes duplicating or paralleling that development impossible. I don't see any reason why these artificial intelligences with divergent chaotically emergent capabilities would have to be able to teach each other things as easily as we do.
I've long been of the opinion culture / "social software" is vital for producing knowledge for humans and that this social software has been fine tuned and iterated upon for a very long time (and possibly co-developed with the evolution of our brain hardware so that both are co-specialized for each other). This software is likely very specialized to our brain structures and might not generalize well to other intelligence substrates. Thus even if we do develop neural network structres that can run on silicone and match human level intelligence, those entities would still need a lot of interaction with reality to develop their own culture that would allow them understand the world at the level we do. Edit: Is it even possible to efficiently run the necessary theories of human mind on non-human brains to operate effectively in our epistemic landscape? Communication requires huge amounts of inference using theories of mind to resolve ambiguity and add implicit information.
It is possible that we can "recompile" human knowledge to run on other types of intelligences, but I suspect this will be far more difficult that commonly expected. Just look at how hard (nigh impossible) it is today to even boostrap a new general purpose computer without depending on the output other general purpose computers.
We simply do not understand cognition or semantics well enough to know the answers to these questions adequately or with any degree of certainty.
It’s still the case that computer programs can be copied and restored from backup today, and we aren’t going to have Star Trek transporters any time soon.
Even if it does take an enormous investment and many years to train a computer to do a task, you can make as many copies as you want and everyone can have one. Educating people doesn’t have that kind of payoff.
That’s why you see billions spent in the effort to make a self-driving car. But it turns out that real-world experience isn’t enough, so far. Waymo has collected > 20 million miles of driving experience.
It’s not enough because machine learning is very inefficient at learning from experience. That could improve, though.
Even if it does take an enormous investment and many years to train a computer to do a task, you can make as many copies as you want and everyone can have one. Educating people doesn’t have that kind of payoff.
That’s why you see billions spent in the effort to make a self-driving car. But it turns out that real-world experience isn’t enough, so far. Waymo has collected > 20 million miles of driving experience.
It’s not enough because machine learning is very inefficient at learning from experience. That could improve, though.
That's a great point. However, whereas humans share their experience through the effective, but extremely lossy technology called language; machines can store and share their experience without loss. Sure there are scaling limits, but a brain dump back to the mothership is a lot quicker and more perfectly preserved than sitting around listening to grandpa's stories.
Really fascinating counter-argument to this view is here: https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...
The basic idea is that, the smarter you are and the faster you can think, the more information you can extract from weak signals. It involves creativity, and building models, and some minimum amount of in-built bias, but the upshot is that the amount of information a super-intelligent AI could infer from very small amounts of experience might absolutely stagger the imagination.
The basic idea is that, the smarter you are and the faster you can think, the more information you can extract from weak signals. It involves creativity, and building models, and some minimum amount of in-built bias, but the upshot is that the amount of information a super-intelligent AI could infer from very small amounts of experience might absolutely stagger the imagination.
Even if it is right, it is still predicated on a massive "critical mass" of knowledge and insight about reality that such an AI would have to spend a long time accumulating. It also doesn't propose solutions to performance scaling limitations.
Are you of the opinion that you can't get experience by reading books or watching youtube?
Do you think everything taught in an oral form (as opposed to hands on) is completely useless?
Are all military strategy books used by military academies a waste of time, since you can only learn by leading a real army?
Do you think everything taught in an oral form (as opposed to hands on) is completely useless?
Are all military strategy books used by military academies a waste of time, since you can only learn by leading a real army?
All of those things have to be integrated with experience to be useful. Often the learner already has a lot of that experience, so the data can be integrated easily.
You can't understand those books or information without having a lifetime of experience to contextualize them. You can't convert them from rote knowledge into skills without trying them out.
You can't understand those books or information without having a lifetime of experience to contextualize them. You can't convert them from rote knowledge into skills without trying them out.
Everything in the above comment is wrong.
In AlphaZero for example, there were 44 million training games total for 700,000 steps of training for the full 9 hours.
Turning that to human-scale numbers, 44million games with on average 60 moves, at 1 second thinking time per move,
> 44000000*60/60/60/24/365 = 83,7138508371 years of training experience in 9 hours
The whole field of Reinforcement learning has agents training and playing games for many orders of magnitude more time than a human ever will. In-fact, we can scale this to over 100k of actions per second, in a single machine:
https://github.com/alex-petrenko/sample-factory
Then, there is also distributed Reinforcement Learning, where hundreds of agents can play at different machines and share experience, see AlphaZero, LeelaZero, R2D2 agent, R2D3 agent, Apex, Acer, Asynchronous PPO.
> but the data isn't useful without the context of experience
The experience is the data in Reinforcement Learning.
> and all processing power can do it overfit model without experience.
That is wrong, the agents perform what is called exploration to avoid getting stuck in simple strategies.
> Even if we put AI into an army of robots running around and experiencing things, there are still scaling limits to encoding and communicating knowledge and understanding.
True, but machines scale better because they speak the same language, or they can learn to tune their language to get their message across.
> Human organizations are a great example of the scaling limits of intelligence.
Human organization is a testament to how far we can get with something as limiting as the commonly used language. The language that we use to communicate is subject to misinterpretation due to our subjective experiences, this limitation is not shared by machines.
In AlphaZero for example, there were 44 million training games total for 700,000 steps of training for the full 9 hours.
Turning that to human-scale numbers, 44million games with on average 60 moves, at 1 second thinking time per move,
> 44000000*60/60/60/24/365 = 83,7138508371 years of training experience in 9 hours
The whole field of Reinforcement learning has agents training and playing games for many orders of magnitude more time than a human ever will. In-fact, we can scale this to over 100k of actions per second, in a single machine:
https://github.com/alex-petrenko/sample-factory
Then, there is also distributed Reinforcement Learning, where hundreds of agents can play at different machines and share experience, see AlphaZero, LeelaZero, R2D2 agent, R2D3 agent, Apex, Acer, Asynchronous PPO.
> but the data isn't useful without the context of experience
The experience is the data in Reinforcement Learning.
> and all processing power can do it overfit model without experience.
That is wrong, the agents perform what is called exploration to avoid getting stuck in simple strategies.
> Even if we put AI into an army of robots running around and experiencing things, there are still scaling limits to encoding and communicating knowledge and understanding.
True, but machines scale better because they speak the same language, or they can learn to tune their language to get their message across.
> Human organizations are a great example of the scaling limits of intelligence.
Human organization is a testament to how far we can get with something as limiting as the commonly used language. The language that we use to communicate is subject to misinterpretation due to our subjective experiences, this limitation is not shared by machines.
If the universe is a game you are playing, then yes playing that game is "experience", but for an AI to engage with reality it has to have experience in reality, not a game. The ability to play go very well doesn't enable an AGI to better understand reality.
> The experience is the data in Reinforcement Learning.
This is very true, and the critical problem. Data about how reality responds to an AI's actions is very sparse right now.
AIs do have a potential advantage in communications efficiency, but at some level of scale compression will happen, locally "irrelevant" data will be discarded and simplified approximations replace it. None of this will change the "big O" of the scalability of intelligence, just the constant factors. There is no exponential kickoff point.
> The experience is the data in Reinforcement Learning.
This is very true, and the critical problem. Data about how reality responds to an AI's actions is very sparse right now.
AIs do have a potential advantage in communications efficiency, but at some level of scale compression will happen, locally "irrelevant" data will be discarded and simplified approximations replace it. None of this will change the "big O" of the scalability of intelligence, just the constant factors. There is no exponential kickoff point.
What is the difference between experiencing reality and a game?
The difference I can see is that there is no one explicit objective function, but this doesn't stop generally capable agents [1], and doesn't imply that inverse RL is not possible.
> The ability to play go very well doesn't enable an AGI to better understand reality.
I disagree, model based RL constructs a model of the agent's reality and can use it to plan ahead, train the agent, or do some form of monte-carlo tree search. The latter is something very similar to how we imagine and think about the future.
[1] https://deepmind.com/blog/article/generally-capable-agents-e...
The difference I can see is that there is no one explicit objective function, but this doesn't stop generally capable agents [1], and doesn't imply that inverse RL is not possible.
> The ability to play go very well doesn't enable an AGI to better understand reality.
I disagree, model based RL constructs a model of the agent's reality and can use it to plan ahead, train the agent, or do some form of monte-carlo tree search. The latter is something very similar to how we imagine and think about the future.
[1] https://deepmind.com/blog/article/generally-capable-agents-e...
I'm in total agreement about the potential of growing AGI out of these methods, but there will be bottlenecks well before the gods of the singularity come knocking.
> What is the difference between experiencing reality and a game?
Finding out the consequences of an action is a lot more expensive in reality than a simulation of a game.
Finding out the consequences of an action is a lot more expensive in reality than a simulation of a game.
There is nothing fundamentally different between an infinite horizon game and reality.
The reality is, there are a lot of unknowns in this space, and to dismiss the possibility of an "evil super-intelligence" out-of-hand is, IMO, every bit as silly as buying into the "OMG, AI is unleashing the demon" fear-mongering. I just think it's too early to know for sure the answers to some of these questions, like:
1. Is AGI possible at all (for whatever definition of "AGI" you want to use)?
2. If the answer to (1) is "yes", then what about ASI (Artificial Super Intelligence) - is that possible?
3. If (1) and (2) are both "yes", then is it possible to create an ASI that will either be benevolent, or controllable, or both?
And so on... we just don't know yet. And that in and of itself arguably suggests the exercise of a certain measure of caution.