Artificial intelligence risk research(80000hours.org)
80000hours.org
Artificial intelligence risk research
https://80000hours.org/career-guide/top-careers/profiles/artificial-intelligence-risk-research/
78 comments
The possibility of human-level artificial intelligence poses significant risks to society [...]
No. It simply does not outside of the realm of science fiction. Understanding HGI will help us understand enough about ourselves and our motivations and ethics that it won't be an issue.
Humans augmented by machine learning systems, on the other hand, are here now and a wholly different question. Institutions empowered by AI research have a power / person ratio unlike any of those of the past.
No. It simply does not outside of the realm of science fiction. Understanding HGI will help us understand enough about ourselves and our motivations and ethics that it won't be an issue.
Humans augmented by machine learning systems, on the other hand, are here now and a wholly different question. Institutions empowered by AI research have a power / person ratio unlike any of those of the past.
The purpose of the profile isn't to argue a risk exists. We largely defer to the people we take to be experts on the issue, especially Nick Bostrom. We think he presents compelling arguments in Superintelligence, and although it's hard to say anything decisive in this area, if you think there's even modest uncertainty about whether AGI will be good or bad, it's worth doing more research into the risks.
If you haven't read Bostrom's book yet, I'd really recommend it. http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
If you haven't read Bostrom's book yet, I'd really recommend it. http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
I read this book. Got about 4 chapters in before I had to give up at the sheer ridiculousness of the whole thing. The problem with this entire line of reasoning is that at this point it is nothing more an a thought experiment. Many of the key underlying assumptions that are required for Artificial General Intelligence simply have not been realised, and while Weak AI is progressing at a strong rate, we are hardly anywhere close to a place where we should start worrying about all this stuff.
It's about as likely as a meteor hitting the planet and wiping out the entire human life. Possible? Sure. Should I be panicking about this right now? Nah.
The book is nonsense. I'll start paying attention when someone who has real experience in the field of AI research (and I'm not talking charlatans like Yudkowsky here), but someone like say Norvig come out and say it's a reasonable concern today.
It's about as likely as a meteor hitting the planet and wiping out the entire human life. Possible? Sure. Should I be panicking about this right now? Nah.
The book is nonsense. I'll start paying attention when someone who has real experience in the field of AI research (and I'm not talking charlatans like Yudkowsky here), but someone like say Norvig come out and say it's a reasonable concern today.
The expert consensus says 10% chance of human-level AI in 10 years: http://www.givewell.org/labs/causes/ai-risk/ai-timelines
Many computer science professors have publicly said they think AI poses significant risks. There's a list here: http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-ri...
Also see this open letter, signed by hundreds of experts: http://futureoflife.org/ai-open-letter/
Many computer science professors have publicly said they think AI poses significant risks. There's a list here: http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-ri...
Also see this open letter, signed by hundreds of experts: http://futureoflife.org/ai-open-letter/
Your last link, the open letter, says nothing about human or > human level AI. Just "robust and beneficial usage" of AI. In all likelihood that means the current AI technology and the letter is aimed at (I'm assuming) people who are trying to use these techniques in things such as modern weapon systems. While that is of course a concern, it's not the same as the concern for a Skynet like scenario.
Some experts in the second link you gave are concerned, sure. But I can probably find an equal number who dismiss it as well. There isn't a clear consensus over AGI. I still remain skeptical. Same with your first link, which tries to "forecast" AGI. People can't forecast next month's weather correctly, so forgive me for not believing in a 10% chance in 10 years.
Actually, I take back my appeal to authority argument in its entirety, because I just remembered the first thing I saw in my AI class was a video of experts claiming the exact same thing. The video was from the 50s.
EDIT: Found it: https://www.youtube.com/watch?v=rlBjhD1oGQg
Some experts in the second link you gave are concerned, sure. But I can probably find an equal number who dismiss it as well. There isn't a clear consensus over AGI. I still remain skeptical. Same with your first link, which tries to "forecast" AGI. People can't forecast next month's weather correctly, so forgive me for not believing in a 10% chance in 10 years.
Actually, I take back my appeal to authority argument in its entirety, because I just remembered the first thing I saw in my AI class was a video of experts claiming the exact same thing. The video was from the 50s.
EDIT: Found it: https://www.youtube.com/watch?v=rlBjhD1oGQg
The letter isn't (just) about modern weapon systems. It was put together by this group: http://futureoflife.org/ai-news/
Also, no-one is worried about a skynet scenario. The worrying scenario is just any powerful system that optimises for something that's different from what humans want.
Second, the point is that even uncertainty is enough for action. For AI to not be a problem, you'd need to be very confident that it'll occur a long way in the future, and that there's nothing we can do until it's closer. As you've said, we don't have confidence in the timeline. We have large uncertainty. And that's more reason for action, especially research.
Consider analogously:
"We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."
Seems like a bad argument to me.
Also, no-one is worried about a skynet scenario. The worrying scenario is just any powerful system that optimises for something that's different from what humans want.
Second, the point is that even uncertainty is enough for action. For AI to not be a problem, you'd need to be very confident that it'll occur a long way in the future, and that there's nothing we can do until it's closer. As you've said, we don't have confidence in the timeline. We have large uncertainty. And that's more reason for action, especially research.
Consider analogously:
"We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."
Seems like a bad argument to me.
Except, in the case of climate research, we have a plethora of evidence of the possible harmful effects and we can see it happening today. Everything about the potential harmful effects of AI is pure conjecture, because there is no human-level general intelligence AI system that exists today. It's, once again, something philosophers like Bostrom will make a career out of.
I'm 100% with Torvalds on this when he laughs at the prepostorous notion that AI will become a doomsday scenario. I think it'll become more and more specialised, branch out to other fields and become reasonably good. But there's a huge leap to go from there to HGI.
> any powerful system that optimises for something that's different from what humans want.
Except this notion rests on the premise that humans will not be in full control, which leads to the exponential growth argument which leads back to the Skynet like scenario.
> the point is that even uncertainty is enough for action
And that action is...what exactly? People won't stop building intelligent systems. There is no real path that we have from where we are to HGI, so it's not like researchers have a concrete path. Just what exactly does this research look like?
> "We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."
Extremely Poor analogy. We have decades worth of concrete data that tells us the nature and reality of climate change. We demonstrably know that it's a threat. Can you say the same about AI?
Also, I'll take the time to re-iterate how heavily skeptical I remain of groups like MIRI that are spearheaded by people who don't believe in the scientific method, believe in stuff like cryogenics, have history of trying to profit off of someone else's copyrighted material and have someone managed to convince a whole lot of people that donating them is the best way to fight off the AI doomsday scenario. People should do their research before linking to stuff like that. :(
I'm 100% with Torvalds on this when he laughs at the prepostorous notion that AI will become a doomsday scenario. I think it'll become more and more specialised, branch out to other fields and become reasonably good. But there's a huge leap to go from there to HGI.
> any powerful system that optimises for something that's different from what humans want.
Except this notion rests on the premise that humans will not be in full control, which leads to the exponential growth argument which leads back to the Skynet like scenario.
> the point is that even uncertainty is enough for action
And that action is...what exactly? People won't stop building intelligent systems. There is no real path that we have from where we are to HGI, so it's not like researchers have a concrete path. Just what exactly does this research look like?
> "We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."
Extremely Poor analogy. We have decades worth of concrete data that tells us the nature and reality of climate change. We demonstrably know that it's a threat. Can you say the same about AI?
Also, I'll take the time to re-iterate how heavily skeptical I remain of groups like MIRI that are spearheaded by people who don't believe in the scientific method, believe in stuff like cryogenics, have history of trying to profit off of someone else's copyrighted material and have someone managed to convince a whole lot of people that donating them is the best way to fight off the AI doomsday scenario. People should do their research before linking to stuff like that. :(
I'm not saying we don't know whether climate change poses a tail risk or not (it obviously does). I'm just saying that claiming uncertainty isn't a good reason to avoid action.
In general, if there's a poorly understood but potentially very bad risk, then (a) more research to understand the risk is really high priority (b) if that research doesn't rule out the really bad scenario, we should try to do something to prevent it.
With AI, unfortunately waiting until the evidence that it's harmful is well established is not possible, because then it could be too late.
What AI risk research could involve is laid out in detail in the link.
In general, if there's a poorly understood but potentially very bad risk, then (a) more research to understand the risk is really high priority (b) if that research doesn't rule out the really bad scenario, we should try to do something to prevent it.
With AI, unfortunately waiting until the evidence that it's harmful is well established is not possible, because then it could be too late.
What AI risk research could involve is laid out in detail in the link.
"poorly understood but potentially very bad risk" is something you could say about the risk of an alien invasion.
> And that action is...what exactly?
I would guess A. the development of "known safe" high-level primitives, and B. a coupling of education in the craft of AI with education in the engineering of AI.
The "profession" of developing strong, general-purpose AI should basically look like a cross between cryptography and regular old capital-e-Engineering: like crypto, you'd constantly hear important things about "not rolling your own" low-level AI algorithms; and, like Engineering, the point of the job would be making sure the thing you're building is constructed so that it doesn't "exceed tolerances."
The research goals in the field of "friendly AI" are thus twofold:
• work in understanding possible computational models for minds, to understand what sort of tolerances there can be—what knobs and dials and initial conditions each type of mind comes with;
• and work in development of safe high-level primitives for constructing minds.
Both of these are the subjects of active papers. Note that these can both also be described as plain-old "AI research"; they're not just abstract philosophy, these papers are steps toward something people can build. The research within the subfield of "friendly" AI just has different criteria for what makes for a "promising avenue of research", e.g. ignoring black-box solutions because there are no knobs for humans to tweak.
> MIRI ... spearheaded by
MIRI is two things, and it's best to keep them mentally separate. It's a research organization, like Bell Labs—and it's a nonprofit foundation that funnels money into that research organization.
Yudkowsky is the director (head cheerleader) of the nonprofit. He doesn't really touch the research organization. The research org will succeed or fail on its merits (mostly whether it hires good researchers), but the leadership of the nonprofit has not-much to do with that success or failure, any more than AT&T had any impact on the success or failure of Bell Labs. You can believe in the research org known as MIRI even if you actively distrust Yudkowsky.
I would guess A. the development of "known safe" high-level primitives, and B. a coupling of education in the craft of AI with education in the engineering of AI.
The "profession" of developing strong, general-purpose AI should basically look like a cross between cryptography and regular old capital-e-Engineering: like crypto, you'd constantly hear important things about "not rolling your own" low-level AI algorithms; and, like Engineering, the point of the job would be making sure the thing you're building is constructed so that it doesn't "exceed tolerances."
The research goals in the field of "friendly AI" are thus twofold:
• work in understanding possible computational models for minds, to understand what sort of tolerances there can be—what knobs and dials and initial conditions each type of mind comes with;
• and work in development of safe high-level primitives for constructing minds.
Both of these are the subjects of active papers. Note that these can both also be described as plain-old "AI research"; they're not just abstract philosophy, these papers are steps toward something people can build. The research within the subfield of "friendly" AI just has different criteria for what makes for a "promising avenue of research", e.g. ignoring black-box solutions because there are no knobs for humans to tweak.
> MIRI ... spearheaded by
MIRI is two things, and it's best to keep them mentally separate. It's a research organization, like Bell Labs—and it's a nonprofit foundation that funnels money into that research organization.
Yudkowsky is the director (head cheerleader) of the nonprofit. He doesn't really touch the research organization. The research org will succeed or fail on its merits (mostly whether it hires good researchers), but the leadership of the nonprofit has not-much to do with that success or failure, any more than AT&T had any impact on the success or failure of Bell Labs. You can believe in the research org known as MIRI even if you actively distrust Yudkowsky.
"have history of trying to profit off of someone else's copyrighted material"
That's quite a strong claim - could you provide evidence for it? Are you referring to HPMOR? HPMOR has always been given away for free, both online and in print, and J.K. Rowling has explicitly allowed non-commercial Harry Potter fanfics.
That's quite a strong claim - could you provide evidence for it? Are you referring to HPMOR? HPMOR has always been given away for free, both online and in print, and J.K. Rowling has explicitly allowed non-commercial Harry Potter fanfics.
He was eventually dissuaded because a vocal majority of HP fans were against it (because JKR has only allowed non-commercial use). But here is the original announcement I found from a forum -
The Singularity Institute for Artificial Intelligence, the nonprofit I work at, is currently running a Summer Challenge to tide us over until the Singularity Summit in October (Oct 15-16 in New York, ticket prices go up by $100 after September starts). The Summer Challenge grant will double up to $125,000 in donations, ends at the end of August, and is currently up to only $39,000 which is somewhat worrying. I hadn't meant to do anything like this, but:
I will release completed chapters at a pace of one every 6 days, or one every 5 days after the SIAI's Summer Challenge reaches $50,000, or one every 4 days after the Summer Challenge reaches $75,000, or one every 3 days if the Summer Challenge is completed. Remember, the Summer Challenge has until the end of August, after that the pace will be set. (Just some slight encouragement for donors reading this fic to get around to donating sooner rather than later.) A link to the Challenge and the Summit can be found in the profile page, or Google "summer singularity challenge" and "Singularity Summit" respectively.
The Singularity Institute for Artificial Intelligence, the nonprofit I work at, is currently running a Summer Challenge to tide us over until the Singularity Summit in October (Oct 15-16 in New York, ticket prices go up by $100 after September starts). The Summer Challenge grant will double up to $125,000 in donations, ends at the end of August, and is currently up to only $39,000 which is somewhat worrying. I hadn't meant to do anything like this, but:
I will release completed chapters at a pace of one every 6 days, or one every 5 days after the SIAI's Summer Challenge reaches $50,000, or one every 4 days after the Summer Challenge reaches $75,000, or one every 3 days if the Summer Challenge is completed. Remember, the Summer Challenge has until the end of August, after that the pace will be set. (Just some slight encouragement for donors reading this fic to get around to donating sooner rather than later.) A link to the Challenge and the Summit can be found in the profile page, or Google "summer singularity challenge" and "Singularity Summit" respectively.
The worrying scenario is just any powerful system that optimizes for something that's different from what humans want.
Like AI based-HFT? Which humans want what? I'm pretty sure, that if there's one constant, that there always will be some humans on either side of the argument.
Like AI based-HFT? Which humans want what? I'm pretty sure, that if there's one constant, that there always will be some humans on either side of the argument.
Also, no-one is worried about a skynet scenario. The worrying scenario is just any powerful system that optimises for something that's different from what humans want.
So Transcendence in leu of The Terminator. It's still hollywood fiction.
So Transcendence in leu of The Terminator. It's still hollywood fiction.
Global warming will not happen because The Day After Tomorrow is a movie. 12 Monkeys is a movie therefore weaponized biotechnology is without risk. "Hollywood made a movie vaguely like this" isn't much of an argument against anything.
Don't straw man me, bro.
My point is that the entire idea of an buggy AI destroying humanity is absolutely divorced from the reality of AI research. It's not unimaginable to the lay-person, but there is simply no way of getting there from the neural networks we're making now.
To begin, can you go ahead and explain what self-awareness means in terms of a neural network you are training?
My point is that the entire idea of an buggy AI destroying humanity is absolutely divorced from the reality of AI research. It's not unimaginable to the lay-person, but there is simply no way of getting there from the neural networks we're making now.
To begin, can you go ahead and explain what self-awareness means in terms of a neural network you are training?
[deleted]
[deleted]
> we are hardly anywhere close to a place where we should start worrying about all this stuff
Here's a comparison: wouldn't it be great if we had started thinking about climate change way back at the beginning of the industrial revolution, before we decided to create tons of open-air coal plants?
There are reasons to solve problems before they're problems.
Here's a comparison: wouldn't it be great if we had started thinking about climate change way back at the beginning of the industrial revolution, before we decided to create tons of open-air coal plants?
There are reasons to solve problems before they're problems.
While I agree that there should most definitely be general research in the area, and it is of course something worth exploring, what I don't agree with is the hype that is being built around this. Hollywood has played its part of course, but people who should really know better seem to be under the impression that as soon as we achieve AGI (which is a big /if/), we're doomed immediately. They ignore that all this is still academic and while industry practices follow sooner than is usual these days, it's not like one day a scientist creates AI and the next day, Skynet launches Nuclear Missiles.
So is it worth exploring? Sure. But I'm not going to be concerned about something for which there is zero evidence to support it.
There are no reasons to solve problems before there is foolproof evidence that they are indeed problems.
So is it worth exploring? Sure. But I'm not going to be concerned about something for which there is zero evidence to support it.
There are no reasons to solve problems before there is foolproof evidence that they are indeed problems.
[deleted]
Two examples outside of science fiction:
- The horse population declined dramatically after their work abilities were exceeded on every dimension by machines
- Homo Neanderthalensis disappeared after the appearance of Homo Sapiens.
I'm not saying these are perfect analogies. But you are saying they're completely irrelevant. Why is that?
- The horse population declined dramatically after their work abilities were exceeded on every dimension by machines
- Homo Neanderthalensis disappeared after the appearance of Homo Sapiens.
I'm not saying these are perfect analogies. But you are saying they're completely irrelevant. Why is that?
Good old Homo Sapiens is already disappearing thanks to Homo Oeconomicus.
I don't see a problem if we move on to Homo faber next.
How are you so confident that the risks of machine learning are in the realm of science fiction? Machine learning, at least, has the plausible potential to permanently reduce demand on labor by capital owners. Undemanded laborers lose their ability to negotiate or strike as a power counterbalance to capital owners.
At the very least, that future is too complicated to form such a confident opinion that the risks of machine learning are in the realm of science fiction.
At the very least, that future is too complicated to form such a confident opinion that the risks of machine learning are in the realm of science fiction.
There are certainly changes happening (and more on the way!) as a result of a combination of Moore's Law and ML research. I don't think they are inherently significantly more risky any more than other technical advances as long as we have a chance to think about and understand them. We definitely need to think through the social implications of computers.
I think it's important to separate those real developments from the movie-plot-threat of a super-AI escaping human control.
I think it's important to separate those real developments from the movie-plot-threat of a super-AI escaping human control.
>It simply does not outside of the realm of science fiction.
Why?
Why?
In short, because human intelligence evolved in a different environment than AI. We are the results of the need to survive and reproduce, and the majority of our "intelligence" is based on this foundation. Algorithms have no such darwinian pressures, and there is no point in developing AI replacements for them, since humans abound.
You could make basically the same argument for why humans will never invent heavier-than-air flight - because avian flight 'evolved based on a need to survive and reproduce', and 'artificial flight doesn't have darwinian pressures'.
But humans did make artificial flying machines.
I don't mean to be unkind, but your argument doesn't have close to the level of certainly that we should bet our species on.
No one knows where the roadblocks to AGI are, or what the timelines might be; but there is what seems like huge progress happening recently in an area which might eventually lead there. While no one knows the path, many intelligent people have thought about this without finding any theoretical roadblocks. Please don't publicly dismiss the concerns as 'science fiction' without a little more thought.
But humans did make artificial flying machines.
I don't mean to be unkind, but your argument doesn't have close to the level of certainly that we should bet our species on.
No one knows where the roadblocks to AGI are, or what the timelines might be; but there is what seems like huge progress happening recently in an area which might eventually lead there. While no one knows the path, many intelligent people have thought about this without finding any theoretical roadblocks. Please don't publicly dismiss the concerns as 'science fiction' without a little more thought.
You could make basically the same argument for why humans will never invent heavier-than-air flight
You're not getting it.
The effective argument of the danger seer is akin to "humans will learn to fly but then our cities will be covered with feathers" -
Humans have incentive to produce flexible, learning, language-understanding machines. Humans don't an incentive to produce a thing that will suddenly go rogue and decide to kill us.
It is entirely reasonable to argue that the tendency of humans themselves to go rogue and decide kill other humans comes from the evolutionary processes that produced humans rather than from the quality of human intelligence. We can see other relatively intelligent animals that live with in more (and in less) harmony than humans. No reason intelligence implies entity X won't fully cooperate with the people who intentionally engineered it.
I would argue that we human imagine that AGIs would have the same "downsides" as ourselves simply because we don't have any conception of intelligent things besides ourselves.
Maybe a carefully human-engineered intelligence would for reasons unknown turn into the things we see on movies. Maybe the next generation of supercolider will create an subatomic chain reaction that destroys the earth. But it's premature to worry about these particular hypothetical, especially given humans with ordinary AIs do pose problems.
You're not getting it.
The effective argument of the danger seer is akin to "humans will learn to fly but then our cities will be covered with feathers" -
Humans have incentive to produce flexible, learning, language-understanding machines. Humans don't an incentive to produce a thing that will suddenly go rogue and decide to kill us.
It is entirely reasonable to argue that the tendency of humans themselves to go rogue and decide kill other humans comes from the evolutionary processes that produced humans rather than from the quality of human intelligence. We can see other relatively intelligent animals that live with in more (and in less) harmony than humans. No reason intelligence implies entity X won't fully cooperate with the people who intentionally engineered it.
I would argue that we human imagine that AGIs would have the same "downsides" as ourselves simply because we don't have any conception of intelligent things besides ourselves.
Maybe a carefully human-engineered intelligence would for reasons unknown turn into the things we see on movies. Maybe the next generation of supercolider will create an subatomic chain reaction that destroys the earth. But it's premature to worry about these particular hypothetical, especially given humans with ordinary AIs do pose problems.
IMO you are now making a different argument from what I was replying to above. That's fine, but I just want to be clear.
So: Yes, we might all get along fine; ideally the more intelligent entity would be even more enlightened that we are; etc - those are all valid possibilities, and no one is saying they are implausible.
All people are saying is that some bad possibilities are also plausible (Eliezer Yudkowsky writes some great stuff about all the accidental ways we could screw up trying to program an AGI to be benevolent), and that, as the downside of the bad possibilities is potentially so great, then, in the absence of any argument why they definitely won't happen, we need to think carefully about them.
Some would say that, because the potential consequences of a bad outcome are so great, we need to be able to prove the bad outcomes won't happen before we build this.
So: Yes, we might all get along fine; ideally the more intelligent entity would be even more enlightened that we are; etc - those are all valid possibilities, and no one is saying they are implausible.
All people are saying is that some bad possibilities are also plausible (Eliezer Yudkowsky writes some great stuff about all the accidental ways we could screw up trying to program an AGI to be benevolent), and that, as the downside of the bad possibilities is potentially so great, then, in the absence of any argument why they definitely won't happen, we need to think carefully about them.
Some would say that, because the potential consequences of a bad outcome are so great, we need to be able to prove the bad outcomes won't happen before we build this.
One has to be selective in considering what possibilities are credible here - but that is true about anything.
In my opinion, Eliezer Yudkowsky and those who follow his argument are not intellectually credible and should dismissed with zero consideration.
In looking at this issue, I would note that Yudkowsky's (Bostrone's) ultimate argument is exactly homologous to Pascal's Wager. [1]
IE, since there's a hypothetical small probability that some entity [A Future AGI/Christian God/Flying Spaghetti Monster] could exist and the stakes for ignoring the possibility are "infinite" then we must "rationally" labor now to deal with possibility regardless of not understanding the mechanism involved in that entity.
And the thing is that modern science and modern statistics pretty hinge on ignoring, uh, "guff", demanding that extraordinary claims submit their extraordinary evidence and so-forth.
Further, the other fallacious argument pushed by Yudkowsky et al involves treating an AGI as a wish-granting Genie - human make overt logical requests and the AGIs "twist" these in a world-changing/destroying fashion. The problem is the over-request-format is identical to "logical specification" approach of gofai, the generally carded view that AI can be achieved with only the symbolic specification of how the world works (notably Yudkowsky and friend simultaneously acknowledge AI won't be based on symbolic specification/gofai and ground their argument on AGIs operating on explicit orders only, which boil down to symbolic specification. The Bostrone example is you tell the Genie "make me a thousand paper clips" and it turns the planet into paper clips "just to be sure").
[1] https://en.wikipedia.org/wiki/Pascal's_Wager
[2]https://en.wikipedia.org/wiki/Wish
[3]
In my opinion, Eliezer Yudkowsky and those who follow his argument are not intellectually credible and should dismissed with zero consideration.
In looking at this issue, I would note that Yudkowsky's (Bostrone's) ultimate argument is exactly homologous to Pascal's Wager. [1]
IE, since there's a hypothetical small probability that some entity [A Future AGI/Christian God/Flying Spaghetti Monster] could exist and the stakes for ignoring the possibility are "infinite" then we must "rationally" labor now to deal with possibility regardless of not understanding the mechanism involved in that entity.
And the thing is that modern science and modern statistics pretty hinge on ignoring, uh, "guff", demanding that extraordinary claims submit their extraordinary evidence and so-forth.
Further, the other fallacious argument pushed by Yudkowsky et al involves treating an AGI as a wish-granting Genie - human make overt logical requests and the AGIs "twist" these in a world-changing/destroying fashion. The problem is the over-request-format is identical to "logical specification" approach of gofai, the generally carded view that AI can be achieved with only the symbolic specification of how the world works (notably Yudkowsky and friend simultaneously acknowledge AI won't be based on symbolic specification/gofai and ground their argument on AGIs operating on explicit orders only, which boil down to symbolic specification. The Bostrone example is you tell the Genie "make me a thousand paper clips" and it turns the planet into paper clips "just to be sure").
[1] https://en.wikipedia.org/wiki/Pascal's_Wager
[2]https://en.wikipedia.org/wiki/Wish
[3]
We also don't have an incentive to write programs that grant access to people who know a special incantation like '() { :; };' and yet we do. Point being we can fuck this up very easily, like any other bit of software.
I myself am on fence, on the one hand it is a bit like worrying about over population on mars, and there really is nothing we can productively do right now to research AI safety, the best thing we have are convolutional long/short term memory nets, and there's not much you can say about it.
But at the same time worrying is a great move if the alternative is to just wait until we can do something.
I myself am on fence, on the one hand it is a bit like worrying about over population on mars, and there really is nothing we can productively do right now to research AI safety, the best thing we have are convolutional long/short term memory nets, and there's not much you can say about it.
But at the same time worrying is a great move if the alternative is to just wait until we can do something.
We unintentionally produce programs while fail or produce small variations on their original purpose - give control to X person rather than Y person.
We haven't unintentionally produced a program that had a whole range of unexpected behaviors. An intelligence having human-like "survival instincts" and etc would be a big variation on a tool.
We haven't unintentionally produced a program that had a whole range of unexpected behaviors. An intelligence having human-like "survival instincts" and etc would be a big variation on a tool.
I'm not saying that HGI is impossible, but that the "risk scenarios" are complete fiction. In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior. This will change the questions of risk into something that's tangible and addressable.
On Ada Lovelace's 200 year birthday, we might remember her quote:
"The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform."
She was very smart, but she was wrong about that.
>In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior.
You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things?
If so, I disagree with your assertion: it is not obvious that those things have to be done before a human level AGI could be made.
It might be a good idea to do so, but its not clear that they are a necessary requirement. Some people would say this is why AGI is dangerous.
Maybe there's a path to bootstrapping an AGI, where we build something from relatively simple bits we can understand, that then becomes very much more complex than we can understand.
This happens all the time: for example, we get very complex classification behaviour from deep nets: [simple pieces] + [lots of data] doing something more complex than we have explicitly trained them to.
If we were convinced that "In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior" was a necessary thing to build before we could build a HGI, people would be less worried. But its not; and that's partially why people are worried. In fact, some of the people working on AI risk could be said to be basically racing to understand these things before other folk build an AGI.
>In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior.
You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things?
If so, I disagree with your assertion: it is not obvious that those things have to be done before a human level AGI could be made.
It might be a good idea to do so, but its not clear that they are a necessary requirement. Some people would say this is why AGI is dangerous.
Maybe there's a path to bootstrapping an AGI, where we build something from relatively simple bits we can understand, that then becomes very much more complex than we can understand.
This happens all the time: for example, we get very complex classification behaviour from deep nets: [simple pieces] + [lots of data] doing something more complex than we have explicitly trained them to.
If we were convinced that "In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior" was a necessary thing to build before we could build a HGI, people would be less worried. But its not; and that's partially why people are worried. In fact, some of the people working on AI risk could be said to be basically racing to understand these things before other folk build an AGI.
You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things?
It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.
That may or may not be ethics as such but if something we make on purpose, we'd need to understand it. And considering it's difficulties, making it on purpose seems necessary, contrary to movies etc.
It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.
That may or may not be ethics as such but if something we make on purpose, we'd need to understand it. And considering it's difficulties, making it on purpose seems necessary, contrary to movies etc.
Yes - but no one provided any argument for that.
Instead, andreyf relied on it as a fact. I pointed out we can't assume that.
As such: its reasonable to fear AGI coming before we have that stuff sorted out.
Hence, as such: there is a real risk here, which the safety folks are trying to mitigate; hence they aren't worried about a threat scenario easily dismissable by andreyf's argument.
>It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.
Ie. If you, or anyone had convincingly argued this, then it would be reasonable to dismiss (or at least greatly reduce) the concerns of the safety folk. But no one has, so its not reasonable to.
Instead, andreyf relied on it as a fact. I pointed out we can't assume that.
As such: its reasonable to fear AGI coming before we have that stuff sorted out.
Hence, as such: there is a real risk here, which the safety folks are trying to mitigate; hence they aren't worried about a threat scenario easily dismissable by andreyf's argument.
>It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.
Ie. If you, or anyone had convincingly argued this, then it would be reasonable to dismiss (or at least greatly reduce) the concerns of the safety folk. But no one has, so its not reasonable to.
It is absolutely reasonable to fear how ML may be misused by a human, and those concerns fall mostly in the realm of political theory and economics.
My claim is that I have seen no evidence that it is reasonable to fear a fictional self-aware autonomous AI (what I call HGI). The evidence people point to seems to have a very tenuous relationship with the reality of AI research.
Once we have even an inkling of an idea of how one might approach creating an artificial HGI, we can discuss how we can prevent it from being evil and taking over the world. Until there's actually a proposal for how one might construct one, though, any such discussion is worse than useless.
How can you talk about the risk of something when you literally have no idea what it is you are describing or how it's built, embodied with magical properties imagined by science fiction authors? To me, this makes about as much sense as talking about stopping aliens which could very well exist and could very well wipe out our civilization, i.e. great subject for wrapping up a party or going to the movies, but not exactly material for peer reviewed journals.
My claim is that I have seen no evidence that it is reasonable to fear a fictional self-aware autonomous AI (what I call HGI). The evidence people point to seems to have a very tenuous relationship with the reality of AI research.
Once we have even an inkling of an idea of how one might approach creating an artificial HGI, we can discuss how we can prevent it from being evil and taking over the world. Until there's actually a proposal for how one might construct one, though, any such discussion is worse than useless.
How can you talk about the risk of something when you literally have no idea what it is you are describing or how it's built, embodied with magical properties imagined by science fiction authors? To me, this makes about as much sense as talking about stopping aliens which could very well exist and could very well wipe out our civilization, i.e. great subject for wrapping up a party or going to the movies, but not exactly material for peer reviewed journals.
Me: It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.
Feral: Yes - but no one provided any argument for that.
Me: I should rephrase, I sort-of do above but still, negation of "it requires understanding" is "AGI could happen without understanding, at random, in some fashion".
And I'd say that is the position that's no has provided to support for - again, akin to the worries that bring a giant supercollider online could destroy the earth. I mean, given that an AGI is an unknown, maybe bringing the supercollider online, another unknown, could create an AGI, which would then destroy the earth!
Plus, human beings have generally not succeed in producing complex engineering achievements by accidents. The atomic bomb required a massive engineering effort, heavier-than-air flight required considerable effort, alchemists never synthesized CPUs by baking minerals at random, even largest pieces of modern software have not threatened to "achieve consciousness" even as they malfunction repeatedly.
My argument is "consciousness by accident" is the extraordinary claim which requires extraordinary evidence. And AndreyF also adds - the dangers of ill-intentioned humans using AIs is here today, why worry about the hypothetical "getting out of control" problem when you humans who have historically inflicted vast amounts of misery on others of their species - oh, and consider several madmen fighting each other, that couldn't happen, not in Syria or whatever place one might name.
Feral: Yes - but no one provided any argument for that.
Me: I should rephrase, I sort-of do above but still, negation of "it requires understanding" is "AGI could happen without understanding, at random, in some fashion".
And I'd say that is the position that's no has provided to support for - again, akin to the worries that bring a giant supercollider online could destroy the earth. I mean, given that an AGI is an unknown, maybe bringing the supercollider online, another unknown, could create an AGI, which would then destroy the earth!
Plus, human beings have generally not succeed in producing complex engineering achievements by accidents. The atomic bomb required a massive engineering effort, heavier-than-air flight required considerable effort, alchemists never synthesized CPUs by baking minerals at random, even largest pieces of modern software have not threatened to "achieve consciousness" even as they malfunction repeatedly.
My argument is "consciousness by accident" is the extraordinary claim which requires extraordinary evidence. And AndreyF also adds - the dangers of ill-intentioned humans using AIs is here today, why worry about the hypothetical "getting out of control" problem when you humans who have historically inflicted vast amounts of misery on others of their species - oh, and consider several madmen fighting each other, that couldn't happen, not in Syria or whatever place one might name.
I thought much the same once. This paper convinced me otherwise: https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...
I'd be interested in your take on it.
I'd be interested in your take on it.
Only skimmed it, but seems like my thoughts would be similar to manish_gill's above [1]: the assumptions are rooted in pop science fantasy.
More specifically, there are an infinite number of reasons for and against doing anything. Once we understand how humans weigh these reasons and choose between them and model them in an HGI system, we will be able to give it the values and morals that its creators choose.
1. https://news.ycombinator.com/item?id=10724108
More specifically, there are an infinite number of reasons for and against doing anything. Once we understand how humans weigh these reasons and choose between them and model them in an HGI system, we will be able to give it the values and morals that its creators choose.
1. https://news.ycombinator.com/item?id=10724108
Stuart Russell (co-author of AI:MA, one of MIRI's research advisors) argues on http://edge.org/conversation/the-myth-of-ai#26015 that AI systems with "the ability to make high-quality decisions" (where "quality refers to the expected outcome utility of actions taken" and the utility function is represented in the system's programmed decision criteria) raises two problems:
"1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
"2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task."
The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concern...). The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives."
The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free.
The second of these also has some intuitive force; https://intelligence.org/?p=12234 shows Omohundro's idea can be stated formally, so it's not purely sci-fi. Averting the "Sorcerer's Apprentice" problem in full generality would mean averting this problem, since we'd then simply be able to give AI systems the right goals and let them go wild. Absent that, if AI systems become much more cognitively capable than humans, we'll probably need to actively work on some approach that violates Omohundro's assumptions (and the assumptions of the formalism above). Bostrom and MIRI both talk about a lot of interesting ideas along these lines.
"1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
"2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task."
The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concern...). The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives."
The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free.
The second of these also has some intuitive force; https://intelligence.org/?p=12234 shows Omohundro's idea can be stated formally, so it's not purely sci-fi. Averting the "Sorcerer's Apprentice" problem in full generality would mean averting this problem, since we'd then simply be able to give AI systems the right goals and let them go wild. Absent that, if AI systems become much more cognitively capable than humans, we'll probably need to actively work on some approach that violates Omohundro's assumptions (and the assumptions of the formalism above). Bostrom and MIRI both talk about a lot of interesting ideas along these lines.
What is "an AI system with the ability to make high-quality decisions"? Do automated derivative trading models count? Do systems which decide how much to bid on a RTB ad exchange count?
The first problem is not new. We have a similar problem with some corporations, for example.
"A sufficiently capable intelligent system" is as real as "sufficiently hostile aliens". It's hard to argue and reason about a fictional system with a assortment of properties picked by someone aiming to spreading fear.
The first problem is not new. We have a similar problem with some corporations, for example.
"A sufficiently capable intelligent system" is as real as "sufficiently hostile aliens". It's hard to argue and reason about a fictional system with a assortment of properties picked by someone aiming to spreading fear.
The problems aren't completely unprecedented (else we'd have basically no knowledge about them), but they become more severe in the scenarios Bostrom/Russell/etc. are talking about.
I would say that the central concern is with notional systems that can form detailed, accurate models of the world and efficiently search through the space of policies that can be expected to produce a given outcome according to the model. This can be a recommender system that tells other agents what policies to adopt, or it can execute the policies itself.
If the search process through policies is sufficiently counter-intuitive and opaque to operator inspection, the "Sorcerer's Apprentice" problem becomes much more severe than it is in ordinary software. As the system becomes more capable, it can look increasingly safe and useful in its current context and yet remain brittle in the face of changes to itself and its environment. This is also where convergent instrumental goals become more concerning, because systems with imperfectly understood/designed policy selection criteria (introducing an element of randomness, from our perspective) seem likely to converge on adversarial policies due to the general fact of resource limitations.
There's no reason to think this kind of system is inevitable, but it's worth investigating how likely we are to be able to develop superhuman planning/decision agents, on what timescale, and whether there are any actions we could take in advance to make it possible to use such systems safely. At this point not enough research-hours have gone into this topic to justify any strong conclusions about whether we can (or can't) make much progress today.
http://givewell.org/labs/causes/ai-risk gives a good summary of this topic.
I would say that the central concern is with notional systems that can form detailed, accurate models of the world and efficiently search through the space of policies that can be expected to produce a given outcome according to the model. This can be a recommender system that tells other agents what policies to adopt, or it can execute the policies itself.
If the search process through policies is sufficiently counter-intuitive and opaque to operator inspection, the "Sorcerer's Apprentice" problem becomes much more severe than it is in ordinary software. As the system becomes more capable, it can look increasingly safe and useful in its current context and yet remain brittle in the face of changes to itself and its environment. This is also where convergent instrumental goals become more concerning, because systems with imperfectly understood/designed policy selection criteria (introducing an element of randomness, from our perspective) seem likely to converge on adversarial policies due to the general fact of resource limitations.
There's no reason to think this kind of system is inevitable, but it's worth investigating how likely we are to be able to develop superhuman planning/decision agents, on what timescale, and whether there are any actions we could take in advance to make it possible to use such systems safely. At this point not enough research-hours have gone into this topic to justify any strong conclusions about whether we can (or can't) make much progress today.
http://givewell.org/labs/causes/ai-risk gives a good summary of this topic.
Human level AI would have a profound impact on the value of human labor. If you don't consider everyone being unemployable an issue to society then I am not sure what would pose a significant risk.
Most people that worry about this topic don't worry about human level AI, it is the vastly greater than human level AI that is the concern.
Most people that worry about this topic don't worry about human level AI, it is the vastly greater than human level AI that is the concern.
I think ML will first make (some) people significantly more productive, but yes, the economic shift in productivity we're starting to see now is very much worth thinking and talking about.
How does human level AI make everyone unemployable?
If the AI is at human level then it can do all human jobs. Given that humans require a minimum income to survive, if the total per hour cost of AI is below this level then all jobs will be taken by AI (why employ an expensive human when you can get a cheaper AI). This is what happened to human computers (people, mostly women, who performed mathematical calculations) in the 1950s when electronic computers became cheaper and faster per calculation. All those jobs ceased to exist.
There are some caveats here. The cost of AI labor needs to be below that of the human minimum. Given there is no effective limit to how much AI labor you can create, the likely result is the cost of capital will fall to near zero making AI labor near free. No human will be able to compete against this except in providing personal services to the owners of capital in fluff jobs.
The more interesting thing is that with the cost of capital and labour at near zero we should get massive deflation to the point that it will be very cheap to provide all people with a very high basic income that will make the need for employment unnecessary. This would be a great world to live in, but it would have a profound effect on the society unlike anything since the rise of agriculture.
There are some caveats here. The cost of AI labor needs to be below that of the human minimum. Given there is no effective limit to how much AI labor you can create, the likely result is the cost of capital will fall to near zero making AI labor near free. No human will be able to compete against this except in providing personal services to the owners of capital in fluff jobs.
The more interesting thing is that with the cost of capital and labour at near zero we should get massive deflation to the point that it will be very cheap to provide all people with a very high basic income that will make the need for employment unnecessary. This would be a great world to live in, but it would have a profound effect on the society unlike anything since the rise of agriculture.
"If the AI is at human level then it can do all human jobs."
I don't see how that follows. How does a computer program do manual labor?
I don't see how that follows. How does a computer program do manual labor?
By having access to a robot.
Actually most manual labor is done by machines controlled to a greater or less extent by a human brain. AI just removes the human brain out of the production loop.
Actually most manual labor is done by machines controlled to a greater or less extent by a human brain. AI just removes the human brain out of the production loop.
So we're going to need robots to replace all the humans. I'm not sure how that fits in the zero marginal cost narrative.
I'm not sure I really understand the economics either.
Is AI going to be free? Who are the owners of capital going to sell their production to if people have no income? What are people who don't own machines going to do? Curl up in a ball and let themselves starve to death?
I'm not sure I really understand the economics either.
Is AI going to be free? Who are the owners of capital going to sell their production to if people have no income? What are people who don't own machines going to do? Curl up in a ball and let themselves starve to death?
It is not that we need to replace all the humans, but that the owners of capital and land will be able to replace all the humans.
The best way to think about production is that everything produced is a combination of four things: labor, capital (past production), energy, and matter (land and raw materials). If you have labor that can be used to build itself (AI) then you can expand labour and capital to a very large number. You can also build new sources of energy very cheaply so the only limiting resource is matter. If you have the energy, capital and labor you can recycle most raw materials very cheaply. This leaves land as the only factor in production that does not scale.
What would all this mean in practice. Physical objects and services would become very cheap to make (effectively free) and all wealth would be tied to the ownership of land. Given the state owns most land, it should be very cheap for the state to provide all the material objects and services that any citizen needs. Unless the state actively wants to stave to death its citizens then there is not need for anyone to go hungry - if the state does want to stave to death people it does not need AI.
The best way to think about production is that everything produced is a combination of four things: labor, capital (past production), energy, and matter (land and raw materials). If you have labor that can be used to build itself (AI) then you can expand labour and capital to a very large number. You can also build new sources of energy very cheaply so the only limiting resource is matter. If you have the energy, capital and labor you can recycle most raw materials very cheaply. This leaves land as the only factor in production that does not scale.
What would all this mean in practice. Physical objects and services would become very cheap to make (effectively free) and all wealth would be tied to the ownership of land. Given the state owns most land, it should be very cheap for the state to provide all the material objects and services that any citizen needs. Unless the state actively wants to stave to death its citizens then there is not need for anyone to go hungry - if the state does want to stave to death people it does not need AI.
I think what you're saying makes some amount of sense, but I still feel there's a lot of handwaving and hidden assumptions around the way such a system gets bootstrapped.
If you have links to more detailed ressources on that topic, I'd be interested in reading them.
If you have links to more detailed ressources on that topic, I'd be interested in reading them.
The way the system gets bootstrapped is humans create the first human level AI. The definition of a human level AI is it is able to do all the activities a human can do and so it can also make human level AIs too.
Once you have one human level AI you can make more replacing each human in the production of the human level AI - you basically start at the end point and working backwards replacing each job performed by a human with an AI. Once you get to the final job you will have a production system that can build AIs without any human labor. The system can make as many extra AI’s as wanted without any human labor - it is detached from the human labor market.
Unfortunately I don’t have any links discussing this. I think not too many people dwell on human level AI’s because the expectation is with AI’s if they can get to human level they will not stay at human level. Once you have AI’s that can make AI’s then they can make better AI’s until they far surpass human level. Understanding what these AI’s will be like is impossible for us.
Once you have one human level AI you can make more replacing each human in the production of the human level AI - you basically start at the end point and working backwards replacing each job performed by a human with an AI. Once you get to the final job you will have a production system that can build AIs without any human labor. The system can make as many extra AI’s as wanted without any human labor - it is detached from the human labor market.
Unfortunately I don’t have any links discussing this. I think not too many people dwell on human level AI’s because the expectation is with AI’s if they can get to human level they will not stay at human level. Once you have AI’s that can make AI’s then they can make better AI’s until they far surpass human level. Understanding what these AI’s will be like is impossible for us.
> Once you have one human level AI you can make more replacing each human in the production of the human level AI - you basically start at the end point and working backwards replacing each job performed by a human with an AI.
Again, I don't see how that follows. Once you know how to build something, you can do it again. Of course. But what if building a functioning AI that can replace a human costs $100 bn?
Certainly it's not going to be economically viable to replace tasks that do not produce high ROI, even if you assume AIs are paired with indestructible robot bodies made out of unobtainium that are as versatile as the human body (have we even built one of those ever?).
I'm familiar with the idea that AIs could make better AIs and lead to some intelligence singularity, etc. Again, I think there's a lot of handwaving behind those ideas. We have no idea what the complexity of building more complex intelligence is, or what it even means.
Again, I don't see how that follows. Once you know how to build something, you can do it again. Of course. But what if building a functioning AI that can replace a human costs $100 bn?
Certainly it's not going to be economically viable to replace tasks that do not produce high ROI, even if you assume AIs are paired with indestructible robot bodies made out of unobtainium that are as versatile as the human body (have we even built one of those ever?).
I'm familiar with the idea that AIs could make better AIs and lead to some intelligence singularity, etc. Again, I think there's a lot of handwaving behind those ideas. We have no idea what the complexity of building more complex intelligence is, or what it even means.
I'm not sure that because you envision a different, more plausible scenario, that completely invalidates the original claim, which was only simply denied before sketching out the alternative.
* I don't see how "Understanding HGI" prevents a "human-level artificial intelligence" from doing anything it pleases.
* I don't see how "Understanding HGI" prevents a "human-level artificial intelligence" from doing anything it pleases.
My second claim is not there to invalidate the first, but rather to point out that there are two "AI problems", and that one is a lot closer to science fiction than the other.
I've expounded on the denial: we will need to first figure out what motivates humans before we can replicate their levels of intelligence. So in order to solve HGI, we will need to understand HGI, which means we will be able to control HGI. We will understand personalities and values in a systematic way, and we will be able to choose the personalities and values of any one instance of HGI.
I've expounded on the denial: we will need to first figure out what motivates humans before we can replicate their levels of intelligence. So in order to solve HGI, we will need to understand HGI, which means we will be able to control HGI. We will understand personalities and values in a systematic way, and we will be able to choose the personalities and values of any one instance of HGI.
> we will need to first figure out what motivates humans before we can replicate their levels of intelligence
I would not assume this with certainty.
There are many systems in nature that consist of simple pieces, when composed, yield enormous complexity. Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives. We might get lucky and discover the means to generate intelligences without understanding them.
As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic...
I would not assume this with certainty.
There are many systems in nature that consist of simple pieces, when composed, yield enormous complexity. Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives. We might get lucky and discover the means to generate intelligences without understanding them.
As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic...
>Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives. We might get lucky and discover the means to generate intelligences without understanding them.
On that premise, the risk of accidentally discovering the right bit-string that morphs into a all powerful evil AI seems to be minuscule. Since at least human intelligence consists of multiple dimensions/factors and only some combinations thereof can be described as "intelligent", given a certain environment. Accidentally discovering something "thinkable": maybe, brute forcing the right combination for human-surpassing intelligence? Most likely not.
On that premise, the risk of accidentally discovering the right bit-string that morphs into a all powerful evil AI seems to be minuscule. Since at least human intelligence consists of multiple dimensions/factors and only some combinations thereof can be described as "intelligent", given a certain environment. Accidentally discovering something "thinkable": maybe, brute forcing the right combination for human-surpassing intelligence? Most likely not.
You make it sound like strong AI will be programmed with ALGOL 68 control structures and loops, hand-coded (i.e. the right bit-string). If that is your conception in I agree with you. However this sounds much too like the "expert systems" of the eighties.
Biology doesn't have any need for "the right bit-string" (modulo DNA, different purpose) to make intelligence work, and one shouldn't have such an expectation of an emergent strong AI, either. Instead of "the right bit-string", think more "the right model"
Biology doesn't have any need for "the right bit-string" (modulo DNA, different purpose) to make intelligence work, and one shouldn't have such an expectation of an emergent strong AI, either. Instead of "the right bit-string", think more "the right model"
I didn't particularly think of ALGOL with the bit-string, but more of the parameter space of the mentioned black-box model. Which I'd assume, coding for a black-box, is quite huge, with lots of local maxima that could appear 'intelligent'. I was referring to discovering the global maxima in that parameter space more or less by chance, as highly unlikely.
I agree with the problem being more one of finding the 'right model', as I'd hope that this will reduce the parameter space greatly. That's why I agree with gggp's post that we'd need to know more about human intelligence, thus also human motivation.
I agree with the problem being more one of finding the 'right model', as I'd hope that this will reduce the parameter space greatly. That's why I agree with gggp's post that we'd need to know more about human intelligence, thus also human motivation.
"Anyone who expects a source of power from the transformation of these atoms is talking moonshine." - Rutherford & "It's like shooting birds in the dark in a country where there are only a few birds," - Albert Einstein - The Pittsburgh Press - Dec 28, 1934 - On the idea that man might some day utilize the atom's energy. Einstein was essentially making the same "parameter space" argument you're making.
There will have to be more true breakthroughs to get to AI, yes; I do agree though that it will take a "long" time, however before this happens.
There will have to be more true breakthroughs to get to AI, yes; I do agree though that it will take a "long" time, however before this happens.
Look at where we are now with ML, and ask what we would need to do to "accidentally" create HGI.
That title's wording is clickbait, not an argument. Mathematicians know exactly why neural networks work – because they contain a model of the data, and they try to develop ways to build (train) those models so that they do it better. The model may not have human concepts associated with it, but that doesn't mean it's somehow "out of control".
In order to achieve HGI, one would have to train models of quite a few domains (the same way schools create models in people's minds), as well as tons of ethical and practical heuristics to weigh an infinite number of reasons for or against doing something.
That title's wording is clickbait, not an argument. Mathematicians know exactly why neural networks work – because they contain a model of the data, and they try to develop ways to build (train) those models so that they do it better. The model may not have human concepts associated with it, but that doesn't mean it's somehow "out of control".
In order to achieve HGI, one would have to train models of quite a few domains (the same way schools create models in people's minds), as well as tons of ethical and practical heuristics to weigh an infinite number of reasons for or against doing something.
> Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives.
I strongly doubt that.
>As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic....
If you read the HN comments on that very article, I recall posting some links to papers giving theoretical grounding for why deep learning works so well.
I strongly doubt that.
>As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic....
If you read the HN comments on that very article, I recall posting some links to papers giving theoretical grounding for why deep learning works so well.
This article is meant to be illustrative, its particulars are not the crux of my main point (to which you seem to imply only doubt sufficies to refute)
Well no. What I think refutes your point is that we possess increasingly good foundational theories for what minds are and how they work. I doubt that any intelligence we successfully create is going to be a black-box.
"It simply does not outside of the realm of science fiction."
If you could prove that, you'd put a lot of minds at ease. By "prove" I mean not necessarily a full mathematical proof, but certainly more than a few paragraphs of English text.
I'm skeptical myself, seems like the more we learn about biology the more it is doing that we did not realize, but skepticism is at best the first step to proof.
If you could prove that, you'd put a lot of minds at ease. By "prove" I mean not necessarily a full mathematical proof, but certainly more than a few paragraphs of English text.
I'm skeptical myself, seems like the more we learn about biology the more it is doing that we did not realize, but skepticism is at best the first step to proof.
I think it will be obvious once HGI is sufficiently understood, in a way that will not need more than a paragraph, if that.
The fear and idea of somehow creating an HGI without understanding how to control it came from The Terminator. It's pure fantasy.
The fear and idea of somehow creating an HGI without understanding how to control it came from The Terminator. It's pure fantasy.
That doesn't even rise to the level of proof-by-disbelief; it's proof by hypothetical future disbelief. You've given no more reason to be confident in your hypothetical than the concerns about hypothetical self-improving AIs.
You're failing to sell this argument to someone generally inclined to agree with you about the outcome.
You're failing to sell this argument to someone generally inclined to agree with you about the outcome.
What is proof-by-disbelief? I'm not trying to prove that HGI is impossible, nor do I think it's impossible.
However, until one proposes a concrete methodology for modeling the world which includes the method that is doing the modeling (self-awareness), a decision making process, and a way to take action autonomously and without intervention, it's very hard to talk generally about preventing catastrophic bugs.
However, until one proposes a concrete methodology for modeling the world which includes the method that is doing the modeling (self-awareness), a decision making process, and a way to take action autonomously and without intervention, it's very hard to talk generally about preventing catastrophic bugs.
Just because we will understand, in principle, how to control it, doesn't mean we'll engineer that control particularly well.
the problem probably is human-level intelligence is still stupid, so it's a risk either way
I guess. A lot of people already did.
http://lukemuehlhauser.com/if-youre-an-ai-safety-lurker-now-...