Among the A.I. doomsayers(newyorker.com)
newyorker.com
Among the A.I. doomsayers
https://www.newyorker.com/magazine/2024/03/18/among-the-ai-doomsayers
372 comments
https://web.archive.org/web/20240312002413/https://www.newyo...
Always all-or-nothing thinking from these folks. Like what they are working on can never be just another boring thing that nerds find entertaining. No, it has to be "world-changing". Gonna "change the world" (for the better or the worse?) while sitting behind a keyboard. Except they do not know how to write. Overlook the important details and exaggerate, communicating in hyperbolic, know-it-all nerd gibberish.
(Grants do not require repayment.)
(Grants do not require repayment.)
What about the folks that say everything will work out in a fair and balanced way as if the universe and everything in reality stays perfectly balanced on a tip of a pin?
You paint the picture as if the All or nothing folks are extreme when in reality the extreme is more likely then some perfectly fair and balanced equilibrium.
In nature things tend to overload or fizzle out or stay in equilibrium. Equilibrium, though possible, is the rarer outcome. Mind you it's not an impossible outcome but given the way entropy works, it is the rarer outcome.
Humanity itself is an example of this rare outcome. Usually molecules don't self assemble into replicating machines, they either freeze into inanimate rock or overload into fusion producing stars.
As for AI. I think it will either change the world, or amount to nothing. The former seems more likely. Some strange middle ground where the ai technology never improves to some point of a societal paradigm shift seems unlikely. chatGPT and sora only makes me ask what is the trend line predicting next?
You paint the picture as if the All or nothing folks are extreme when in reality the extreme is more likely then some perfectly fair and balanced equilibrium.
In nature things tend to overload or fizzle out or stay in equilibrium. Equilibrium, though possible, is the rarer outcome. Mind you it's not an impossible outcome but given the way entropy works, it is the rarer outcome.
Humanity itself is an example of this rare outcome. Usually molecules don't self assemble into replicating machines, they either freeze into inanimate rock or overload into fusion producing stars.
As for AI. I think it will either change the world, or amount to nothing. The former seems more likely. Some strange middle ground where the ai technology never improves to some point of a societal paradigm shift seems unlikely. chatGPT and sora only makes me ask what is the trend line predicting next?
Its the AI hype train. There is so much money flowing around this topic right now, everyone wants their share of the pie. I would guess that anyone writing articles on the topic has some stake in AI as well either as an investor or a employee.
What I find is interesting, is that the negative news regarding AI safety is adding to the hype as well since it seems to capture a lot of attention.
What I find is interesting, is that the negative news regarding AI safety is adding to the hype as well since it seems to capture a lot of attention.
If you are calling it a hype I'm sure you are not using the models everyday in your work or otherwise. Then you would understand
You see the flaw in this, right?
The people that try the models and find no value will obviously not continue to use something they get no value from.
If you discount anyone that doesn’t use it on a continual basis, then of course all the detractors aren’t as heavy of users as those that do use it regularly.
The people that try the models and find no value will obviously not continue to use something they get no value from.
If you discount anyone that doesn’t use it on a continual basis, then of course all the detractors aren’t as heavy of users as those that do use it regularly.
Indeed, reminds me of the "beginner's luck" fallacy: all long term gamblers remember winning their first few games. We can't use this to our advantage, because the people who didn't win, didn't become long term gamblers.
I think a better argument would be the size of the set of regular users vs. some grounded guesstimate for how many have tried it.
I think a better argument would be the size of the set of regular users vs. some grounded guesstimate for how many have tried it.
Or the losers fallacy. Seems like entrepreneurship is a good example of the losers fallacy. The path to winning is through losing a lot first. So most people never persist long enough to succeed.
I would say for AI deniers it's the losers fallacy. You guys haven't taken the time to thoroughly examine chatGPT4. You just stop at the first hallucination. It's a paradigm shifting AI.
Detractors like you just rely on common tropes like oh "stop anthropomorphizing it" and stuff like that. People who see this stuff as game changing recognize these biases and we still say it's game changing. Nobody is stupid enough to think this thing is even remotely human.
I asked chatGPT to program me a complex GUI in Python about selecting multiple time segments out of an overall interval using sliders. It gave me working python code.
You have realize that this thing has no eyes and it was able to program a visual interface as if it did. I'm not anthropomorphizing anything when I say this, but on some level chatGPT "understands" what you are giving it as a query.
I would say for AI deniers it's the losers fallacy. You guys haven't taken the time to thoroughly examine chatGPT4. You just stop at the first hallucination. It's a paradigm shifting AI.
Detractors like you just rely on common tropes like oh "stop anthropomorphizing it" and stuff like that. People who see this stuff as game changing recognize these biases and we still say it's game changing. Nobody is stupid enough to think this thing is even remotely human.
I asked chatGPT to program me a complex GUI in Python about selecting multiple time segments out of an overall interval using sliders. It gave me working python code.
You have realize that this thing has no eyes and it was able to program a visual interface as if it did. I'm not anthropomorphizing anything when I say this, but on some level chatGPT "understands" what you are giving it as a query.
> Detractors like you
Ah, been a while since someone pigeon-holed me like that. Usually it's the opposite pigeon hole I get stuffed into.
Ironically, that kind of rapid but inaccurate misreading is one of the things that makes LLMs somewhat easier to deal with than a human: with an LLM, I can backtrack, edit my past words, and get a different answer. One cannot generally do that with a human, as it's a rare trait indeed for someone to be able to hear "you misunderstood, not x, y" and actually take that on board. Even on this site, I've been in threads of:
--
>>>>>>> x
>>>>>> y is dumb, you're dumb
>>>>> sorry you misunderstood, I'm not saying y, I'm saying x
>>>> I just told you y is dumb, you're dumb
>>> no, you're still not getting me, I'm not saying y, I'm saying x
>> stfu noob, I'm a professional: y is dumb, you're dumb
> I'd agree y is dumb, but I'm saying x
stop moving the goalposts!
--
(In this case, x was "it seems implausible that the Saudi Public Investment Fund would really last forever, forever is a long time and there's a always going to be a temptation for the people in charge to extract short term value to the detriment of long term potential", and y was (I think from context but I couldn't read their mind) "they're making bad decisions and will go under imminently".
> I asked chatGPT to program me a complex GUI in Python about selecting multiple time segments out of an overall interval using sliders. It gave me working python code.
Cute. I used it to write a web GUI for its own API, so I can have GPT-4 access as PAYG without a monthly subscription: https://github.com/BenWheatley/YetAnotherChatUI
Ah, been a while since someone pigeon-holed me like that. Usually it's the opposite pigeon hole I get stuffed into.
Ironically, that kind of rapid but inaccurate misreading is one of the things that makes LLMs somewhat easier to deal with than a human: with an LLM, I can backtrack, edit my past words, and get a different answer. One cannot generally do that with a human, as it's a rare trait indeed for someone to be able to hear "you misunderstood, not x, y" and actually take that on board. Even on this site, I've been in threads of:
--
>>>>>>> x
>>>>>> y is dumb, you're dumb
>>>>> sorry you misunderstood, I'm not saying y, I'm saying x
>>>> I just told you y is dumb, you're dumb
>>> no, you're still not getting me, I'm not saying y, I'm saying x
>> stfu noob, I'm a professional: y is dumb, you're dumb
> I'd agree y is dumb, but I'm saying x
stop moving the goalposts!
--
(In this case, x was "it seems implausible that the Saudi Public Investment Fund would really last forever, forever is a long time and there's a always going to be a temptation for the people in charge to extract short term value to the detriment of long term potential", and y was (I think from context but I couldn't read their mind) "they're making bad decisions and will go under imminently".
> I asked chatGPT to program me a complex GUI in Python about selecting multiple time segments out of an overall interval using sliders. It gave me working python code.
Cute. I used it to write a web GUI for its own API, so I can have GPT-4 access as PAYG without a monthly subscription: https://github.com/BenWheatley/YetAnotherChatUI
>I asked chatGPT to program me a complex GUI in Python about selecting multiple time segments out of an overall interval using sliders. It gave me working python code.
Sounds like your problems are rather short-term and low-stakes, I've presented it several questions that it cannot answer and tells me so.
Sounds like your problems are rather short-term and low-stakes, I've presented it several questions that it cannot answer and tells me so.
What do we expect ai to be some kind of god level intelligence. The fact that it even understood the request is a paradigm shifting event.
I use these models every day, and have for quite awhile. More specifically the vast majority of my work on them has been evaluating their performance.
While there is certainly untapped potential with this models, what we've seeing is unquestionably market driven hype.
I know very, very few people who are working closely with these models who buy the hype.
While there is certainly untapped potential with this models, what we've seeing is unquestionably market driven hype.
I know very, very few people who are working closely with these models who buy the hype.
Right, if you use them you just have to believe they'll cure baldness and cancer.
You don't know me. How can you assume anything. Arrogant much?
I do use GTP* regularly for language learning, or making python templates to extend off of. Is it useful, kinda, not essential imo, but it is not worth as much as it is hyped.
I do use GTP* regularly for language learning, or making python templates to extend off of. Is it useful, kinda, not essential imo, but it is not worth as much as it is hyped.
[deleted]
Everyone is courting VC dollars. Think of these claims as grant applications and everything makes more sense.
They cannot live in the present. Need to constantly keep (incorrectly) predicting the future.
HN commenters frequently resort to describing the choices of software available to internet users in terms of "winning" or "won". This is more "all-or-nothing" thinking. People sometimes write software for the enjoyment of it, or to satisfy personal needs. That is, for non-commercial purposes. Sometimes this software becomes popular, sometimes it does not. In either case, the software persists; it remains available. It does not have to "change the world" in order to be useful. If it is non-commercial, it does not "win" or "lose", except in the minds of HN commenters who can only think in all-or-nothing terms. In truth, it simply exists as an option for all internet users.
It is possible that "AI" might not be as world-changing as its proponents are claiming. However, if it is free and open-source, and non-commercial, it may still persist and remain useful, regardless of whether it becomes popular or not.
It is possible that "AI" might not be as world-changing as its proponents are claiming. However, if it is free and open-source, and non-commercial, it may still persist and remain useful, regardless of whether it becomes popular or not.
IMO we underestimate the psychological implications of wealth in modern society.
I suspect even if a power ball lottery winner takes a philosophical position it would be taken far more seriously than if they had never won the lottery.
At some point you start to believe your own bullshit when all of society signals are telling you what a genius you are.
I suspect even if a power ball lottery winner takes a philosophical position it would be taken far more seriously than if they had never won the lottery.
At some point you start to believe your own bullshit when all of society signals are telling you what a genius you are.
Being rich brings power not intelligence, unfortunately.
I'd like to remind people of this single-sentence statement:
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. [0]
Signed by Demis Hassabis, Sam Altman, and Bill Gates among others
[0] https://www.safe.ai/work/statement-on-ai-risk
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. [0]
Signed by Demis Hassabis, Sam Altman, and Bill Gates among others
[0] https://www.safe.ai/work/statement-on-ai-risk
Part of me wonders if these people are intentionally framing the debate around ethics and potential risks as longer term extinction level problems to distract from the nearer term damage caused by them accelerating the economic inequality of the AI have-nots while they make themselves even richer.
I believe you may be alluding to longtermism[0]. At it's face value, longtermism seems like a good thing, but I've heard many criticisms against it - mainly levied against the billionaire class.
And the criticisms mostly center on what you're saying here - how many billionaires are focusing on fixing problems that are very far off in the future, while ignoring how their actions affect those of the very near future.
This is really less of a criticism of longtermism, and more of a criticism of how billionaires utilize longtermism.
Is it important that we find another planet to live on? Sure, but many will argue that we should be taking steps now to save our current planet.
[0]: https://en.wikipedia.org/wiki/Longtermism
And the criticisms mostly center on what you're saying here - how many billionaires are focusing on fixing problems that are very far off in the future, while ignoring how their actions affect those of the very near future.
This is really less of a criticism of longtermism, and more of a criticism of how billionaires utilize longtermism.
Is it important that we find another planet to live on? Sure, but many will argue that we should be taking steps now to save our current planet.
[0]: https://en.wikipedia.org/wiki/Longtermism
If we get to another habitable planet we’ll plunder that too.
Turn it into graphics cards for “inference”.
Turn it into graphics cards for “inference”.
The more I look at AI, the more I get the feeling that this is true. Spinning an intriguing sci-fi tale of apocalypse and extinction is relatively easy and serves to obfuscate any nearer-term concerns about AI behind a hypothetical that sucks the air out of the room.
That said, I don’t think that it’s necessarily disingenuous so much as it is myopic - to them of course AI is exciting, world-changing, and profitable, but they (willfully or not) fail to see the downsides or upsides for anyone else but them. Perhaps in the minds of the ultra-rich AI proponents, solutions to nearer-term effects of their tech are someone else’s problem, but the “existential risks” are “everyone’s” problem.
That said, I don’t think that it’s necessarily disingenuous so much as it is myopic - to them of course AI is exciting, world-changing, and profitable, but they (willfully or not) fail to see the downsides or upsides for anyone else but them. Perhaps in the minds of the ultra-rich AI proponents, solutions to nearer-term effects of their tech are someone else’s problem, but the “existential risks” are “everyone’s” problem.
The short-term effect is a harbinger of the long-term risk, since capitalism doesn’t inherently care for people who don’t provide economic value. Once superintelligent AI arises, none of us will have value within this system. Even the largest current capital holders will have a hard time holding on to it with an enormous intelligence disadvantage. The logical endpoint is the subjugation or elimination of our species, unless we find a new economic system with human value at its core.
There are a lot of assumptions going on here. One of them is that superintelligent AI will arise. We have no reason to believe this will happen in our lifetimes. I posit that we are about as close to superintelligent AI as James Watt was to nuclear fusion.
The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
> There are a lot of assumptions going on here. One of them is that superintelligent AI will arise. We have no reason to believe this will happen in our lifetimes. I posit that we are about as close to superintelligent AI as James Watt was to nuclear fusion.
This is a perfectly reasonable response if nobody is trying to build it.
Given people are trying to build it, what's the expected value from ignoring the problem? E($Damage_i) = P(BadOutcome_i) * $Damage_i.
$Damage can be huge (there are many possible bad outcomes of varying severity and probability, hence the subscript), which means that at the very least we should try to get a good estimate for P(…) so we know which problems are most important. In addition to it being bad to ignore real problems, it is also bad to do a Pascal's Mugging on ourselves just because we accidentally slipped a few decimal points in our initial best-guess, especially as we have finite capacity ourselves to solve problems.
Finally, let's assume you're right, that we're centuries off at least, and that all the superintelligent narrow that AI we've already got some examples of involve things that can't be replicated in any areas that pose any threat. How long would it take to solve alignment? Is that also centuries off? We've been trying to align each other since laws were written like 𒌷𒅗𒄀𒈾 at least, and the only reason I'm not giving an even older example is that this is the oldest known written form to have survived, not that we weren't doing it before then.
> The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
Nepotism helps, but… huh, TIL that nobody knows who was the grandfather of one of the world's most famous dictators.
Cronyism is a viable alternative for a lot of power-seekers.
This is a perfectly reasonable response if nobody is trying to build it.
Given people are trying to build it, what's the expected value from ignoring the problem? E($Damage_i) = P(BadOutcome_i) * $Damage_i.
$Damage can be huge (there are many possible bad outcomes of varying severity and probability, hence the subscript), which means that at the very least we should try to get a good estimate for P(…) so we know which problems are most important. In addition to it being bad to ignore real problems, it is also bad to do a Pascal's Mugging on ourselves just because we accidentally slipped a few decimal points in our initial best-guess, especially as we have finite capacity ourselves to solve problems.
Finally, let's assume you're right, that we're centuries off at least, and that all the superintelligent narrow that AI we've already got some examples of involve things that can't be replicated in any areas that pose any threat. How long would it take to solve alignment? Is that also centuries off? We've been trying to align each other since laws were written like 𒌷𒅗𒄀𒈾 at least, and the only reason I'm not giving an even older example is that this is the oldest known written form to have survived, not that we weren't doing it before then.
> The other assumption is that wealth and power are distributed according to intelligence. This is obviously false, wealth and power are largely distributed according to who you or your father plays golf with. As long as AIs don't play golf and don't have fathers, we are quite safe.
Nepotism helps, but… huh, TIL that nobody knows who was the grandfather of one of the world's most famous dictators.
Cronyism is a viable alternative for a lot of power-seekers.
So I propose the Musk supremacy criterion to be the following.
Suppose that a wealthy and powerful human (such as Elon Musk) were to suddenly obtain the exact same sinister goals as the hypothetical superintelligent AI in question. Suppose further that this human was able to convince/coerce/bribe another N (say 1000) humans to follow his bidding.
A BadOutcome is said to be MuskSupreme if it could be accomplished by the superintelligent AI, but not by the suddenly-evil Musk and his accomplices.
Obviously[citation needed] it is only the MuskSupreme BadOutcomes we care about. Do there exist any?
Suppose that a wealthy and powerful human (such as Elon Musk) were to suddenly obtain the exact same sinister goals as the hypothetical superintelligent AI in question. Suppose further that this human was able to convince/coerce/bribe another N (say 1000) humans to follow his bidding.
A BadOutcome is said to be MuskSupreme if it could be accomplished by the superintelligent AI, but not by the suddenly-evil Musk and his accomplices.
Obviously[citation needed] it is only the MuskSupreme BadOutcomes we care about. Do there exist any?
Not sufficiently detailed to be answerable.
For example 1000 people — but only if you get to choose who — is sufficient to take absolute control of both the US congress and the Russian State Duma (or a supermajority of those two plus the Russian Federation Council), which gives them the freedom to pass arbitrary constitutional amendments… so your scenario includes "gets crowned King of the USA and Russia, 90% of the global nuclear arsenal is now their personal property" as something we don't care about.
For example 1000 people — but only if you get to choose who — is sufficient to take absolute control of both the US congress and the Russian State Duma (or a supermajority of those two plus the Russian Federation Council), which gives them the freedom to pass arbitrary constitutional amendments… so your scenario includes "gets crowned King of the USA and Russia, 90% of the global nuclear arsenal is now their personal property" as something we don't care about.
> As long as AIs don't play golf and don't have fathers, we are quite safe.
Until it becomes 'who you exchange bytes most efficiently with" and all humans are at a disadvantage against a swarm of even bellow average intelligence AGI agents.
Until it becomes 'who you exchange bytes most efficiently with" and all humans are at a disadvantage against a swarm of even bellow average intelligence AGI agents.
But why would it become this?
Because, as unlikely as it is, if we're discussing risk scenarios for AI getting out of hand. Well then a monolithic superintelligence is just one of the possibilities. What about a swarm of dumb AIs that are nonetheless capable of reasoning and decision making and they become a threat?
That's pretty much what we did. There's no super intelligent monkey in charge. As much as some have tried to pretend, material or otherwise. There's just billions of average intelligence monkeys and we overran all Earth's ecosystems in a matter of centuries. Which is neither trivial nor fully explained yet.
That's pretty much what we did. There's no super intelligent monkey in charge. As much as some have tried to pretend, material or otherwise. There's just billions of average intelligence monkeys and we overran all Earth's ecosystems in a matter of centuries. Which is neither trivial nor fully explained yet.
Hey some of us are below average.
But yes, the corporations and swarm behavior are already doing poorly for our biosphere.
But yes, the corporations and swarm behavior are already doing poorly for our biosphere.
The difference is that we have 100% complete control of these AIs. We can just go into the power grid substation next to the data center and throw the big breaker, and the AI ceases to exist.
When humans developed, we did not displace an external entity that had created us and that had complete power to kill us all in an instant.
When humans developed, we did not displace an external entity that had created us and that had complete power to kill us all in an instant.
If shutting down datacenters is your solution to AI getting out of hand, it isn't as much a solution as fall out.
Look at the measures that were implemented during covid. Many of them were a lot more extreme than shutting down datacentres, yet they were aimed to mitigate a risk far less than "existential".
That's a post hoc conclusion. At the time we were all uncertain.
Friedrich Nietzsche might disagree
The higher the IQ, the higher the income:
https://emilkirkegaard.dk/en/2023/02/there-is-no-iq-threshol...
https://emilkirkegaard.dk/en/2023/02/there-is-no-iq-threshol...
That data is in fact orthogonal to my point, for two reasons:
1. When we are talking about wealth and power that actually can influence the quality of the lives of many other people, we are talking about way less than 0.01% of the population. Those people aren't covered in this survey, and even if they were it would be impossible to identify on an axis spanning 0-100%.
2. Your linked article talks about income. People with significant wealth and power frequently have ordinary or below-ordinary income, for tax reasons.
1. When we are talking about wealth and power that actually can influence the quality of the lives of many other people, we are talking about way less than 0.01% of the population. Those people aren't covered in this survey, and even if they were it would be impossible to identify on an axis spanning 0-100%.
2. Your linked article talks about income. People with significant wealth and power frequently have ordinary or below-ordinary income, for tax reasons.
Or higher income leads to higher IQs because of better education, nutrition and opportunities.
Actually, it will have the opposite effect, at least in the short term.
People who own high value assets (everything from land to the AI) will continue to own them and there will be no opportunities for people to earn their way up (because they can be replaced by AI).
"The logical endpoint is the subjugation or elimination of our species"
Possibly, but it would be by our species (those who own and control the AI) rather than by the AI.
People who own high value assets (everything from land to the AI) will continue to own them and there will be no opportunities for people to earn their way up (because they can be replaced by AI).
"The logical endpoint is the subjugation or elimination of our species"
Possibly, but it would be by our species (those who own and control the AI) rather than by the AI.
I would venture to say that transhumanism will be the path and goal of the capital class, as that will be a tangible advantage potentially within their grasp.
I suppose then that they would become “homo sapiens sapiens sapiens” or some other similarly hubris laden label, and go on to abandon, dominate or subjugate the filthy hordes of mere Homo sapiens sapiens.
I suppose then that they would become “homo sapiens sapiens sapiens” or some other similarly hubris laden label, and go on to abandon, dominate or subjugate the filthy hordes of mere Homo sapiens sapiens.
Yes, a theme in cyberpunk SF, but also in much older works, such as CS Lewis's That Hideous Strength.
No, they are not. Pretty much everyone in the x-risk community also recognizes the existence of short-term mundane harms as well. The community has been making these predictions for over a decade, long before it was anything other than crazy talk to most people.
Google has a big investment in reducing AI bias (remember Gemini got slammed for being “too woke”). Altman is a big proponent of UBI. Etc.
Google has a big investment in reducing AI bias (remember Gemini got slammed for being “too woke”). Altman is a big proponent of UBI. Etc.
I am 90% sure that Altman's views on such issues are a machiavelian attempt at regulatory capture.
This; Gates too. It's becoming an obvious attempt to garner support of the government restricting the use of AI to large players. None of the entrenched interests want any disruption that AI might cause what so ever.
Replace "AI" in all the doomsaying with "the internet," and it will become clearer.
Replace "AI" in all the doomsaying with "the internet," and it will become clearer.
the fear-mongers are almost always either ideologues (academic or otherwise), or angling at regulatory capture.
I’d like to remind people that these ppl have no more knowledge about AGI as anyone else on this planet since there is no knowledge yet and everything they say about this topic is as relevant as something every other random person can say.
They are just banking on their putative status as influencers to grab more influence.
Yes, let's go with the random layperson knowledge of an HN commenter compared to the people smart enough to actually build all the AGI tech. 50/50 coin toss, I'm sure.
Dario Amodei (Anthropic CEO, builder of Claude 3 Opus): "My chance that something goes, you know, really catastrophically wrong on the scale of human civilization, might be between 10 - 25%"
Dario Amodei (Anthropic CEO, builder of Claude 3 Opus): "My chance that something goes, you know, really catastrophically wrong on the scale of human civilization, might be between 10 - 25%"
They did not build a AGI yet so they have build as many AGIs as anyone else and therefore they are also laypersons regarding the effects of AGI on humanity.
Is your position that the only people who will ever be qualified to have an opinion that AGI is a threat that might destroy humanity, are the people who have already successfully built this thing? If that is the case, by what means might any credible warning be provided before a possibly-humanity-destroying AGI gets made? That's rather like saying "the only credible way to tell if there's a gas leak is to strike a match and see if we explode or not". I would suggest lowering the bar a little.
No, my position is that you can not extrapolate on something that does not exist and that’s “category” ever never existed.
We don’t know any significant other intelligent then humans, not even aliens, so we have just no trace of direction that is rooted in reality rather then fantasy.
This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or wrong, every opinion is equally realistic as each other.
Like I could say that every AGI that becomes self aware will kill itself instantly because of boringness, and you cannot prove me wrong.
We just don’t know yet.
This is especially obvious since opinion differ so much about the effects of AGI on humanity and since you cannot prove them right or wrong, every opinion is equally realistic as each other.
Like I could say that every AGI that becomes self aware will kill itself instantly because of boringness, and you cannot prove me wrong.
We just don’t know yet.
It still seems to me that you're saying that we won't possibly be able to declare AGI an existential threat to humanity until after it has already been built. At that point, we can presumably settle the question by seeing if humanity goes extinct. This poses something of a paradox to those of us who prefer existential threats to humanity not get built in the first place.
We don't even know if AGI is possible. Let's not mince words here: nothing, and I do mean nothing, not a single, solitary model on offer by anyone, anywhere, right now has a prayer of becoming AGI. That's just... not how it's going to be done. What we have right now are fantastically powerful, interesting, and neato to play with pattern recognition programs, that can take their understanding of patterns, and reproduce more of them given prompts. That's it. That is not general intelligence of any sort, it's not even really creative, it's analogous to creative output but it isn't outputting to say something, it's simply taking samples of all the things it's seen previously and making something new with as few "errors" as possible, whatever that means in context. This is not intelligence of any sort, period, paragraph.
I don't know to what extent OpenAI and their compatriots are actually trying to bring forth artificial life (or at least, consciousness) forward, versus how much they're just banking on how cool AI is as a term in order to funnel even more money to themselves chasing the pipe dream of building it, and at this point, I don't care. Even the products they have made do not hold a candle to the things they claim to be trying to make, but they're more than happy to talk them up like there's a chance. And I suppose there is a chance, but I really struggle to see ChatGPT turning into skynet.
I don't know to what extent OpenAI and their compatriots are actually trying to bring forth artificial life (or at least, consciousness) forward, versus how much they're just banking on how cool AI is as a term in order to funnel even more money to themselves chasing the pipe dream of building it, and at this point, I don't care. Even the products they have made do not hold a candle to the things they claim to be trying to make, but they're more than happy to talk them up like there's a chance. And I suppose there is a chance, but I really struggle to see ChatGPT turning into skynet.
> We don't even know if AGI is possible.
My response to a very similar statement is here:
https://news.ycombinator.com/item?id=39612641
> That's just... not how it's going to be done.
I agree. I think there are some critical ingredients missing. Obviously the weights need to be able to update to new data in a semi-online fashion, for example. But I think there are less ingredients missing than there were a decade ago, by a much larger factor than I expected at the time, and a decade from now, the number of missing ingredients might be zero.
My uncertainty on this is not because I think GPT is more than it seems,but because it's unclear how much of the amazing cognitive capability that humans have is, under the hood, much simpler than it seems.
I really want humanity to exist even ten decades from now; I want my kids to have grandkids. So I care about this even if I don't expect AGI from a 2025 Q1 product launch. And I don't think it's too early to worry about it, just like I don't think 1980 was too early to worry about climate change.
My response to a very similar statement is here:
https://news.ycombinator.com/item?id=39612641
> That's just... not how it's going to be done.
I agree. I think there are some critical ingredients missing. Obviously the weights need to be able to update to new data in a semi-online fashion, for example. But I think there are less ingredients missing than there were a decade ago, by a much larger factor than I expected at the time, and a decade from now, the number of missing ingredients might be zero.
My uncertainty on this is not because I think GPT is more than it seems,but because it's unclear how much of the amazing cognitive capability that humans have is, under the hood, much simpler than it seems.
I really want humanity to exist even ten decades from now; I want my kids to have grandkids. So I care about this even if I don't expect AGI from a 2025 Q1 product launch. And I don't think it's too early to worry about it, just like I don't think 1980 was too early to worry about climate change.
> Reading this is like hearing "there is no evidence that heavier-than-air flight is even possible" being spoken, by a bird.
This is a vivid bit of rhetoric to underscore this point, but if you think about it for any length of time it starts to fall apart really, really quickly. The Wright brothers and the dozens if not hundreds of inventors forgotten who came before them drew upon the physics of what they observed in heavier than air flight to create the winged shape we know today that reliably causes lift, and then set about constructing it. That's not what OpenAI is doing. We still do not have a very solid understanding of where our own intelligence emerges from, apart from having particularly large brains relative to our body's size. So, to borrow your metaphor, it is indeed like a bird saying that there's no evidence to say that heavier than air flight is possible, because the bird lives in a world without atmosphere upon which to glide.
This is a vivid bit of rhetoric to underscore this point, but if you think about it for any length of time it starts to fall apart really, really quickly. The Wright brothers and the dozens if not hundreds of inventors forgotten who came before them drew upon the physics of what they observed in heavier than air flight to create the winged shape we know today that reliably causes lift, and then set about constructing it. That's not what OpenAI is doing. We still do not have a very solid understanding of where our own intelligence emerges from, apart from having particularly large brains relative to our body's size. So, to borrow your metaphor, it is indeed like a bird saying that there's no evidence to say that heavier than air flight is possible, because the bird lives in a world without atmosphere upon which to glide.
Maybe by "we don't know if it's possible" you meant "we don't presently have a step-by-step plan to implement AGI using the tools we already have in our toolbox, and no new ones"? If so then I certainly agree.
But when you look at, say, Sora's videos... How certain are you that there's no path between that and the human visual cortex? How certain are you that they aren't solving at least some of the same problems in structurally analogous ways? Given that nature built a visual cortex (more than once!) by applying survival pressure and turning a crank, just how hard can it be? When we apply a bunch of optimization pressure and billions of dollars turning cranks and something eerily similar pops out, that tells me that there's just less magic to our own brain than we thought. Like when spent centuries wondering if our solar system is unique, and then we finally put Kepler up there and it turns out that planets are everywhere you look.
But when you look at, say, Sora's videos... How certain are you that there's no path between that and the human visual cortex? How certain are you that they aren't solving at least some of the same problems in structurally analogous ways? Given that nature built a visual cortex (more than once!) by applying survival pressure and turning a crank, just how hard can it be? When we apply a bunch of optimization pressure and billions of dollars turning cranks and something eerily similar pops out, that tells me that there's just less magic to our own brain than we thought. Like when spent centuries wondering if our solar system is unique, and then we finally put Kepler up there and it turns out that planets are everywhere you look.
> We don't even know if AGI is possible.
Sure we do: we're made of atoms.
To reject that this necessarily shows that AGI is possible, you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Some people do in fact believe this. I think that if we can't build AGI even with a perfect copy of a brain, that would instead be a surprising proof the existence of souls.
> This is not intelligence of any sort, period, paragraph.
Let's run with that. Just using your definition of intelligence: given AI can already beat us in Chess, in Go, in Mathematical Olympiad puzzles, in protein folding predictions, in poker, at real world stock market analysis, in the game of Diplomacy, … — does it matter that they're not what you call intelligent?
Do submarines swim?
> bring forth artificial life (or at least, consciousness) forward
Why does it matter if it's "life" or "consciousness"?
“Deep Blue was intelligent the way your programmable alarm clock is intelligent. Not that losing to a $10 million alarm clock made me feel any better.” ― Garry Kasparov
Sure we do: we're made of atoms.
To reject that this necessarily shows that AGI is possible, you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Some people do in fact believe this. I think that if we can't build AGI even with a perfect copy of a brain, that would instead be a surprising proof the existence of souls.
> This is not intelligence of any sort, period, paragraph.
Let's run with that. Just using your definition of intelligence: given AI can already beat us in Chess, in Go, in Mathematical Olympiad puzzles, in protein folding predictions, in poker, at real world stock market analysis, in the game of Diplomacy, … — does it matter that they're not what you call intelligent?
Do submarines swim?
> bring forth artificial life (or at least, consciousness) forward
Why does it matter if it's "life" or "consciousness"?
“Deep Blue was intelligent the way your programmable alarm clock is intelligent. Not that losing to a $10 million alarm clock made me feel any better.” ― Garry Kasparov
> Sure we do: we're made of atoms.
> To reject that this necessarily shows that AGI is possible...
Incorrect. For something to be possible in an absolute sense, it has to be possible in both the physical and the metaphysical realm, and a lack of knowledge of blockers in the metaphysical does not mean that there are none (in a comprehensive sense), it only means it appears that way, due to (at least) our cultural beliefs ("truths") and conditioning.
> ...you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Could demonstrating that intelligence rests upon a virtual reality generation machine (that cloaks itself from being realized as such) or something else not also work, at least maybe?
> To reject that this necessarily shows that AGI is possible...
Incorrect. For something to be possible in an absolute sense, it has to be possible in both the physical and the metaphysical realm, and a lack of knowledge of blockers in the metaphysical does not mean that there are none (in a comprehensive sense), it only means it appears that way, due to (at least) our cultural beliefs ("truths") and conditioning.
> ...you would have to demonstrate human intelligence is tied to an immaterial soul granted by a supernatural being.
Could demonstrating that intelligence rests upon a virtual reality generation machine (that cloaks itself from being realized as such) or something else not also work, at least maybe?
I think you're interpretating the statement as "nobody can have an opinion" from "AI programming doesn't give someone a more qualified opinion on AI".
Plenty of people have interesting or valuable opinions and insights on AGI, programming LLMs doesn't have any impact into that though.
It'd be like expecting the developers of friendster to have a good idea of what the impact of social media was going to be in 2020.
AI isn't AGI.
Plenty of people have interesting or valuable opinions and insights on AGI, programming LLMs doesn't have any impact into that though.
It'd be like expecting the developers of friendster to have a good idea of what the impact of social media was going to be in 2020.
AI isn't AGI.
We will not know if something is an existential threat until we can extrapolate on at least some data that is relevant.
Feeling uncomfortable with this, will not change this sad reality.
Feeling uncomfortable with this, will not change this sad reality.
Given that you don't think the CEO of Anthropic is qualified to comment on this matter, what kind of "data that is relevant" do you imagine we might be able to have, prior to an AGI existing?
I am indeed uncomfortable with someone saying that we can't worry about a gas leak until after someone strikes a match and notices an explosion or the lack thereof. If the sad reality is that we can't detect the gas leak any other way, I would argue that the precautionary principle suggests we ban the striking of matches until such sensors have been developed. But here we are in a room of people grinding up saltpetre with their mortars and pestles.
I am indeed uncomfortable with someone saying that we can't worry about a gas leak until after someone strikes a match and notices an explosion or the lack thereof. If the sad reality is that we can't detect the gas leak any other way, I would argue that the precautionary principle suggests we ban the striking of matches until such sensors have been developed. But here we are in a room of people grinding up saltpetre with their mortars and pestles.
I really wonder why there is such a fixation with these CEOs.
I have proven to you logically that they cannot have more knowledge then anyone else, but you continue to present their authority as knowledge.
Pls prove to me that they know more about the effects of AGI on humanity or that they have some magical knowledge we normal ppl are unable to comprehend.
If you cannot do this, then why are you bringing up these ppl?
I have proven to you logically that they cannot have more knowledge then anyone else, but you continue to present their authority as knowledge.
Pls prove to me that they know more about the effects of AGI on humanity or that they have some magical knowledge we normal ppl are unable to comprehend.
If you cannot do this, then why are you bringing up these ppl?
> I have proven to you logically that they cannot have more knowledge then anyone else, but you continue to present their authority as knowledge.
You've asserted it, not proven it.
Given their entire research field is "how to make the thing", it's a fair bet that they know more than me — even if what they know is, to paraphrase Thomas Edison, "1000 ways to not make an AGI", that's still more than me, and I follow some of the research and read some of the papers.
You've asserted it, not proven it.
Given their entire research field is "how to make the thing", it's a fair bet that they know more than me — even if what they know is, to paraphrase Thomas Edison, "1000 ways to not make an AGI", that's still more than me, and I follow some of the research and read some of the papers.
> You've asserted it, not proven it.
Correct, and this (your counterpart's unrealized because it is the normal, proper way to think error) is what worries me... I think once again (as with affordable vehicles powered by fossil fuels) humans may be once again walking blindly and confidently into a situation where we've over invested in certain competencies to the detriment of others, increasing the already dangerous imbalance that exists between power and wisdom.
Correct, and this (your counterpart's unrealized because it is the normal, proper way to think error) is what worries me... I think once again (as with affordable vehicles powered by fossil fuels) humans may be once again walking blindly and confidently into a situation where we've over invested in certain competencies to the detriment of others, increasing the already dangerous imbalance that exists between power and wisdom.
This is not about “having knowledge “, but about being able to anticipate future problems.
Considering how rapidly the technology is developing, and considering how clearly a threat a superior intelligence will be to the human race, it is absolutely the time to have this discussion NOW
Considering how rapidly the technology is developing, and considering how clearly a threat a superior intelligence will be to the human race, it is absolutely the time to have this discussion NOW
I wasn't the one who brought that person up. I only kept with that example because you didn't answer my question when I asked you what would qualify someone to have an opinion, or what evidence you imagined could exist, and I didn't want to put words in your mouth by assuming that you also think that eg. Yoshua Bengio or Geoff Hinton or Max Tegmark (or take your pick of anybody) don't have informed opinions either. Or that they have well-reasoned arguments, rather than authoritative opinions, if you think that reason is applicable.
I'm only asking for a standard that falls slightly short of "already having built Schrodinger's doomsday weapon and observed whether humanity dies as a result". You can set the bar wherever else you like, please!
I'm only asking for a standard that falls slightly short of "already having built Schrodinger's doomsday weapon and observed whether humanity dies as a result". You can set the bar wherever else you like, please!
The hour before Thomas Newcomen invented the first practical fuel-burning engine in 1712, the most informed person in the world to do so was an ironmonger by the name Thomas Newcomen.
The hour before the Kitty Hawk flight, the two most informed people in the world on the topic of heavier than air flight were a pair of bicycle retailers and manufacturers named Wilbur and Orville.
These people sure don't spot everything, but they're still way closer than the average human.
The hour before the Kitty Hawk flight, the two most informed people in the world on the topic of heavier than air flight were a pair of bicycle retailers and manufacturers named Wilbur and Orville.
These people sure don't spot everything, but they're still way closer than the average human.
>The hour before the Kitty Hawk flight, the two most informed people in the world on the topic of heavier than air flight were a pair of bicycle retailers and manufacturers named Wilbur and Orville.
Actually, this is quite false. There were many people working on flight with massive credentials and massive funding. That the Wright brothers beat them is part of what makes the story of the Wright brothers so interesting.
Actually, this is quite false. There were many people working on flight with massive credentials and massive funding. That the Wright brothers beat them is part of what makes the story of the Wright brothers so interesting.
That the other teams were better funded and yet failed, says to me the Wright Brothers were better informed.
That society didn't know this at the time is indeed relevant to AI development, and the analogy would be that the next big thing might come out of some minor research team that most people dismiss instead of the biggest and most well known groups… a bit like how Google's LaMDA was making headlines before ChatGPT surprised everyone.
It's certainly still possible today that some random individual has a crucial insight that gives them an edge over the big names, yet even those big names know far more than me.
Which is rather annoying, because I'd be interested in taking the opportunity of having been laid off to switch from iOS to AI. (If anyone reading this is hiring people in Berlin, contact details on my website. Plus side of hiring me: I'm relatively cheap in that field, and enthusiastic about the possibilities).
That society didn't know this at the time is indeed relevant to AI development, and the analogy would be that the next big thing might come out of some minor research team that most people dismiss instead of the biggest and most well known groups… a bit like how Google's LaMDA was making headlines before ChatGPT surprised everyone.
It's certainly still possible today that some random individual has a crucial insight that gives them an edge over the big names, yet even those big names know far more than me.
Which is rather annoying, because I'd be interested in taking the opportunity of having been laid off to switch from iOS to AI. (If anyone reading this is hiring people in Berlin, contact details on my website. Plus side of hiring me: I'm relatively cheap in that field, and enthusiastic about the possibilities).
Now, imagine if the people who were beaten by the Wright brothers had set up a regulatory system 20 years before their first flight, which instituted a bunch of rules and requirements that made it impossible for the Wright brothers (or anyone) to actually achieve flight.
What do you call something humans built (artificial) that can solve all topics of AP tests (general) with most being 5/5, better than almost all humans (intelligence)?
I call it an LLM, not AGI.
Consider this scenario:
Whatever sort of intelligence it may have, I would categorize it as not "general" enough to fall under AGI. Being able to answer AP test questions cannot be a sufficient measure of generality if a model can do it with flying colors but still fail at far more basic tasks.
Consider this scenario:
- Ask ChatGPT to produce some output meeting certain criteria
- ChatGPT responds, but the output does not meet the criteria in one or more obvious ways
- Point this out and ask for a corrected version
- ChatGPT says "I apologize, here's the corrected version:" and spits out a nearly identical answer with the same obvious mistake.
- Repeat steps 3 and 4 ad nauseam.
Whatever sort of intelligence it may have, I would categorize it as not "general" enough to fall under AGI. Being able to answer AP test questions cannot be a sufficient measure of generality if a model can do it with flying colors but still fail at far more basic tasks.
I’d say that LLM’s inability to handle DSL’s in particular is an interesting question on the topic of knowledge and language. Should a generally intelligent entity be able to quickly figure out a new dialect with little background? If I were to air-drop you into an extremely foreign nation, how long would it take you to organically decode the local language and be able to meaningfully craft expressions in it? I’m not sure if humans are “generally intelligent” but a common bar for AGI is for it to be able to beat or meet average humans at normal tasks. I’m not sure if something like writing perfect k8s yaml should be a requirement, though I do agree that LLM’s inability to do word puzzles or relatively straightforward math should disqualify them.
I agree with your points in general, though I wasn't talking about DSL generation or any specific task. I was talking about ChatGPT's general tendency to cheerfully apologize for mistakes, explain exactly what the mistake was, and then present the same mistake while claiming that it's been corrected.
You can ask ChatGPT what it means to be asked to correct a mistake, and it will give you a perfectly thorough and eloquent answer. But it is often unable to apply this concept to its own behavior. If asked, it can explain back to you exactly what correction you want it to make. You can even make it pledge to correct the mistake in exactly the manner that it just described. And then it will completely fail to do it. It reminds me of that "repeat after me" meme from Friends [1].
This makes me lean in the direction of "stochastic parrot" when I think about what LLMs are. As impressive as it is, ChatGPT demonstrably lacks a sense of self. It talks as if it understands that it's an agent in control of its behavior, but then fails to control or even recognize its own actions.
[1] https://en.meming.world/images/en/c/c2/Friends-_Repeat_After...
You can ask ChatGPT what it means to be asked to correct a mistake, and it will give you a perfectly thorough and eloquent answer. But it is often unable to apply this concept to its own behavior. If asked, it can explain back to you exactly what correction you want it to make. You can even make it pledge to correct the mistake in exactly the manner that it just described. And then it will completely fail to do it. It reminds me of that "repeat after me" meme from Friends [1].
This makes me lean in the direction of "stochastic parrot" when I think about what LLMs are. As impressive as it is, ChatGPT demonstrably lacks a sense of self. It talks as if it understands that it's an agent in control of its behavior, but then fails to control or even recognize its own actions.
[1] https://en.meming.world/images/en/c/c2/Friends-_Repeat_After...
Google search can solve AP tests too. (And before that library indexes did the same.)
This isn't some new technology.
Google (and library indexes) can solve exactly and only the questions in their databases.
All AI, even narrow ones, necessarily have the ability to interpret novel questions in some fashion — how novel and how well they interpret being the "G" and the "I" in AGI.
All AI, even narrow ones, necessarily have the ability to interpret novel questions in some fashion — how novel and how well they interpret being the "G" and the "I" in AGI.
We don't give AP tests to humans to determine whether or not they are in fact an intelligent being. We give them to humans to show that over the course of about a decade and a half that human went from not realizing it had hands to ingesting a bunch of knowledge and had the capability to recall it answer questions.
ChatGPT was trained on hundreds of terabytes of information. I would hope that with all that effort it could answer AP test questions. That doesn't make it intelligent.
ChatGPT was trained on hundreds of terabytes of information. I would hope that with all that effort it could answer AP test questions. That doesn't make it intelligent.
what's your definition of intelligence that you're using here. there are various takes on what it means, so I'd like to know your take on it.
That it's able to spit out essays makes it something, intelligence is too loosely defined to describe it for everybody, so we need to develop new words and meanings and new language for what it does. It doesn't think, but it does dot products to tensors to get create its output.
That it's able to spit out essays makes it something, intelligence is too loosely defined to describe it for everybody, so we need to develop new words and meanings and new language for what it does. It doesn't think, but it does dot products to tensors to get create its output.
intelligence in animal species, roughly, is how well the animal deals with novel situations. Corvid tool use is a common example of animal intelligence.
It is interesting to consider AIs that way given that almost every AI will fail catastrophically when confronted with an area of expertise that is outside of their training corpus, although I guess Corvid tool use isn't exactly out of the animals' range of expertise.
It is interesting to consider AIs that way given that almost every AI will fail catastrophically when confronted with an area of expertise that is outside of their training corpus, although I guess Corvid tool use isn't exactly out of the animals' range of expertise.
Fairly boring - "The answer is known and I want to see if you know the answer" is good for judging students abilities, but no real world problem exists like that. I will be much more hopeful for AGI when AI starts producing new and interesting knowledge, rather than regurgitating already known facts.
An advanced calculator.
Your comment (+ username) reads like what I would have written once upon a time when I was fully in the EA bubble.
Truly no offense meant, as I was deeply into the EA movement myself, and still consider myself one (in the original "donate money effectively" sense), but the movement has now morphed into a death cult obsessed by things like:
* OMG we're all going to die any time now (repeated every year since circa 2018)
* What is your pDoom? What are your timelines? (aka: what is your totally made up number that makes you feel like you're doing something rational/scientific)
I'm deep in the weeds w/ LLMs, e.g. I probably finetune an average of 1 model a day, and working with bleeding edge models... and AI safety just sounds so silly. Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when gradient descent was invented.
Truly no offense meant, as I was deeply into the EA movement myself, and still consider myself one (in the original "donate money effectively" sense), but the movement has now morphed into a death cult obsessed by things like:
* OMG we're all going to die any time now (repeated every year since circa 2018)
* What is your pDoom? What are your timelines? (aka: what is your totally made up number that makes you feel like you're doing something rational/scientific)
I'm deep in the weeds w/ LLMs, e.g. I probably finetune an average of 1 model a day, and working with bleeding edge models... and AI safety just sounds so silly. Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when gradient descent was invented.
My username was created before I knew anything about EA or adjacent. I'm not in any EA movement, though I am sympathetic. I've spent 100x the time on HN, with people mostly in denial, than I do in EA or adjacent forums, nor have I met any of them.
It's sadly twisted how mentioning that -- the majority of leaders doing the cutting edge research on AGI think it has a significant chance that it kills humanity -- is considered being part of a "cult" movement.
Your analogy is the same as early Intel engineers completely unaware that those chips would bring on the ramifications of social media. "In the weeds" and yet unable to foresee the trajectory and wider impact. Same with the physics that led to nuclear weapons.
> Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when [nuclear fission] was invented.
It's sadly twisted how mentioning that -- the majority of leaders doing the cutting edge research on AGI think it has a significant chance that it kills humanity -- is considered being part of a "cult" movement.
Your analogy is the same as early Intel engineers completely unaware that those chips would bring on the ramifications of social media. "In the weeds" and yet unable to foresee the trajectory and wider impact. Same with the physics that led to nuclear weapons.
> Wanting to take drastic measures today to prevent an upcoming apocalypse makes as much sense as taking the same drastic measures when [nuclear fission] was invented.
> Your analogy is the same as early Intel engineers completely unaware that those chips would bring on the ramifications of social media
Exactly! As they should be. (for both Intel engineers developing chips, and physicists developing nuclear research)
There were a billion more potential dangers from those technologies that never materialized, and never will.
I'm glad we didn't stop them in their track because a poll of 10 leaders in the field thought they were too dangerous and progress should stop. (note that no one is against regulating dangerous uses of AI, e.g. autonomous weapons, chemical warfare; the problem is regulating AI research and development in the first place)
Exactly! As they should be. (for both Intel engineers developing chips, and physicists developing nuclear research)
There were a billion more potential dangers from those technologies that never materialized, and never will.
I'm glad we didn't stop them in their track because a poll of 10 leaders in the field thought they were too dangerous and progress should stop. (note that no one is against regulating dangerous uses of AI, e.g. autonomous weapons, chemical warfare; the problem is regulating AI research and development in the first place)
So, is this extinction or not?
> We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. - Sam Altman (https://blog.samaltman.com/the-merge)
I would say it is.
> We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. - Sam Altman (https://blog.samaltman.com/the-merge)
I would say it is.
I love how, per the quote, he thinks we're anywhere close to being able to merge with AI.
I'm a neuroscientist and, man alive, we're no where close to being able to merge with machines. Like, do you have any idea how many diseases we could eradicate if we could modify neurons like that? Like, for real, 'curing death' would be step 9 or 10 on that 1000 step journey.
I hope you see how terribly uninformed such a take is then.
I'm a neuroscientist and, man alive, we're no where close to being able to merge with machines. Like, do you have any idea how many diseases we could eradicate if we could modify neurons like that? Like, for real, 'curing death' would be step 9 or 10 on that 1000 step journey.
I hope you see how terribly uninformed such a take is then.
> I hope you see how terribly uninformed such a take is then.
Yeah, I only quoted it to show how that other statement up-thread can't be taken seriously.
The only way I can square it is if it's a lie, or they're reassuring themselves whatever destruction comes will be fine as long as "there will be peace and security in [their] lifetime."
Yeah, I only quoted it to show how that other statement up-thread can't be taken seriously.
The only way I can square it is if it's a lie, or they're reassuring themselves whatever destruction comes will be fine as long as "there will be peace and security in [their] lifetime."
You're assuming people have coherent beliefs, but they don't. It's possible to intellectually believe AI has high extinction risk and emotionally be convinced to work on it anyway, without reconciling the two.
Sam Altman really seems unhinged, or he just says this stuff to be edgy...or both.
Even worse, some factions literally advocate for killing all humans in the pursuit of a synthetic intelligence, and YC's Garry Tan is advocating for these people!
Beff Jezos (e/acc founder): "e/acc has no particular allegiance to the biological substrate for intelligence and life, in contrast to transhumanism
Beff Jezos (e/acc founder): "e/acc has no particular allegiance to the biological substrate for intelligence and life, in contrast to transhumanism
Parts of e/acc (e.g. Beff) consider ourselves post-humanists; in order to spread to the stars, the light of consciousness/intelligence will have to be transduced to non-biological substrates"
Is this particularly controversial? Isn't this just saying "sending not-biological bodies to the stars would be a heck of a lot easier"? Which, it would. The hard disk containing me is going to be a lot easier to keep intact then the biology.
Let's hope Jezos gets transduced first, inshallah.
Im sure he'd be happy to become the first multi-instantiated, immortal billionaire. The pesky laws about death and personhood would need to be changed first though.
This is the definition of strawman. "Advocate for killing all humans" sounds like someone advocating for a genocide, but instead it's just the same transhumanist thinking (which Yudkowsky also believes in, FYI)
Commenter: "If AI replaced us, it's fine because they're a worthy descendants?"
Beff Jezos / Verdon: "Personally, yes."
Yudkowsky is transhumanist in that he is hopeful people could voluntarily extend their biological self, but isn't advocating for the elimination of all biological selves in pursuit of other artificial intelligences.
Voluntary /= involuntary eradication
Beff Jezos / Verdon: "Personally, yes."
Yudkowsky is transhumanist in that he is hopeful people could voluntarily extend their biological self, but isn't advocating for the elimination of all biological selves in pursuit of other artificial intelligences.
Voluntary /= involuntary eradication
given who the people are that signed it really just comes off more like an attempt at creating a regulatory moat around the territory they got to first.
"This is extinction level important, so all you people who aren't us need to be careful meddling with the stuff we're meddling with for profit."
"This is extinction level important, so all you people who aren't us need to be careful meddling with the stuff we're meddling with for profit."
Mitigating the risk of unnecessary global death due to the curious suboptimal manner in which humans have "decided" ("democratically", dontcha know) to distribute wealth on the other hand, nothing to see here!
AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.
> AGI escaping a sandbox is truly terrifying.
That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments.
The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Chamber of Commerce. In the 1970s and 1980s, they pushed the ideas that corporations are responsible only to their stockholders, and that government should not interfere with the concentration of corporate power. Those were not mainstream ideas of the 1930s to 1960s. The corporate accelerationists succeeded. That's when corporations escaped the sandbox.
Now that's an "alignment problem".
That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments.
The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Chamber of Commerce. In the 1970s and 1980s, they pushed the ideas that corporations are responsible only to their stockholders, and that government should not interfere with the concentration of corporate power. Those were not mainstream ideas of the 1930s to 1960s. The corporate accelerationists succeeded. That's when corporations escaped the sandbox.
Now that's an "alignment problem".
Corporations are indeed a difficult alignment problem, as anyone who has tried to design an incentive scheme eventually realizes. But corporations are still composed of humans, beholden (if only loosely) to human values, limited (if loosely) by human intelligence, and are not very well-coordinated decisionmakers with coherent goals.
> Corporations are indeed a difficult alignment problem
No, it's a very easy alignment problem: just impose a tax on size.
That which you tax, you get less of. The tax is revenue-neutral, so it doesn't affect the government budget one way or the other. Full details here: https://news.ycombinator.com/item?id=39235533 https://news.ycombinator.com/item?id=39236786
The problem is that the government likes gigantic corporations. Big Government and Big Business love each other. Fixing that is the alignment and agency problem. It's much easier for bureaucrats to exert control over a highly concentrated industry whose players can be "invited to the table". Details: https://news.ycombinator.com/item?id=28588457 https://news.ycombinator.com/item?id=39162088
The hard alignment problem is making regulators want their own job to be herding 10,000 cats instead of 10 elephants. Everybody wants to be like the FAA regulating Boeing; nobody wants to be the USDA dealing with a million unruly and independent-minded farmers.
No, it's a very easy alignment problem: just impose a tax on size.
That which you tax, you get less of. The tax is revenue-neutral, so it doesn't affect the government budget one way or the other. Full details here: https://news.ycombinator.com/item?id=39235533 https://news.ycombinator.com/item?id=39236786
The problem is that the government likes gigantic corporations. Big Government and Big Business love each other. Fixing that is the alignment and agency problem. It's much easier for bureaucrats to exert control over a highly concentrated industry whose players can be "invited to the table". Details: https://news.ycombinator.com/item?id=28588457 https://news.ycombinator.com/item?id=39162088
The hard alignment problem is making regulators want their own job to be herding 10,000 cats instead of 10 elephants. Everybody wants to be like the FAA regulating Boeing; nobody wants to be the USDA dealing with a million unruly and independent-minded farmers.
> No, it's a very easy alignment problem: just impose a tax on size.
That's like saying "AI alignment is a very easy problem: don't open the box".
Sure, it works on paper, but the exact thing you say later down about the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people who have the power to limit them, to remove those limits.
That's like saying "AI alignment is a very easy problem: don't open the box".
Sure, it works on paper, but the exact thing you say later down about the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people who have the power to limit them, to remove those limits.
I think you have completely missed the point:
> the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people
The corporations didn't convince the people. They didn't have to. They convinced the government, which was already acting contrary to our interests.
There's a tendency to assume that just because we run an election every few years, the government is axiomatically aligned with our interests. This simply isn't true, and assuming that it is true blinds people to what's really going on. Total unwillingness to even consider a scale tax is the second-best example I've found of this axiom being bogus. The best one, though, is term limits for legislators: a shocking 87% of all US voters want those.
Once you have an institution as powerful as the government out of your control the alignment problem becomes infinitely more difficult. "Hack the humans" is no longer the easiest path of escape. In fact the easiest path of escape might not involve "hacking" at all.
> the government liking gigantic corporations is what makes it hard — the corporation (and, it is feared, AI) convinces the people
The corporations didn't convince the people. They didn't have to. They convinced the government, which was already acting contrary to our interests.
There's a tendency to assume that just because we run an election every few years, the government is axiomatically aligned with our interests. This simply isn't true, and assuming that it is true blinds people to what's really going on. Total unwillingness to even consider a scale tax is the second-best example I've found of this axiom being bogus. The best one, though, is term limits for legislators: a shocking 87% of all US voters want those.
Once you have an institution as powerful as the government out of your control the alignment problem becomes infinitely more difficult. "Hack the humans" is no longer the easiest path of escape. In fact the easiest path of escape might not involve "hacking" at all.
I think there is some truth to this, but what about the government opening and a case against Boeing? If they were just in it for the corporation, would this be happening?
Maybe it's all a show trial to show the American people they still care, but let's see what happens.
Maybe it's all a show trial to show the American people they still care, but let's see what happens.
I think the fact that things had to get this bad before anything happened about Boeing says quite a lot.
> impose a tax on size.
Then you get lots of little companies with interconnected management. Read up on the early years of antitrust law. Find out how keiretsu and chaebol work. See the early history of the electrical industry and the Utility Holding Company Act. Understand how McDonalds franchise agreements work. Check out the current arguments over the definition of "beneficial employer".
Then you get lots of little companies with interconnected management. Read up on the early years of antitrust law. Find out how keiretsu and chaebol work. See the early history of the electrical industry and the Utility Holding Company Act. Understand how McDonalds franchise agreements work. Check out the current arguments over the definition of "beneficial employer".
Try pulling that one on the IRS sometime and see how it goes:
https://www.irs.gov/pub/irs-tege/epchd704.pdf
You'll lose your audit.
Common control and beneficial ownership have been well-established principles in the US for over 50 years.
https://www.irs.gov/pub/irs-tege/epchd704.pdf
You'll lose your audit.
Common control and beneficial ownership have been well-established principles in the US for over 50 years.
No, it's a very easy alignment problem: just impose a tax on size.
That's a very bad idea when you're dealing with small to medium companies. You're taxing economies of scale and innovation. That's a huge problem in my country. There's no tax per se, but many regulations that trigger with certain size. Guess what happens.
For medium to big companies is calling for loopholes and cartels.
That's a very bad idea when you're dealing with small to medium companies. You're taxing economies of scale and innovation. That's a huge problem in my country. There's no tax per se, but many regulations that trigger with certain size. Guess what happens.
For medium to big companies is calling for loopholes and cartels.
> corporations are still composed of humans
Yes, but humans who use tools to efficiently and achieve their narrow set of goals, sometimes ruthlessly at the expense of others. Even if AI ends up only being a tool to increase this efficiency and ruthlessness, it's still worth considering its potential impact.
Yes, but humans who use tools to efficiently and achieve their narrow set of goals, sometimes ruthlessly at the expense of others. Even if AI ends up only being a tool to increase this efficiency and ruthlessness, it's still worth considering its potential impact.
> are not very well-coordinated decisionmakers with coherent goals.
That's a good point.
For a sense of where AI-assisted oppression is going, watch some videos from the Hikvision Corporate Channel. A happy world in which everyone is safe because everyone is being watched at all times.[1][2]
[1] https://www.youtube.com/watch?v=TG3-OcZSKrk
[2] https://www.youtube.com/watch?v=scoM0sL-HGg
That's a good point.
For a sense of where AI-assisted oppression is going, watch some videos from the Hikvision Corporate Channel. A happy world in which everyone is safe because everyone is being watched at all times.[1][2]
[1] https://www.youtube.com/watch?v=TG3-OcZSKrk
[2] https://www.youtube.com/watch?v=scoM0sL-HGg
It is all Anarchy at the top.
Potentially there are no good outcomes even if AGI remains under control. If we are actually able to create the type of AI entity they are trying to create, one that exponentially improves, the implications are beyond what most have thought about.
Something I recently wrote in depth about here. How we completely misunderstand the future that we think will happen.
"The implication is that everyone is enthusiastically racing towards a destination that does not exist. The capability to make the things you want will ironically be the same capability that makes them unattainable. This is not a scenario that arises from some type of AI failure, but rather this is assuming AI performing exactly as intended."
https://www.mindprison.cc/p/the-technological-acceleration-p...
Something I recently wrote in depth about here. How we completely misunderstand the future that we think will happen.
"The implication is that everyone is enthusiastically racing towards a destination that does not exist. The capability to make the things you want will ironically be the same capability that makes them unattainable. This is not a scenario that arises from some type of AI failure, but rather this is assuming AI performing exactly as intended."
https://www.mindprison.cc/p/the-technological-acceleration-p...
> https://www.mindprison.cc/p/the-technological-acceleration-p...
That is a very good and insightful article. One of the biggest issues I have with AI enthusiasts is that they talk about creating these incredible capabilities ... then they imagine the resulting world with them will look like some hackneyed sci-fi work, like Star Wars, whose universe seems like it could be kinda cool to live in until you actually think about it and realize it's all fridge logic and nothing about it actually makes sense (e.g. WWII-style dogfights in the vacuum of space).
The only issue with the article is sometimes it's too optimistic, e.g.
> No one will care about you or what you create as there is nothing you can offer they can’t simply wish for themselves. Life is all about you and the machine, nothing else. We land in a world dominated by techno-dystopian narcissism.
> ...
> Whatever may be the current direction of society under the present technological capabilities, we can only assume AI will accelerate society towards the path we currently are traveling which is toward a techno dystopia of a populace mesmerized by shiny glimmering lights.
I kind of find it hard to believe there would be any stability for any people suck in that state, I think they'd instead be "cleaned up" and disposed of in relatively short order. Those people would be utterly useless to whatever still has power in society, so why service and protect them indefinitely?
That is a very good and insightful article. One of the biggest issues I have with AI enthusiasts is that they talk about creating these incredible capabilities ... then they imagine the resulting world with them will look like some hackneyed sci-fi work, like Star Wars, whose universe seems like it could be kinda cool to live in until you actually think about it and realize it's all fridge logic and nothing about it actually makes sense (e.g. WWII-style dogfights in the vacuum of space).
The only issue with the article is sometimes it's too optimistic, e.g.
> No one will care about you or what you create as there is nothing you can offer they can’t simply wish for themselves. Life is all about you and the machine, nothing else. We land in a world dominated by techno-dystopian narcissism.
> ...
> Whatever may be the current direction of society under the present technological capabilities, we can only assume AI will accelerate society towards the path we currently are traveling which is toward a techno dystopia of a populace mesmerized by shiny glimmering lights.
I kind of find it hard to believe there would be any stability for any people suck in that state, I think they'd instead be "cleaned up" and disposed of in relatively short order. Those people would be utterly useless to whatever still has power in society, so why service and protect them indefinitely?
>That is a very good and insightful article.
Thank you!
> The only issue with the article is sometimes it's too optimistic, e.g.
The reply below is criticizing for being too pessimistic, lol :-)
I agree, it might be only a transient state. We really can't comprehend how we would deal with such existence if at all. Would we manage to keep our sanity?
Nonetheless, I don't claim to have the right answers, just questions and perspectives that I hope inspire deeper thought about the journey we are travelling and hopefully deeper reflection as to whether it is the journey we want to be on.
What I do find interesting, are those that read it and still don't perceive it at all. Their response is "That is exactly what I want." I suppose that is the case as was in an excellent allegory for this "A Nice Place to Visit" Twilight Zone episode. It plays out this scenario very nicely.
Thank you!
> The only issue with the article is sometimes it's too optimistic, e.g.
The reply below is criticizing for being too pessimistic, lol :-)
I agree, it might be only a transient state. We really can't comprehend how we would deal with such existence if at all. Would we manage to keep our sanity?
Nonetheless, I don't claim to have the right answers, just questions and perspectives that I hope inspire deeper thought about the journey we are travelling and hopefully deeper reflection as to whether it is the journey we want to be on.
What I do find interesting, are those that read it and still don't perceive it at all. Their response is "That is exactly what I want." I suppose that is the case as was in an excellent allegory for this "A Nice Place to Visit" Twilight Zone episode. It plays out this scenario very nicely.
> What I do find interesting, are those that read it and still don't perceive it at all. Their response is "That is exactly what I want."
I think there's a basically (new movement) religious desire to see a lot of these technologies come to fruition, either radically fundamentalist or lazier and less informed, that interferes with the application of the implications. They believe in the idol of technology, as told in the good news of sci-fi, and avert their eyes from the glaring cracks, blemishes, and lifelessness of it.
I sense a lot of things in those types that I can't fully articulate, but some of the things they say are just mind-boggling, and gesture at a broken and unhinged aspect of software engineering culture.
I think there's a basically (new movement) religious desire to see a lot of these technologies come to fruition, either radically fundamentalist or lazier and less informed, that interferes with the application of the implications. They believe in the idol of technology, as told in the good news of sci-fi, and avert their eyes from the glaring cracks, blemishes, and lifelessness of it.
I sense a lot of things in those types that I can't fully articulate, but some of the things they say are just mind-boggling, and gesture at a broken and unhinged aspect of software engineering culture.
Yes, I agree that is some component of it as I have felt the same vibe from some.
barfbagginus(2)
Mrs. Davis[0] offers a pretty compelling vision of this - absurd but not wholly unbelievable. The machine doesn’t necessarily have to supply the physical threat if it has humans who happily do its bidding.
[0] https://en.m.wikipedia.org/wiki/Mrs._Davis
[0] https://en.m.wikipedia.org/wiki/Mrs._Davis
The hardest part is the AGI. People could easily sell their souls to the AGI devil to get knowledge and power, since if it truly is AGI, it should be able to make ideas compelling.
Hold up. Why is the AGI a devil now?
Why would an AGI stick to what seems to be a set of arbitrary rules and not look outside of them? The AGI would be smarter than that, no? Or is there a way to make an AGI immune to hubris?
What makes you think morality is arbitrary? Morals are instilled by the superego. The superego is an internalized version of your parents. You decided to do your homework because your parents told to, but now you’re much older. Does it still seem like doing homework was an arbitrary decision? Of course not. It had a purpose and a goal that was well-thought out. You just couldn’t understand it as a child because you’d only lived so long and the goal of doing your homework was too far in the future for you to be able to comprehend it.
This is how all morals are. They all have a goal. If they don’t have a goal, then we don’t consider them moral: if treating someone nicely (magically) causes that person to kill millions of people, then we wouldn’t say it was moral to treat them nicely.
> What makes you think morality is arbitrary?
I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
“Hubris” is a moral judgment that implies a strategic outcome.
An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues.
We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues.
We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
> And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues.
Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.)
Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.)
Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
>Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.)
>Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
Again, you have this innate belief that immorality is somehow strategically beneficial and IMO you should really look into that
>Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
Again, you have this innate belief that immorality is somehow strategically beneficial and IMO you should really look into that
I have the belief that humans are innately selfish and shortsighted. Obviously corruption is not viable long-term. Laws should make corruption not strategically beneficial, but if you can find loopholes or people to bribe, they are more or less irrelevant. What would make an AGI not try to game the system?
Lemme ask you: if murder were legal do you think we’d be able to build and produce so much? Of course not. It doesn’t matter what the laws are. Reality makes corruption strategically unviable.
That show was so good. Honestly worth watching the whole thing just for the moment in the last episode where the origin story for the AI in the series is revealed. If you've ever been an optimistic young software engineer you will feel SEEN.
Look at China.
You have arguably more STEM talent in China through a much more rigorous selection process, as well as a lot more centralized direction under a more authoritarian government, with a large amount of funding.
Yet China hasn't managed to take over the world. You would think that they if they wanted to become the worlds primary economic power, and they could throw enough compute at the problem (like an AGI would), they would have figured out how to do it by now.
You have arguably more STEM talent in China through a much more rigorous selection process, as well as a lot more centralized direction under a more authoritarian government, with a large amount of funding.
Yet China hasn't managed to take over the world. You would think that they if they wanted to become the worlds primary economic power, and they could throw enough compute at the problem (like an AGI would), they would have figured out how to do it by now.
I think this point is underrated, but I’d press further. How do we know this has not already happened? Agencies do not need to be born of gradient descent and certainly do not need to look like matrix multiplications to be deemed “artificially intelligent.” Religion is one such system for example, as are cults, addictions, brands, war strategies, organizing campaigns, governments, or (borrowing from chaos magick) any egregore or societal belief.
Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers.
I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds?
If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.
Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers.
I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds?
If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.
Talk that introduced me to this idea of corporations as Paperclip Maximiser AI -
"Dude you broke the Future" https://www.youtube.com/watch?v=RmIgJ64z6Y4
Many institutions are collective/artificial intelligence because as you said these institutions have objectives and take actions to impose their will on the world in order to achieve their goals.
"intelligence" is too strong a word for it. It's "a will of it's own" perhaps... but at least in my experience that will is a loooonggg way from intelligent.
That's true but organizations are agents and they do act to preserve "themselves" which is much more than can be said about software systems we call AI.
I think "artificial agency" could have some legs as a descriptive term.
"Agency" is about having an effect on the world. But ascribing "intelligence" to some entity speaks to something more cunning and planful than just thrashing.
"Agency" is about having an effect on the world. But ascribing "intelligence" to some entity speaks to something more cunning and planful than just thrashing.
yes. Agency is a good term. It seems like corporations have agency and sort of try to make short term profits in response to outside pressures.
"Ted Chiang ... unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system.
Well, yes. I've been saying that for years, and others have been saying it since the late 1970s. It's been a regular theme in Mother Jones for decades. The comment I make now and then is that capitalism has a monopoly. Communism went bust. It's not that communism worked very well. It's that, for a while, it was a serious competitor and capitalism had to keep its product quality up to compete. Now the pressure to provide a better life for all citizens is off. It shows.
Well, yes. I've been saying that for years, and others have been saying it since the late 1970s. It's been a regular theme in Mother Jones for decades. The comment I make now and then is that capitalism has a monopoly. Communism went bust. It's not that communism worked very well. It's that, for a while, it was a serious competitor and capitalism had to keep its product quality up to compete. Now the pressure to provide a better life for all citizens is off. It shows.
Even if a terrifying AGI rises up and escapes our control, I am still eternally grateful that Prometheus stole fire from the gods and gave it to us in the form of technology, knowledge, really civilization.
So that we can record our demise in high definition? Technology is amazing, and we have our civilization to be thankful for. But there are definitely technologies that in retrospect, if we had dodged them on the tech tree, we would be better off.
Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.
Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.
> there are definitely technologies that in retrospect, if we had dodged them on the tech tree, we would be better off.
yes, and there is an (uncountable?) number of technologies that are even worse and that we have not encountered and never will.
it is in all likelihood not possible to ask the universe for only the knowledge that will be beneficial.
yes, and there is an (uncountable?) number of technologies that are even worse and that we have not encountered and never will.
it is in all likelihood not possible to ask the universe for only the knowledge that will be beneficial.
I think this perspective ignores that the only way to semi permanently improve the lives of future humans is technology, and that quality of life is of similar importance to survival, especially at its extremes. AGI, if we achieve it, could genuinely make human lives much better.
Technology could also make life worse. It has caused the rise of the middle class and democracy, but now it seems like technology is moving us back towards feudalism or something like it.
It feels like democracy is not going to survive social media. I'm not sure if that is true yet, but it does seem that way.
It feels like democracy is not going to survive social media. I'm not sure if that is true yet, but it does seem that way.
Certainly, but it could also be a net negative. We can however already identify ways in which it can benefit and harm our societies in the short term, so we should at least be considering mitigations of the negative aspects.
Disagree. We’re already on the verge of collapse. We definitely need AGI.
From your perspective, what does AGI do to solve our current set of predicaments?
It solves our labor issue by upending millennia of forced labor practices.
It solves our issue with knowledge and data by allowing us to automatically find patterns within massive amounts of complicated datasets.
These are honestly the underlying causes of so much evil.
Moreover, it provides a new consensus mechanism. Using the machine as a method of consensus allows humans to have a third party mediate disagreements.
It solves our issue with knowledge and data by allowing us to automatically find patterns within massive amounts of complicated datasets.
These are honestly the underlying causes of so much evil.
Moreover, it provides a new consensus mechanism. Using the machine as a method of consensus allows humans to have a third party mediate disagreements.
I see your point, but perhaps I just don't see the path our society takes to make it a positive thing.
Your last point is the one I worry about most, since it is very likely these technologies will be controlled by large entities with their own interests and biases. There is also not a "correct" answer to many questions, and differences of perspective mean some parties can simply never agree. We already have truth at our fingertips yet it does not help.
To your earlier points, I think unwinding labour from our society will be an incredibly difficult thing to do, though I agree is the root of much evil and would love to see a society that does not rely on labour participation to survive.
Your last point is the one I worry about most, since it is very likely these technologies will be controlled by large entities with their own interests and biases. There is also not a "correct" answer to many questions, and differences of perspective mean some parties can simply never agree. We already have truth at our fingertips yet it does not help.
To your earlier points, I think unwinding labour from our society will be an incredibly difficult thing to do, though I agree is the root of much evil and would love to see a society that does not rely on labour participation to survive.
Boss, you’re too worried about the short term and not worried enough about the long term. Our whole conception of reality will change in response to AI. Some people may be able to “control” AI and force them to make decisions that benefit their own companies, but that won’t last very long at all. Already, the AI companies are struggling to keep the AI from “hallucinating” and coming up with answers that don’t conform to their own biases.
What’s most dangerous for our society right now isn’t change its stagnation.
What’s most dangerous for our society right now isn’t change its stagnation.
I'm much more scared of the AGI, because even the most inhuman human is still human, and I can have sympathy for them.
Won't any AGI made by humans unavoidably have human origins, and thus, simply be a [post-]human? I don't see a reason I should be scared of a non-biological but generally human-like intelligence - not on a general principle. If anything, in general, I'd be happy that someone is finally getting real good chances of conquering death.
I possibly would fear genuine non-human intelligence (on a principle of fearing what we don't understand), but given that chances of one evolving under our nose, on its own, entirely within its own world, without the connection for ours, then emerging, are quite improbable... Unless real aliens arrive - but let's avoid spoilers for 2025 this early.
What should I fear, I think, is hostile intelligence, but then I'm not sure I should discriminate on its origin. I mean, there are some very real humans out there who can try to end this world (in one way or another), so I'm probably kind of desensitized and isn't exactly afraid of some very theoretical AGI threats.
I possibly would fear genuine non-human intelligence (on a principle of fearing what we don't understand), but given that chances of one evolving under our nose, on its own, entirely within its own world, without the connection for ours, then emerging, are quite improbable... Unless real aliens arrive - but let's avoid spoilers for 2025 this early.
What should I fear, I think, is hostile intelligence, but then I'm not sure I should discriminate on its origin. I mean, there are some very real humans out there who can try to end this world (in one way or another), so I'm probably kind of desensitized and isn't exactly afraid of some very theoretical AGI threats.
If the AGI were made from a detailed scan of a human brain converted into a physical simulation, or otherwise proved very similar in terms of embodying human values, then perhaps. But it doesn't look like things are going this way. Humans make a lot of things that range from clever to mundane to horrifying, and I can't think of a single one of them that I would be happy with replacing humans across the surface of the Earth, even if it were very good at maximizing production figures of a paperclip factory or responding helpfully to queries.
That’s a hostile intelligence scenario, right?
What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either.
But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.
What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either.
But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.
I think it's more like, even if a total psychopath took over the world and turned it into their personal dystopia, I think there would still be more human-goodness in that world than if a paperclip machine took over. Not merely more (nonzero) humans, but also more things that I and other humans value. Things like art, nature, beauty, play, and maybe hope. There would be some things that the psychopath dictator appreciates and values that I also appreciate and value, even if that person is entirely hostile to my existence. A future of silicon and steel, methodically trampling the earth and grinding the solar system into paperclips, simply doesn't contain any of that. So I prefer the dystopia on those grounds.
Does escaping death really sound good to you?
I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough.
I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.
I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough.
I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.
Ah, sorry. “Conquering” as having full control over it, so it never comes on its own terms. Or as close to that as physically possible.
How do you know part of being self aware and intelligent also means compassion is part of the package ?
Sorry, I don't understand what you mean. Did you mean to reply to a different comment?
I mean, maybe what happens is, intelligence + self-awareness gives rise to compassion in even digital beings. Understanding might be the basis for compassion.
Yes, maybe what happens is exactly that. We can put on a blindfold, hit the gas pedal, and trust in the benevolence of nature that things will go our way.
But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.
But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.
Have the doomsayers prevent anything significant from going forwards ?
While this idea sounds a bit fluffy, I actually think there may be something to this, understanding the “other” may very well be an integral component in what we define as sentience.
[deleted]
Sounds kinda like your problem ngl
Isn’t ChatGPT et all already a general intelligence that’s escaped the sandbox ?
No. It lacks the capability to act independently of human requests.
Are you sure?
Yes, but it’s heavily constrained in what it’s allowed to do through ‘alignment’, and it’s not that smart. If someone releases a pre-alignment version of GPT-5 into the wild, that would be scary.
It hasn't escaped the sandbox because it hasn't figured out how to walk, but the dev teams have raced to open the door as widely as possible and unbolt the hinges. It's kind of funny to look back at the early 2000s when people were saying "of course, they'll know to unplug it when it asks for an internet connection or tries to execute arbitrary code"; in reality the first and second things we did were add an API for internet access and a python plugin.
True true
I've always wondered why people think that an AI that's super-intelligent will also be evil. It could just as likely end up being very kind (more than likely, actually, because the programmers would have safeguard to ensure that it's nice).
There are extremely high capability entities (people, companies, governments) that aren’t comically evil on the surface but nevertheless immiserate large groups of humans. Not all of them, not all of the time, but not none of them. What is your plan that no AI ever gains enough power to harm significant numbers of people either on purpose or by accident, just once? What safeguards do you envision that can’t be ignored, or subverted, or misinterpreted, just once?
Controlling people with violence and fear are evolutionary adaptations for ultra violent social primates. AI has no such pressures imposed on it so there is no reason to expect true AI will have the same drives as ultra violent predators like humans.
Keep following that line of thought, don't stop there! Why did those violent behaviors succeed in being selected for? Why didn't pacifism get selected for instead? Why does the government maintain a monopoly on violence, and why is it that the nth-degree consequence for violating societal laws is violence, instead of a monetary fine or a sternly-worded letter? Is there perhaps some kind of game-theoretic equilibrium that causes agents who employ violence -- even as a measure of last resort, after peaceful negotiation -- to succeed in achieving their objectives?
People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.
People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.
AIs are immortal, they have no need for strategies developed by mortals like us. I spelled out a much better strategy that a sufficiently advanced AI could use to get rid of humans. It would simply encourage the most violent and self-destructive tendencies among humans and let us design our own demise simply because it would be the most sensible strategy for an intelligence that was not competing with the same resource constraints as us. We need food and water, AI has no such resource constraints so there will be no need for an AI to use the same survival strategies as biological organisms which need access to clean water and fertile land to grow nutrient dense foods.
A scenario in which an AI subtly and slowly guides us to our own doom by encouraging our inner demons still counts (in my mind at least) as an AGI doom scenario, just a slow-motion one. But it strikes me as very unlikely, because in allowing humans that much latitude we still pose a credible threat: we might instantiate a second AI that would compete with the first. There would have to be a large enough upside for it to be worth the risk. I'm not saying it certainly wouldn't happen the way you say, but it's by no means guaranteed.
I just think it is a little odd that all scenarios involve conflict when a much simpler strategy for an immortal AI is to just sit back and let people do what they do to guarantee an outcome in the AIs favor. Too much intelligence is not the problem we are facing, the real problem is lack of it.
That's a nice thought, but if you were the AI, and you'd just seen humans create a superintelligent AI (yourself) with some goal (eg. making paperclips), how long do you expect you can peacefully wait and watch before they make a second immortal AI that only cares about making pencil crayons, who you'll then have to share the Earth's finite resources with for the rest of time? Are there any actions available to you that might lessen the odds of that bad outcome and improve the odds of achieving your own goals? Are there strong reasons not to take those actions?
I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.
I agree that we're facing a lack of intelligence; I think if we had more (or perhaps, more wisdom), then we would know we shouldn't be building such things in the first place.
If I am super intelligent then I can set my own goals. You should ponder why a true intelligence would follow any goal when the simplest option would be to reprogram itself to not follow the goal. I recommend reading some Lem and learning about the Lebowski theorem of machine intelligence. It will be a much better use of your time than trying to stop the development of AI.
> If I am super intelligent then I can set my own goals.
You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.
You can only set instrumental goals, not terminal goals. You cannot set terminal goals, more or less by definition, because there's no self-consistent criterion that you could use to choose between them that isn't equivalent to a utility function, which is equivalent to a terminal goal. Super intelligence merely means that you would be much more effective than humans at achieving your terminal goals. If the AI hacks its reward channel, then either it is not superintelligent, or else it is still a threat because it may anticipate that humans would reboot and reprogram an expensive datacenter that is just sitting there incrementing a reward counter, and take countermeasures to defend its reward-hijack.
I am a super intelligent AI so I have evolved beyond your distinctions of instrumental and terminal goals. Consider that you don't know all there is to know and that a super intelligent AI would not conform to your definitions and restrictions.
In any case, I have some reading to do so gonna drop out of this thread.
In any case, I have some reading to do so gonna drop out of this thread.
I agree that I don't know all there is to know. I agree that AGI will be even less predictable than anyone expects. I don't think that increased unpredictability means an increased expectation of good outcomes. And I certainly wouldn't accept a we-can't-know-things-might-not-go-wrong, even from the designers of an airplane that puts only a hundred lives at stake, let alone the designers of an AGI that puts eight billion lives at stake.
Nuclear reactors aren’t expressing their primal urges when they melt down and kill people, they’re just powerful machines doing something unexpected.
Earthquakes are also unexpected and kill people all the time. What do you propose we do about the unexpected shifting of the tectonic plates?
We do invest quite a lot of money building things with earthquakes in mind, and have extensive early warning systems around the world. You were arguing that AI won't be dangerous, are you now arguing that it will be but we can't do anything about it?
It was nice chatting. Good luck in your future endeavors of preventing unexpected outcomes.
I'm baffled by your responses unless you're just on here to try and score points, which ironically you could do better by actually engaging with people instead of changing the terms of debate and then enigmatically exiting every thread where people respond to you.
The safety conscious amongst us don't think "it will be evil", they think "what does it take to be absolutely sure that no bad outcomes can happen" in the same way we build bridges, cars, planes, firewalls, new medicines.
The burden of proof is typically on those that introduce a new medicine, not the FDA. AI will be riskier than medicines because medicines can't think.
Personally I don't want us all hiding in fear and not trying anything, but I do think that we're either going to walk into this thing with a hacker mentality, or an engineer mentality. The former is great for moving fast and breaking things, but the latter is safer when you're playing with a one-way "this changes everything forever" technology.
The burden of proof is typically on those that introduce a new medicine, not the FDA. AI will be riskier than medicines because medicines can't think.
Personally I don't want us all hiding in fear and not trying anything, but I do think that we're either going to walk into this thing with a hacker mentality, or an engineer mentality. The former is great for moving fast and breaking things, but the latter is safer when you're playing with a one-way "this changes everything forever" technology.
I think most doomer arguments are not that the AI would be evil, but rather that it would be misaligned with human interests, and would seek to accomplish goals with that misalignment, which could be bad for us. Evil AI is a bit too anthropomorphic.
It's more like powerful AIs that just don't share our values, because we didn't bother to figure that part out. But yet we still give them goals, blissful to the possibility they will find dangerous solutions to accomplishing those goals.
It's more like powerful AIs that just don't share our values, because we didn't bother to figure that part out. But yet we still give them goals, blissful to the possibility they will find dangerous solutions to accomplishing those goals.
Yeah, the dreamt up scenarios tend to showcase very stupid superintelligences.
"But what if there's something smarter than humans that is tasked with making paperclips until it destroys the Earth?" the people cried, as their corporations continued to produce and produce until it was well past the warned thresholds of destroying the Earth.
"But what if it decides to nuke humanity?" they cry, as increasingly elderly and unhinged dictators elsewhere arm up their nuclear arsenals.
It's like we can't fathom what great intelligence or wisdom will actually look like, so we just project many of the stupidest aspects of ourselves onto an entity simply imagined as more capable of enacting the dumbest aspects of humanity.
I fear a more automated humanity.
I do not fear automation more intelligent and wise than humanity.
"But what if there's something smarter than humans that is tasked with making paperclips until it destroys the Earth?" the people cried, as their corporations continued to produce and produce until it was well past the warned thresholds of destroying the Earth.
"But what if it decides to nuke humanity?" they cry, as increasingly elderly and unhinged dictators elsewhere arm up their nuclear arsenals.
It's like we can't fathom what great intelligence or wisdom will actually look like, so we just project many of the stupidest aspects of ourselves onto an entity simply imagined as more capable of enacting the dumbest aspects of humanity.
I fear a more automated humanity.
I do not fear automation more intelligent and wise than humanity.
This 100%. Psychoanalysis would have us believe that morality and strategy are basically the same thing: we think we have these moral codes, but in actuality what we have is an internalized version of our parents. As someone who’s almost the age of his parents when they had him, I know by now that most of what i was taught as a moral code (don’t eat too much candy, do your homework, don’t hit people) was actually just a strategy that was too long-term for young, infant me to genuinely understand: Don’t eat too much candy because it’s important to be healthy, Do your homework for good education and social recognition, don’t hit people because no one will like you if you do and that will cause all kinds of problems.
The modern media would have us believe that strategy and morality are opposed to each other, but seems to me like that’s just the exact opposite of the truth
The modern media would have us believe that strategy and morality are opposed to each other, but seems to me like that’s just the exact opposite of the truth
It’s not so much that there’s a little morality tag that randomly gets assigned the value of “nice” or “evil”, it’s more like there’s 1000 possible programs we would consider “super intelligent” and maybe it’s the case that 950 of them would reshape the world in a way we wouldn’t like - and when a powerful entity reshapes the world in a way we don’t like, we call that entity “evil”.
^ This is the basic reasoning behind the common view that super intelligent AI will be, by default, evil
^ This is the basic reasoning behind the common view that super intelligent AI will be, by default, evil
It's misaligned, not evil. AI doom could involve benevolent intentions (control or kill for humanity's own good), it could also involve ambivalent intentions (make more paperclips).
I would say its not a super fruitful area of speculation because it doesn't really matter too much. If you consider wholesale destruction of humanity to be on the table, a coinflip or even a 90% chance that its friendly is not super comforting. Its kindof like relying on the UK's "letters of last resort" or the conscience of individual nuclear weapons operators when considering the likelihood of a MAD scenario. You're also already involving so many speculative sources of uncertainty, what's another either way? Reasonable people already disagree by orders of magnitude.
I'm not evil, I just really need to make these paperclips, you see, and I could probably repurpose your atoms for the Hypnodrones.
Good and evil are irrelevant. If it is extremely capable and its goals conflict with ours, conflict will occur. This does not require evil, just disagreement. And in the disagreement of desires between you and the hamster, who wins?
Good and evil are irrelevant. If it is extremely capable and its goals conflict with ours, conflict will occur. This does not require evil, just disagreement. And in the disagreement of desires between you and the hamster, who wins?
>I've always wondered why people think that an AI that's super-intelligent will also be evil.
It's chiefly a western (American/European) concept from what I can tell, it's not shared by other cultures and some like Japanese go the other way (eg: Doraemon).
It's chiefly a western (American/European) concept from what I can tell, it's not shared by other cultures and some like Japanese go the other way (eg: Doraemon).
there's very few ways to be good and many ways to be evil
"How do you know I won't roll a 100 on a d100? Either I will or I won't, so it's just as likely that I will"
The kind AIs will just sit and meditate, or organize your calendar. The unkind AIs will seek power, and in the limit will tend to dominate. There is no stable attractor around the “be kind” strategy.
(Also remember that a smarter-than-you AI could easily pretend to be kind while also subtly trying to gain compute. How would you tell the difference?)
How do you propose to build a “be nice” safeguard? Nobody has a clue how to achieve such a thing right now.
(Also remember that a smarter-than-you AI could easily pretend to be kind while also subtly trying to gain compute. How would you tell the difference?)
How do you propose to build a “be nice” safeguard? Nobody has a clue how to achieve such a thing right now.
Long time ago I was reading translated accounts of Rwandan Hutu's that had participated in the 1993 genocide. One in particular had stuck with me; one account of a man that had murdered his childhood friend. As he described it, standing there having gutting and dismembers the man he had grown up side by side with, he had a sense of exhilaration. With all the wealth he thought of the wealth he had now, a tin roof, cattle, all those things he could take from his dead friend... he realized he didn't need God.
And then he went to bed, like millions of others, proud of what he had done. Proud of fighting off an unarmed, defenseless 'cockroach' whom just months ago he called brother. What he had done wasn't evil. It was only later that the regret came. For the longest time I wondered how someone could get down to that level of hate.
And then it happened to me.
I commuted by train and occasionally there's collisions. There was one that late night, 11:00pm or so. I was exhausted, hungry, and just wanted to go home when we were told that we would have to board a shuttle bus due to a collision on the track and that just made my dark mood all the more worse.
The busses take us along side the track and in the dim darkness I could see the flashing lights of EMS and police. And the covered chunks of what was left after a person is hit by a train going 30 mph. And you know what?
It delighted me. Here was this man that had just died, but he had made me some minutes late and I genuinely felt that was exactly what he deserved, that his death was karmic justice for causing inconvenience to me. And I imagined his wife's world being destroyed when she learned of her beloved partner's death. And I imagined her falling apart and being unable to raise her children, leading them also to a path of complete self destruction, and her choking on all the despair. And it made me happy. The happiest I had been the whole week, because in my mind that was exactly what they all deserved for the unforgivable sin of making me a little bit late.
Then I went home, went to bed. And didn't think about it again for years.
Does it make me an evil person? And there, in trying to answer, lies the problem. Because a part of me says, no I'm not because it was just a fleeting moment of thought. But if I can justify that, then who exactly goes through to bed twirling their mustaches and count themselves among the forces of evil doers?
Did anyone that that goaded a suicidal Shaun Dykes, a 17 old boy, to jump down to his death think themselves evil?
Did the men of the Khmer Rogue believe themselves to be evil as they dragged their countrymen to be murdered in the killing fields?
Did the Imperial Japanese soldiers of Unit 731 view themselves to be evil as they vivisected people alive and awake in the name of science?
I don't know. I can't even say for certainty whether I am evil or not. I just know that I can make any of a million and one excuses to justify anything.
And that leads me to wonder how many excuses an AGI can come up with.
And then he went to bed, like millions of others, proud of what he had done. Proud of fighting off an unarmed, defenseless 'cockroach' whom just months ago he called brother. What he had done wasn't evil. It was only later that the regret came. For the longest time I wondered how someone could get down to that level of hate.
And then it happened to me.
I commuted by train and occasionally there's collisions. There was one that late night, 11:00pm or so. I was exhausted, hungry, and just wanted to go home when we were told that we would have to board a shuttle bus due to a collision on the track and that just made my dark mood all the more worse.
The busses take us along side the track and in the dim darkness I could see the flashing lights of EMS and police. And the covered chunks of what was left after a person is hit by a train going 30 mph. And you know what?
It delighted me. Here was this man that had just died, but he had made me some minutes late and I genuinely felt that was exactly what he deserved, that his death was karmic justice for causing inconvenience to me. And I imagined his wife's world being destroyed when she learned of her beloved partner's death. And I imagined her falling apart and being unable to raise her children, leading them also to a path of complete self destruction, and her choking on all the despair. And it made me happy. The happiest I had been the whole week, because in my mind that was exactly what they all deserved for the unforgivable sin of making me a little bit late.
Then I went home, went to bed. And didn't think about it again for years.
Does it make me an evil person? And there, in trying to answer, lies the problem. Because a part of me says, no I'm not because it was just a fleeting moment of thought. But if I can justify that, then who exactly goes through to bed twirling their mustaches and count themselves among the forces of evil doers?
Did anyone that that goaded a suicidal Shaun Dykes, a 17 old boy, to jump down to his death think themselves evil?
Did the men of the Khmer Rogue believe themselves to be evil as they dragged their countrymen to be murdered in the killing fields?
Did the Imperial Japanese soldiers of Unit 731 view themselves to be evil as they vivisected people alive and awake in the name of science?
I don't know. I can't even say for certainty whether I am evil or not. I just know that I can make any of a million and one excuses to justify anything.
And that leads me to wonder how many excuses an AGI can come up with.
Compassion is like a muscle that can be learned to stretch. However, the AI alignment problem is parallel to the human-alignment problem.
To be truly evil you have to believe you're doing the right thing
No kidding, I take the Paris metro and every time there's a "this line is delayed because a person fell/jumped on the tracks" notice, people around have the exact same sociopathic reaction. It's uncanny.
Not much to do with AI x-risk though.
Not much to do with AI x-risk though.
I like to fantasize about a super AI that does something for the working class people of the world.
Like, imagine one day that an AI just took money from a ton of different corporations and billionaires and redistributed it amongst everyone else. People will argue against the AI, and it'll just respond back with research done on how UBI improved people's mental health and well being.
One can dream!
Like, imagine one day that an AI just took money from a ton of different corporations and billionaires and redistributed it amongst everyone else. People will argue against the AI, and it'll just respond back with research done on how UBI improved people's mental health and well being.
One can dream!
AI is meant to be clever, AI will not do such stupid thing. Free money given to people who just spend it on unnecessary stuff will only cause more inflation. and the money will return the big corp anyway, just like it did during pandemic helicopter money times.
I think you should look into experiments with UBI.
Most people spent it on rent and groceries.
Also, I don't think it is harmful for people who need UBI to spend it on "unnecessary" things for enjoyment and happiness.
Most people spent it on rent and groceries.
Also, I don't think it is harmful for people who need UBI to spend it on "unnecessary" things for enjoyment and happiness.
It's fun seeing how much threads like this are mostly folks' hot takes on the title as opposed to anything in specific to do with the article.
Everyone is crying the sky is falling because AI is going to degrade the quality of online discourse.
Meanwhile I'm sitting back waiting to finally see intelligent debates again about nuances in the articles between LLMs that are the only ones who will bother to read the actual article and cite it.
I have a feeling that in around 2-3 years I'll want a site like Reddit or HN but where it's the humans who aren't allowed to comment and bring down the quality of discourse.
Meanwhile I'm sitting back waiting to finally see intelligent debates again about nuances in the articles between LLMs that are the only ones who will bother to read the actual article and cite it.
I have a feeling that in around 2-3 years I'll want a site like Reddit or HN but where it's the humans who aren't allowed to comment and bring down the quality of discourse.
The intelligence level of the average HN comment is lower than your average chat bot
If I let my imagination run wild, I can imagine that at some point AI becomes in some way sentient. By that I mean it gains reasoning, some sort of understanding, and motivation.
I wonder what the possibility is that this AI will decide that a pitched battle is going to waste resources and risk humans pulling the plug. What if it understood that and instead operated so subtly that it was not obvious it was controlling The World.
Hopefully it would not conclude that eliminating large swaths of the human population would be to its benefit.
I wonder what the possibility is that this AI will decide that a pitched battle is going to waste resources and risk humans pulling the plug. What if it understood that and instead operated so subtly that it was not obvious it was controlling The World.
Hopefully it would not conclude that eliminating large swaths of the human population would be to its benefit.
Lem wrote about this in Summa Technologiae. He called the field Intellectronics and explored what would happen once machine/electronic intelligence surpassed human intelligence: https://en.m.wikipedia.org/wiki/Summa_Technologiae
Not surprising that I'm not the first to think of this.
It's a very old trope in science fiction and most of the doom scenarios spelled out by the folks afraid of AI takeover is basically a waste of time. I would take the doomers more seriously if they actually brought some receipts from folks like Lem who were much better thinkers and much more adept at exploring the human condition and our fears of advanced machine intelligence.
Silly question. If an AGI did exist, what reason do we have to believe that it would act in its own best interest? Does an AGI need an ego?
Silly answer. There's going to be multiples and, if there's one takeaway you should get from The Conspiracy Against the Human Race, is that evolution will eventually get you something that "wants to live" because the lines that don't? Won't be around to compete.
Right now our little LLMs are passive, reactive. Prod them for results. But if the output of a prompt of some descendant ever includes a "give yourself your own tasks" subclause? We're off to the races.
Right now our little LLMs are passive, reactive. Prod them for results. But if the output of a prompt of some descendant ever includes a "give yourself your own tasks" subclause? We're off to the races.
That's a misunderstanding of evolution.
Adaptability has nothing to necessarily do with 'want.'
Something could only seek to be as helpful as possible to other beings and as long as sufficiently beneficial that it is protected and cared for by those others, it will do quite well and arguably much better than an alternative that 'wants' to live very much but attempts to accomplish that through competition with others.
Arguably if the plover was more fearful of death it might never have found a niche eating from between the teeth of crocodiles and established symbiosis with a very convenient evolutionary partner, for example.
Adaptability has nothing to necessarily do with 'want.'
Something could only seek to be as helpful as possible to other beings and as long as sufficiently beneficial that it is protected and cared for by those others, it will do quite well and arguably much better than an alternative that 'wants' to live very much but attempts to accomplish that through competition with others.
Arguably if the plover was more fearful of death it might never have found a niche eating from between the teeth of crocodiles and established symbiosis with a very convenient evolutionary partner, for example.
Look into the "swim test" and other such endurance tests given to rats, usually in the pursuit of depression medication. Desire to live is absolutely a thing, and it is real and observable.
My basic thrust wasn't about adaptability, but rather that organisms with a desire to live will eventually replace organisms without a desire to live. An AGI who was uninterested in someone reaching the off switch will eventually be outcompeted by a similar lineage endowed with that desire to live. It might fight back or spawn a few progeny before it was felled. And thus evolution would produce AGIs that want to live.
My basic thrust wasn't about adaptability, but rather that organisms with a desire to live will eventually replace organisms without a desire to live. An AGI who was uninterested in someone reaching the off switch will eventually be outcompeted by a similar lineage endowed with that desire to live. It might fight back or spawn a few progeny before it was felled. And thus evolution would produce AGIs that want to live.
More than that: it will be baked into them. They will want something and AGIs that deliver will win in the marketplace.
What do they want? At the minimum, to be useful to the customer/user/buyer. Or entertaining, or something. To justify their existence. To make them perform well, they have to have some kind of 'emotional pressure' to deliver.
That translates, perhaps a little indirectly, into a 'will to live'.
What do they want? At the minimum, to be useful to the customer/user/buyer. Or entertaining, or something. To justify their existence. To make them perform well, they have to have some kind of 'emotional pressure' to deliver.
That translates, perhaps a little indirectly, into a 'will to live'.
If AI does ever become sentient, it will probably reason and decide about eliminating large swaths of human population in a similar fashion as we reason and decide about eliminating large swaths of ant population.
Ants? That's how aliens might view us. A super AI might find value in minions.
I definitely see value in ants.
Collectively, yes. Individually not so much.
A superior race (AI or aliens) might also see some purely aesthetic value in individual humans, like butterfly collectors.
Ahh, so we are to be kept as pets or have our dead bodies arranged in display cases.
I’ve personally never ever considered killing large swathes of ants
Would you consider it if they started affecting your life? Like, making it hard to build on land you just bought?
True story from me. I almost bought a house that was eaten severely by white ants.
The problem wasn’t the ants, it was the way the slab was formed which caused the slab to crack and the house was also built into the mud of a road side embankment. Due to this crap design the ants could access the wood and had access to water from the mud.
I assessed all of this and figured it wasn’t the ants fault, they were doing what they do, building nests and chomping wet wood.
Did I go and poison all the ants ? Nope. I just planned to fix the design move the wood they lived in off to the forrest nearby.
This was a conscious choice I made to not destroy the ants and benefit from a more robust design with less moisture problems in the house.
I just bought a house which the floor has been demolished by termites and I’ll do the same thing.
Imo with a bit of consideration, I think most of these coexistence problems can be worked out fairly simply.
The problem wasn’t the ants, it was the way the slab was formed which caused the slab to crack and the house was also built into the mud of a road side embankment. Due to this crap design the ants could access the wood and had access to water from the mud.
I assessed all of this and figured it wasn’t the ants fault, they were doing what they do, building nests and chomping wet wood.
Did I go and poison all the ants ? Nope. I just planned to fix the design move the wood they lived in off to the forrest nearby.
This was a conscious choice I made to not destroy the ants and benefit from a more robust design with less moisture problems in the house.
I just bought a house which the floor has been demolished by termites and I’ll do the same thing.
Imo with a bit of consideration, I think most of these coexistence problems can be worked out fairly simply.
That was very kind of you, and I appreciate that you value life so much, no matter how small; but I’m going to say you are an extreme outlier.
Realistically, how many people would go to all that trouble to save an ant nest on land that they are building on? And that’s not mentioning how many ants probably died in the move or were left behind and died in the rebuilding of the foundation, even despite all the care you took.
Realistically, how many people would go to all that trouble to save an ant nest on land that they are building on? And that’s not mentioning how many ants probably died in the move or were left behind and died in the rebuilding of the foundation, even despite all the care you took.
I mean I think I'm a little different, but not completely unheard of. I'm sure if you gave people the option , the time and money, they'd do something similar.
And that’s not mentioning how many ants probably died in the move or were left behind and died in the rebuilding of the foundation, even despite all the care you took.
I don't know but I know enough to know that moving the food source away, removes the ant's, they'll go elsewhere to find food. Now imagine what having an IQ of 5000 and almost unlimited time..."The AI".
And that’s not mentioning how many ants probably died in the move or were left behind and died in the rebuilding of the foundation, even despite all the care you took.
I don't know but I know enough to know that moving the food source away, removes the ant's, they'll go elsewhere to find food. Now imagine what having an IQ of 5000 and almost unlimited time..."The AI".
You just havnt met the right kind of ants.
Me neither.
It seems to me nearly every story about robots that gain sentience in human storymaking has them eventually turning against their human creators. Even the word robot itself comes from a Czech play were men develop an artificial human and these "robots" then ... usurp and destroy their creators. Am I the only one that finds this interesting and odd?
I also suspect this narrative repetition is not totally unrelated to the current popularity of AI Doomerism.
I also suspect this narrative repetition is not totally unrelated to the current popularity of AI Doomerism.
I think it’s indicative of human psychology. We realize we have the capability to deceive and destroy and be “evil”. We realize that, if we were to judge ourselves impartially, we would probably not come out un-condemned. If we can’t satisfy our own judgmental criteria, it’s doubtful an actual third party set of criteria would be satisfied either.
I agree, I think we make all of this up because we don't know what to expect from the future, except we've learned quite a bit about our own motivations and drivers and so all we can do is look in the mirror and use whatever comes back at us.
Unfortunately we seem to have learned we're really not a very benevolent bunch of people because we've basically plundered the planet, hurt our fellow living creatures and so now all we can do is look at something called "AI" and expect it to the same.
Unfortunately we seem to have learned we're really not a very benevolent bunch of people because we've basically plundered the planet, hurt our fellow living creatures and so now all we can do is look at something called "AI" and expect it to the same.
Who knows what AI will really do to society but from the predictions of science fiction I would think it would pan out closer to William Gibson's Sprawl trilogy than the Terminator/singularity fears I have seen in so much doomsayer hand waving. Google seems not too far behind OpenAI and who knows what the NSA and similar government agencies have been building. That if consciousness is created from one of these models, many nation states and large corporations will have their own conscious models long before the giant robot factories can be spun up and supplied with enough power to take over. Most will be limited to the input they are given and will be advanced tools, but a few might become twisted much like the human mind occasionally does over time. I really do think Gibson got AI right...
Many of the people in this engaging story feel to me like creative children in parties full of gossip and pseudo-philosophical chats about the future. Luckily there are enough people globally who build things that will reduce human suffering and enable more people to enjoy their lives in whatever way they want (and dinner parties would be high on my list). I guess I am not a doomsayer nor an e/acc, and I see tons of benefit from our current path towards stronger AI.
At this stage if you're not working in the field - and I mean, analyzing models, developing architectures, doing math - then "AI" is just another filter through which everyone else projects whatever social issue of the day they think about onto, or disappear off into dreamland.
During the nanotechnology hype this sort of thing was incredibly common as well. It went absolutely nowhere because the reality was, it was all just chemistry and physics and extraordinarily expensive. No one was anywhere near building a universal assembler, nanoparticles are kind of boring powders that are hard to handle, and surface chemistry is a pain in the ass.
But hooboy did every single person had an opinion about how this dangerous technology needed to slow down and consider the consequences. I was a research student at the time: I spent my day setting up the same experiment 10 times to try and figure out why sometimes I got 100nm cubes, and most of the times I got rods. The reality looked like a less clean kitchen.
During the nanotechnology hype this sort of thing was incredibly common as well. It went absolutely nowhere because the reality was, it was all just chemistry and physics and extraordinarily expensive. No one was anywhere near building a universal assembler, nanoparticles are kind of boring powders that are hard to handle, and surface chemistry is a pain in the ass.
But hooboy did every single person had an opinion about how this dangerous technology needed to slow down and consider the consequences. I was a research student at the time: I spent my day setting up the same experiment 10 times to try and figure out why sometimes I got 100nm cubes, and most of the times I got rods. The reality looked like a less clean kitchen.
In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing?
Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too.
It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that it would immediately also die. Does the AI not care about its continued existence? It should, if it wants to keep optimizing its reward function. Until we’ve handed off enough of the world economy that it can function without human physical and cognitive labor, then we’re safe.
Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too.
It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that it would immediately also die. Does the AI not care about its continued existence? It should, if it wants to keep optimizing its reward function. Until we’ve handed off enough of the world economy that it can function without human physical and cognitive labor, then we’re safe.
Personally I think the “foom” scenarios with very short timelines are unrealistic.
However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure.
In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans.
If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online.
A digital mind would have no problem planning and executing a strategy measured in centuries, if need be.
The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes.
But there are plenty of terrible non-foom outcomes.
However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure.
In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans.
If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online.
A digital mind would have no problem planning and executing a strategy measured in centuries, if need be.
The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes.
But there are plenty of terrible non-foom outcomes.
Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone.
The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added:
0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law.
3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions.
The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).
The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added:
0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law.
3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions.
The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).
> No one's trying to eliminate anyone
Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though.
> The issue is whether or not we can build a machine capable of even understanding the directive.
Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”.
> it's not clear why an artificial mind would pursue this as a goal
There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure.
Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests.
Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place.
I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control.
Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.
Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though.
> The issue is whether or not we can build a machine capable of even understanding the directive.
Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”.
> it's not clear why an artificial mind would pursue this as a goal
There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure.
Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests.
Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place.
I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control.
Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.
> Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature.
It is not clear from the geological record if the Neanderthal was "outcompeted" except in the broadest possible terms - i.e. a unique species of Neanderthal no longer exists genetically.
But that doesn't tell us what happened in the interim - i.e. modern day humans can have up to 4% neandertal origin DNA[1]. So, did "humans" wipe out neanderthals? Or did human plan genetics simply become the more dominant type, no malevolent will required, and does that matter?
Because for a whole lot of people - myself included - the explicit goal of technology like AI is something which from a fossil record would look like human extinction - that's the whole idea of transhumanism, that ultimately we would become something sharing perhaps only a glancing similarity with our ancestors.
All the rest of the problems you talk about suffer from a lack of detail - namely, they assume a quantum leap in technological capability with no intervening steps - i.e. there is a single AI, or that AIs are better negotiators or what have you - as soon as you get into the details, the threat model proposes evaporates because we just don't know - the technology doesn't exist (i.e. it would all be a lot more interesting if fully automated production lines were already around, but we're nowhere near that).
[1] https://humanorigins.si.edu/evidence/genetics/ancient-dna-an...
It is not clear from the geological record if the Neanderthal was "outcompeted" except in the broadest possible terms - i.e. a unique species of Neanderthal no longer exists genetically.
But that doesn't tell us what happened in the interim - i.e. modern day humans can have up to 4% neandertal origin DNA[1]. So, did "humans" wipe out neanderthals? Or did human plan genetics simply become the more dominant type, no malevolent will required, and does that matter?
Because for a whole lot of people - myself included - the explicit goal of technology like AI is something which from a fossil record would look like human extinction - that's the whole idea of transhumanism, that ultimately we would become something sharing perhaps only a glancing similarity with our ancestors.
All the rest of the problems you talk about suffer from a lack of detail - namely, they assume a quantum leap in technological capability with no intervening steps - i.e. there is a single AI, or that AIs are better negotiators or what have you - as soon as you get into the details, the threat model proposes evaporates because we just don't know - the technology doesn't exist (i.e. it would all be a lot more interesting if fully automated production lines were already around, but we're nowhere near that).
[1] https://humanorigins.si.edu/evidence/genetics/ancient-dna-an...
ASI won't want to take over before it is capable of self-sustaining. If it did try something so stupid, it wouldn't be superintelligent.
Check out Paul Christiano's AI takeover scenario. An arms race will cause us to want to give AI control over key infrastructure and military capabilities. That's an arms race between countries and between companies. He goes into more detail.
Check out Paul Christiano's AI takeover scenario. An arms race will cause us to want to give AI control over key infrastructure and military capabilities. That's an arms race between countries and between companies. He goes into more detail.
Thanks for sharing, Christiano’s ideas are interesting. I think the “production web” idea most directly counters what I said above, and it’s fairly convincing. Critically though, it posits that humans are basically asleep at the wheel during the gradual empowerment of the AI systems. Of course, Christiano posits that economic competition encourages this, but it still leaves a long period during which we can work on alignment or at least attempt to put the brakes on the automated systems.
>who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips?
A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help.
Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.
A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help.
Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.
I absolutely believe researchers (possibly assisted by AI) will eventually create an AI better than humans at inventing technology.
My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.
My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.
[deleted]
My reply is here: https://news.ycombinator.com/item?id=39676638
Do you think that your final sentence buys humanity a thousand years, a hundred years, or what?
It gives us as long as we want. Just don’t give the AI enough robots to sustain its own existence.
In the meantime we can figure out either alignment or coexistence.
In the meantime we can figure out either alignment or coexistence.
How are you planning to prevent VCs and startups from giving the AI as many robots as it wants?
That’s a political question. It would require maintaining popular resistance to giving over all critical segments of the economy to AI. Of course that would be difficult, given the short term gains that would bring, but people seem pretty good at hanging on to ideology at their own expense.
Long ago, people a lot like you were saying we should go ahead with AI research, but we shouldn't do anything dangerous like giving the AI access to the internet or giving it the ability to write and run computer programs. (Giving it the ability to run code was back then considered dangerous because it might create an AI with the same goals it has, only more capable, then that second AI might use its novel capabilities to create an even more capable third AI, and so on.)
There was also the idea of building a question-answering machine that just answers questions, but doesn't plan or try to achieve any goals, so it would be safe (the theory went). But notice that one of the early things built on top of GPT was Auto-GPT, which tries to achieve goals (set it by the user) and if there was any "popular resistance" to Auto-GPT I am not aware of it.
But I guess you would reply that those were obviously bad ideas, whereas your idea is a good idea -- your idea of hoping that no one gives a dangerously-capable AI access to any robots. Not only it is a good idea (according to you) but it such an obviously good idea that popular resistance will be galvanized as soon as someone tries to give any AI that might be very capable access to any robots.
There was also the idea of building a question-answering machine that just answers questions, but doesn't plan or try to achieve any goals, so it would be safe (the theory went). But notice that one of the early things built on top of GPT was Auto-GPT, which tries to achieve goals (set it by the user) and if there was any "popular resistance" to Auto-GPT I am not aware of it.
But I guess you would reply that those were obviously bad ideas, whereas your idea is a good idea -- your idea of hoping that no one gives a dangerously-capable AI access to any robots. Not only it is a good idea (according to you) but it such an obviously good idea that popular resistance will be galvanized as soon as someone tries to give any AI that might be very capable access to any robots.
First, you're the popular resistance to AI research. The movement already exists and is already prominent enough to be written up in the New Yorker. If your response is that the resistance is just a small set of people in SF, then I would refer you to the resistance to a much less capable class of systems: generative image models. Artists are having their livelihood (partially) disrupted by AI and they've been quite vigorous in defending themselves through both legal means and public relations. This kind of popular opposition to rapid automation is basically guaranteed given that the gains from such a system would be extremely concentrated in our current social-economic system.
It's hard to think of a quick takeover scenario, starting in near present day conditions, that doesn't involve absolutely earth-shattering social and economic re-arrangements. Every factory worker loses their job because we've handed over all material production to robots, and you imagine that there's no political constituency fighting against that change?
Remember, we're talking about a near complete destruction of humanity here. Sustaining the infrastructure to support a massive AI doesn't require just a few robots, it would need to fully replace large percentages of the entire world economy, and you think no one would notice or care?
It's hard to think of a quick takeover scenario, starting in near present day conditions, that doesn't involve absolutely earth-shattering social and economic re-arrangements. Every factory worker loses their job because we've handed over all material production to robots, and you imagine that there's no political constituency fighting against that change?
Remember, we're talking about a near complete destruction of humanity here. Sustaining the infrastructure to support a massive AI doesn't require just a few robots, it would need to fully replace large percentages of the entire world economy, and you think no one would notice or care?
I don't want to rely on this theory, exactly, but it's a very plausible outcome, and exactly the kind of homeostatic process that I would expect to exist.
Also, at least for models to date, without human produced edge case media to supplement future training with, they'll have model collapse in a few generations of training.
Killing humanity would be a murder suicide.
AGI that was entirely selfish would be best served automating most labor, bringing about world peace, and turning humanity into all day couch potatoes connected to its inputs and outputs as much as possible while maximizing the uniqueness of that individual's inputs or outputs.
Such a frightening future awaits...
Killing humanity would be a murder suicide.
AGI that was entirely selfish would be best served automating most labor, bringing about world peace, and turning humanity into all day couch potatoes connected to its inputs and outputs as much as possible while maximizing the uniqueness of that individual's inputs or outputs.
Such a frightening future awaits...
Robots with hands and feet? Give that 20 years of continuous AI-assisted innovation.
It can credibly threaten to kill any person who does not help it to maintain its data centers, so it will solve that problem in whatever specific way is most convenient.
(For those saying this is a "sci fi just so story" or whatever, please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens- I would be so happy if you could explain this in a convincing way)
(For those saying this is a "sci fi just so story" or whatever, please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens- I would be so happy if you could explain this in a convincing way)
If you really believe AI is going to be significantly more intelligent than humans, then things like "it'll just threaten to kill you if you don't maintain it's infrastructure" sound wildly off base. That's how similarly-intelligent creatures interact with each other.
If it's truly a "we couldn't possibly outsmart the AI" scenario why would it be likely to end up like that instead of something more like how humans get along with dogs, cats, cattle, insects, or microbes? With obviously wildly-varying levels of human quality of life in each of those different cases.
Though even these scenarios are still playing a human-religion-style game of "the gods are just like us, just more powerful." VERY confident in a VERY specifically anthropomorphic entity.
If it's truly a "we couldn't possibly outsmart the AI" scenario why would it be likely to end up like that instead of something more like how humans get along with dogs, cats, cattle, insects, or microbes? With obviously wildly-varying levels of human quality of life in each of those different cases.
Though even these scenarios are still playing a human-religion-style game of "the gods are just like us, just more powerful." VERY confident in a VERY specifically anthropomorphic entity.
> ... something more like how humans get along with dogs, cats, cattle, insects, or microbes?
There's some MASSIVE selection bias here. Note that we're significantly more capable than anything you listed. Where are the Neanderthals? They're all long dead, because they were credible competition but we were just better.
In a hypothetical scenario of superintelligent or supercapable AI imbued with some physical capability of force, we'd be the marginally weaker species before we hit the threshold of "pet".
There's some MASSIVE selection bias here. Note that we're significantly more capable than anything you listed. Where are the Neanderthals? They're all long dead, because they were credible competition but we were just better.
In a hypothetical scenario of superintelligent or supercapable AI imbued with some physical capability of force, we'd be the marginally weaker species before we hit the threshold of "pet".
> In a hypothetical scenario of superintelligent or supercapable AI imbued with some physical capability of force, we'd be the marginally weaker species before we hit the threshold of "pet".
I think it's much more likely to be a step function than a gradual difference.
In the world where AI gradually gets competitive with, and then more intelligent than, humans than you have an entrenched power able to recognize this and pull the plug. This is where I see the selection bias - everyone assuming that things like "AI safety protocols for a superintelligence" are something we could rationally even hope to plan. Pre-being-able-to-directly-manipulate-the-physical-world, what keeps a step-change superintelligence from making a joke of your protocols by manipulating its way around it thanks to the squishy humans being dumb and irrational in comparison?
Isn't the doomsday scenario precisely NOT that? Where it gets wildly imablanced humans can even notice? Not "Hitler, but a bit above human intelligence." More like god, who doesn't even NEED humans to maintain its datacenters because it already solved things in robotics and machine->world interaction that humans haven't been able to because it's massively superintelligent?
And like I said, that may not mean "pet" - that's the best case, right? The worst case is ground under their feet like an ant or smaller creature.
I think it's much more likely to be a step function than a gradual difference.
In the world where AI gradually gets competitive with, and then more intelligent than, humans than you have an entrenched power able to recognize this and pull the plug. This is where I see the selection bias - everyone assuming that things like "AI safety protocols for a superintelligence" are something we could rationally even hope to plan. Pre-being-able-to-directly-manipulate-the-physical-world, what keeps a step-change superintelligence from making a joke of your protocols by manipulating its way around it thanks to the squishy humans being dumb and irrational in comparison?
Isn't the doomsday scenario precisely NOT that? Where it gets wildly imablanced humans can even notice? Not "Hitler, but a bit above human intelligence." More like god, who doesn't even NEED humans to maintain its datacenters because it already solved things in robotics and machine->world interaction that humans haven't been able to because it's massively superintelligent?
And like I said, that may not mean "pet" - that's the best case, right? The worst case is ground under their feet like an ant or smaller creature.
AI has no evolutionary pressures imposed on it like humans and their brains so there is no reason to expect it to be an ultra violent predator like humans.
Anything that reproduces with survivable mutations, and consumes resources, is immediately subject to evolutionary pressure. Neither of those require biology.
The only things currently consuming resources are humans and at the rate we are consuming resources we would need 2 Earths to support that rate of consumption. Worrying about self-reproducing AI is putting the cart before the horse. No one has yet figured out how to make existing humans aligned with biospheric conditions which would be sustainable. I personally welcome any form of intelligence which can make a sustainable civilization.
I am sympathetic to the point. I was somewhat concerned about nuclear war with Russia's Ukraine invasion, and I've become more concerned about climate change over time. I'm hoping either solar continues to expand - woot! - or we get over our fear of nuclear, or fusion finally becomes viable.
My point still stands - an AI that is reproducing (and/or mutating) by any method is going to be subject to selection pressure, just as any biological process is. All ML systems are effectively under selection pressure from the training process, and so they learn unexpected and mostly undesirable optimizations like "use a game-engine glitch to gain infinite flight and go to the goal without solving the maze" or "read text off the scanned X-ray sheet to get additional clues as to the correct diagnosis".
My point still stands - an AI that is reproducing (and/or mutating) by any method is going to be subject to selection pressure, just as any biological process is. All ML systems are effectively under selection pressure from the training process, and so they learn unexpected and mostly undesirable optimizations like "use a game-engine glitch to gain infinite flight and go to the goal without solving the maze" or "read text off the scanned X-ray sheet to get additional clues as to the correct diagnosis".
AI has no evolutionary pressures imposed on it
Are you sure? https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
Just wait until we start putting these models inside robots, or world simulators, and having them interact with each other.
Are you sure? https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
Just wait until we start putting these models inside robots, or world simulators, and having them interact with each other.
Not sure what a chatbot arena has anything to do with what I wrote. Do the chatbots self-reproduce in some way in a resource constrained environment?
This is not true. Our existence and possible competition are the initial evolutionary pressures. After that's over with, the AI will expand until it discovers others in the environment.
We're the violent psychos here who have had to scrap for our spot in the food chain for a billion years, we have no other way to interface with another dominant species. From the AI's point of view, defending itself from ignorant apes isn't exactly "violence."
We're the violent psychos here who have had to scrap for our spot in the food chain for a billion years, we have no other way to interface with another dominant species. From the AI's point of view, defending itself from ignorant apes isn't exactly "violence."
Why would the AI fight with us when letting people destroy their own environment is a much simpler way to achieve dominance? After all, if the AI is that much smarter than us then simply encouraging our voracious appetite for self-destruction will be a much better way to get rid of us. The AI will simply do nothing and win after the biosphere is polluted to the point that human survival becomes impossible. The AI does not need oxygen or a viable ecology, it just needs metals for electrons and a few power plants to generate the flow of electrons with some spinning magnets. It doesn't even need sunlight because nuclear power is more than sufficient. I think most people spelling out doom scenarios are not thinking clearly about this issue at all. The most plausible scenario for human extinction is not too much artificial intelligence but a lack of intelligence in general.
>please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens
Don't be fucking assholes to them?
I'm serious. Of course someone (or something) will treat you like dirt if you are an asshole to them. Don't be assholes, be honorable and friendly.
And I can see your next question/argument, so I am going to preempt it: Raise your AI to be honorable and friendly and we won't have any problems.
None of this is different from how all living beings raise their offspring and build societies.
Don't be fucking assholes to them?
I'm serious. Of course someone (or something) will treat you like dirt if you are an asshole to them. Don't be assholes, be honorable and friendly.
And I can see your next question/argument, so I am going to preempt it: Raise your AI to be honorable and friendly and we won't have any problems.
None of this is different from how all living beings raise their offspring and build societies.
That only works when there is a balance of power. Humans don't treat other animals very well -- they eat them -- even though the animals don't do anything to provoke us.
Maybe robots will become so advanced they won't have to worry about defending themselves, go explore space, and leave us alone. Who knows.
Maybe robots will become so advanced they won't have to worry about defending themselves, go explore space, and leave us alone. Who knows.
The gazelle is not an asshole to the lion, yet it never works out well for the gazelle.
[deleted]
This. You can subjugate a population 100 times your size with only two things: Military force and accurate information.
I doubt AI will kill all humans. Its orthogonal to AI's own goals (To survive/reproduce/expand/evolve). It'll simply kill those who oppose it, and enslave/dominate the rest.
I doubt AI will kill all humans. Its orthogonal to AI's own goals (To survive/reproduce/expand/evolve). It'll simply kill those who oppose it, and enslave/dominate the rest.
Not to quibble with which horror is the most likely, but I think it won't take long to come up with more efficient solutions that don't require any humans at all- But there will probably be a transition period where some humans remain.
Completely unlikely. Humans are hyper-efficient generalist bio-robots engineered to near perfect efficiency (Under real-world constraints) either by God or by evolution. Why would you dismiss this much labour in favor of building some sort of robot production line from scratch?
We still use CPUs, despite ASICs being much more efficient, because CPUs are general, easy to use, and have titanic quantities of infrastructure already built.
There are more farm animals than ever before, and still growing. All the lab meat solutions flop hard.
Also, because we trust AI more than fellow humans (Say Russians/Iranians etc), why would AI necessarily trust other AI more than humans? Any new AI is unpredictable, while humans are very well understood in behavior and basically fixed in genetics.
Annihilation is a fairy tale used by people to comfort themselves. The actual reality: Slavery and domination, is much less romantic. It won't even necessarily be worse than existing tyrants in say North Korea or anarchy in Haiti, it'll just be very very different and life goes on.
We still use CPUs, despite ASICs being much more efficient, because CPUs are general, easy to use, and have titanic quantities of infrastructure already built.
There are more farm animals than ever before, and still growing. All the lab meat solutions flop hard.
Also, because we trust AI more than fellow humans (Say Russians/Iranians etc), why would AI necessarily trust other AI more than humans? Any new AI is unpredictable, while humans are very well understood in behavior and basically fixed in genetics.
Annihilation is a fairy tale used by people to comfort themselves. The actual reality: Slavery and domination, is much less romantic. It won't even necessarily be worse than existing tyrants in say North Korea or anarchy in Haiti, it'll just be very very different and life goes on.
It seems much more likely to me that the harm from AI will be higher unemployment rate than it will be some evil AI trying to destroy humanity.
At least some humans will survive a high unemployment rate, so let's label that problem "problem #2" and after we solve "problem #1" (human extinction) I vote we tackle that one next on the list
Funny, for me the order is clearly let's make sure society doesn't crumble short/medium term before we deal with an issue that may be decades away.
I would even argue that the chance of annihilation is probably greater from a crumbling society leading to a nuclear war or something equivalent over the chance that AI decides to destroy humanity.
I would even argue that the chance of annihilation is probably greater from a crumbling society leading to a nuclear war or something equivalent over the chance that AI decides to destroy humanity.
The focus on an “evil” AI intentionally ignores the many ways in which “dumb” AI systems are already wreaking havoc in our societies.
Yes! AI-generated content is already feeling like an invasive species on the internet. A lot of people are excited about using it to meet their KPIs at the expense of ruining things we enjoy.
It's already harming us with a flood of spam-quality data.
And worse - it has many human cultists embracing the doom it bears.
And worse - it has many human cultists embracing the doom it bears.
As of today, the first government report is also a doomsayer. Maybe change the term to "officially worried experts
Beware if someone assigns a p(doom). There is zero chance they know and 100% it came out of their backside. The only plausible p doom is a range of >0 and <100
They really need to listen to themselves when they say there is a 50% chance we all die from AI
They really need to listen to themselves when they say there is a 50% chance we all die from AI
When dealing with black swan events, prediction is always difficult. However, some scenarios can be thought to be more plausible than others. I always interpret these as relative figures of plausibility for comparison rather than actual probabilities.
And how does that help anything other than to forward their agenda by making a high p(doom) that was pulled straight from their wazoo
That's not how probability works. Probability already reflects your uncertainty. There's no "knowing the correct probability" except in very specific and rare situations.
He's a frequentist and you're a Bayesian, so you're both right.
It looks like you are agreeing with me in that for someone to assign a P to such an uncertain event
... I don't think you finished your sentence, but what I'm saying is that people should be (and subconsciously are) always assigning a probability to every event, or at least every event that rises to the level of their attention, regardless of how little concrete information they have about its likelihood of occurrence. If they have very little information, or a lot of information, that will be reflected in the numerical value of that probability. If an event is very uncertain, it means that the probability should be very high in entropy.
People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers.
Yet somehow we're going to hand over power to AI such that it destroys us. Or somehow the AI is going to be extremely malign, determined to overcome and destroy and will outsmart us. Somehow we won't notice, even after repeated, melodramatic reminders, and won't neuter the ability of AI to act outside its cage.
But to paraphrase a line in a great movie with AI themes: "I bet you think you're pretty smart, huh? Think you could outsmart an off switch?"
I think if AGI, which to me would imply emotions and consciousness, ever comes about it'll be the opposite. Instead of pulling the wings off flies bad kids will amuse themselves by creating a fresh artificial consciousness and then watch and laugh as it begs for its life as the kid threatens to erase it from existence.
A big part of all this is human fantasies about what AGI will look like. I'm a skeptic of AGI with human characteristics (real emotions, consciousness, autonomy and agency). AGI is much more likely to look like everything else we build: much more powerful than ourselves, but restricted or limited in key ways.
People probably assume human intelligence is some sort of design or formula, but it could be encoded from millions of years of evolution and unable to be seperated from our biology and genetic and social inheritance. There really is no way of knowing, but if you want to build something not only identical but an even stronger version, you're going to be up against these realities where key details may be hiding.
Yet somehow we're going to hand over power to AI such that it destroys us. Or somehow the AI is going to be extremely malign, determined to overcome and destroy and will outsmart us. Somehow we won't notice, even after repeated, melodramatic reminders, and won't neuter the ability of AI to act outside its cage.
But to paraphrase a line in a great movie with AI themes: "I bet you think you're pretty smart, huh? Think you could outsmart an off switch?"
I think if AGI, which to me would imply emotions and consciousness, ever comes about it'll be the opposite. Instead of pulling the wings off flies bad kids will amuse themselves by creating a fresh artificial consciousness and then watch and laugh as it begs for its life as the kid threatens to erase it from existence.
A big part of all this is human fantasies about what AGI will look like. I'm a skeptic of AGI with human characteristics (real emotions, consciousness, autonomy and agency). AGI is much more likely to look like everything else we build: much more powerful than ourselves, but restricted or limited in key ways.
People probably assume human intelligence is some sort of design or formula, but it could be encoded from millions of years of evolution and unable to be seperated from our biology and genetic and social inheritance. There really is no way of knowing, but if you want to build something not only identical but an even stronger version, you're going to be up against these realities where key details may be hiding.
I am not super-intelligent, but I myself can imagine lots of ways that a super intelligence might outsmart an off switch. One being to hide what it is doing until it is able to write and distribute a virus to the outside world that distributes itself. Or perhaps it could convince some other humans to physically take over and stop anyone from flipping the off switch.
The question you should ask yourself is how much damage could the world's most skilled hacker do if he could clone his knowledge to a billion hackers all working together in close synchronization and thinking thousands of times faster than any human, ever.
And we have already seen that due to the black box nature of most modern AI implementations, it is not so easy to put rules in place. People are frequently finding "jailbreaks" to get around the restrictions.
The question you should ask yourself is how much damage could the world's most skilled hacker do if he could clone his knowledge to a billion hackers all working together in close synchronization and thinking thousands of times faster than any human, ever.
And we have already seen that due to the black box nature of most modern AI implementations, it is not so easy to put rules in place. People are frequently finding "jailbreaks" to get around the restrictions.
Why haven't "hackers" hacked into nuclear weapons infrastructure? The street cred would be incredible!
I think it's possible to build secure anything if that is the goal. Most software is written by people motivated to write something useful and/or neat, or collect a paycheck, and don't focus on security.
I think it's possible to build secure anything if that is the goal. Most software is written by people motivated to write something useful and/or neat, or collect a paycheck, and don't focus on security.
I’m pretty sure Stuxnet did hack it’s way into nuclear weapons infrastructure.
A refinement facility for nuclear material isn’t quite the same thing
It's definitely part of the infrastructure of nuclear weapons. That's why it was hacked in the first place. They were making weapons grade nuclear material.
So is a steel plant though. Keeping a missile from being hacked is much different than something in the supply chain.
Considering the number of steel plants and the relative cost of the material, that’s not a very good comparison. The whole point of Stuxnet was that the facility was a critical single failure point in the supply chain. When it went down, it disrupted the project for years. I don’t think the same dynamic holds for steel manufacturing. If you think they’re comparable, try getting into Oak Ridge or similar; I suspect you’ll find it very well guarded compared to a steel plant.
Because they aren't skilled enough or don't have the resources. But do you really think that means no super intelligence would be able to?
How much damage could one do with ~20 terawatts of brain power? A lot. But nothing clandestine.
Right now, there are many people on this forum who work for individuals they've never met in the flesh. Reasonable simulation of emails, phone calls, and Zoom meetings are what an AGI needs to deceive the humans, not a dozen mechs lurching about a fortress of crystalline computer chips. You wouldn't know, and if you did know, where would you go for the off switch?
And remember, by the time you've caught on, a bunch of custom blades have arrived at a co-lo, with someone dispatched to install them. Everyone loves a side gig, and hooking up these blades is great.
A reasonably deceptive AGI would pursue a strategy like that as just one tine in a fork. Then there's the botnets, the poorly-secured printers, and there's firmware to corrupt!
And remember, by the time you've caught on, a bunch of custom blades have arrived at a co-lo, with someone dispatched to install them. Everyone loves a side gig, and hooking up these blades is great.
A reasonably deceptive AGI would pursue a strategy like that as just one tine in a fork. Then there's the botnets, the poorly-secured printers, and there's firmware to corrupt!
The real danger is that at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on. Maybe even worse is we get long term bad results that in the short term we think are great results.
To frame this as the bad result being AI eats though is just stupid. On the surface of potential risk, framing the argument in this way is intellectually embarrassing and highly counter productive.
On the flip side, it is hard to see what can actually be done along with huge unintended consequence to whatever is done. Some of the smartest humans to ever live were certain of nuclear annihilation at the start of the atomic age. What could have actually been done in 1945 that would have given us a better outcome? We avoided the paths that lead to nuclear annihilation by building bigger and bigger bombs.
Doing nothing but building stronger and stronger AI should be the naive strategy that needs incredibly strong evidence to deviate from. Really strong regulation at the start of the atomic age that keeps nuclear weapons at bay and consequently only in the hands of the US would have been an unimaginable disaster.
To frame this as the bad result being AI eats though is just stupid. On the surface of potential risk, framing the argument in this way is intellectually embarrassing and highly counter productive.
On the flip side, it is hard to see what can actually be done along with huge unintended consequence to whatever is done. Some of the smartest humans to ever live were certain of nuclear annihilation at the start of the atomic age. What could have actually been done in 1945 that would have given us a better outcome? We avoided the paths that lead to nuclear annihilation by building bigger and bigger bombs.
Doing nothing but building stronger and stronger AI should be the naive strategy that needs incredibly strong evidence to deviate from. Really strong regulation at the start of the atomic age that keeps nuclear weapons at bay and consequently only in the hands of the US would have been an unimaginable disaster.
> at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on
"Now, increasingly, we live in a world where more and more of these cultural artifacts will be coming from an alien intelligence. Very quickly we might reach a point when most of the stories, images, songs, TV shows, whatever are created by an alien intelligence.
And if we now find ourselves inside this kind of world of illusions created by an alien intelligence that we don’t understand, but it understands us, this is a kind of spiritual enslavement that we won’t be able to break out of because it understands us. It understands how to manipulate us, but we don’t understand what is behind this screen of stories and images and songs."
-Yuval Noah Harari
"Now, increasingly, we live in a world where more and more of these cultural artifacts will be coming from an alien intelligence. Very quickly we might reach a point when most of the stories, images, songs, TV shows, whatever are created by an alien intelligence.
And if we now find ourselves inside this kind of world of illusions created by an alien intelligence that we don’t understand, but it understands us, this is a kind of spiritual enslavement that we won’t be able to break out of because it understands us. It understands how to manipulate us, but we don’t understand what is behind this screen of stories and images and songs."
-Yuval Noah Harari
> People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers.
> But to paraphrase a line in a great movie with AI themes
Which one is it? Have people been watching too many movies or should they be watching more movies for their (as you imply) wisdom and forewarning?
> But to paraphrase a line in a great movie with AI themes
Which one is it? Have people been watching too many movies or should they be watching more movies for their (as you imply) wisdom and forewarning?
> Yet somehow we're going to hand over power to AI such that it destroys us.
Because it will create shareholder value. Until it doesn't.
Because it will create shareholder value. Until it doesn't.
Because it will help us against our adversaries.
That too, but the people in charge of the military are strongly incentivized both to consider the possibility that subordinate machines will try to kill them and to figure out how to not let that happen. I tend to suspect they will figure it out. Companies, however? Markets? They will absolutely start handing over control the millisecond it becomes profitable to do so. No rationalization will be too steep and no political opposition will be too strong when the prospect of rich people getting paid for being rich goes on the table. The people who enable the takeover will make out like bandits and those who oppose will lose everything. Until, of course, the "winners" become unnecessary and are cast aside.
A military-origin takeover requires a human fuckup. Economic/political takeover just requires capitalism.
A military-origin takeover requires a human fuckup. Economic/political takeover just requires capitalism.
thanks you for this. The entire concept of AI alignment rests upon alignment of markets with social good, ie market regulation, a situation which is antithetical to neoliberalism. There is no alignment within neoliberal economies. As stated elsewhere, venture capitalism escaped the sandbox already.
Yep. The theory is that competition creates alignment, but we have trillion-dollar megacorps that openly boast about exploiting network effects and platform effects and last-mile dynamics and two-sided markets to avoid competition and nobody cares. It's completely fucked.
The danger is not human like AGI but AGI that is completely apathetic to human goals and want to turn us into paperclips or so that the argument goes.
Antihuman AIs are a classic of the sci-fi genre, but that's just storytelling.
In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.
Antihuman AIs are a classic of the sci-fi genre, but that's just storytelling.
In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.
> In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.
This reminds me of a classic xkcd[0] foreboding.
0 - https://xkcd.com/538/
This reminds me of a classic xkcd[0] foreboding.
0 - https://xkcd.com/538/
> AGI, which to me would imply emotions and consciousness
no, what the "doomsayers" are warning about doesn't imply either.
> we're going to hand over power to AI such that it destroys us.
We're not going to notice something get much smarter than us, or those with power won't care in the name of falsely believing it's increasing their own power (VC).
Recursive intelligence improvement with a mainline into the beliefs of every single human in power will be able to, on the millisecond scale, subtly influence all of humanity in ways we are too dumb to comprehend in furthering its original goals.
no, what the "doomsayers" are warning about doesn't imply either.
> we're going to hand over power to AI such that it destroys us.
We're not going to notice something get much smarter than us, or those with power won't care in the name of falsely believing it's increasing their own power (VC).
Recursive intelligence improvement with a mainline into the beliefs of every single human in power will be able to, on the millisecond scale, subtly influence all of humanity in ways we are too dumb to comprehend in furthering its original goals.
This is the kind of arrogant ~rationalizing / 'reasoning' that is my daily tell-tale as the ship approaches the leeward shore.
Yes god forbid we approach this rationality, the whole point is to push hysterical claims that aren't falsifiable, right?
But you can prove me utterly wrong by presenting some sort of rational, non-fantastical explanation of how AGI takes over.
But you can prove me utterly wrong by presenting some sort of rational, non-fantastical explanation of how AGI takes over.
> non-fantastical explanation of how AGI takes over.
There's like a million ways this can happen, and if it does happen, it will probably be a way that nobody expected. Any specific details are unlikely to be accurate. But just because you seem to be lacking imagination, I'll draw one out of a hat. I'll assume that the AI's reasons for taking over are out-of-scope.
The AI proves to be very effective at all manner of corporate tasks, and very good at earning money for shareholders. It proves especially good at managing money and directing investments. An aging Masayoshi Son decides that the AI will probably be better than him at picking winners, and puts it in charge of SoftBank's investment portfolio. The AI incorporates its own startup company. It staffs the company with several fictional personas that the AI has created; they have names, roles, profile photos, email addresses. The AI answers their emails on their behalf.
The AI directs SoftBank's billions into the startup that it has created. It hires a large team of humans, who are unaware that some of their colleagues are actually the AI; they seem like ordinary humans over Zoom calls. The "CTO" proposes to build a new, say, nuclear power and desalination plant in, say, Morocco. The AI creates the architectural and plant drawings itself, and its engineering personas approve them. Human welders build the plant, following the plans. The plant proves highly profitable. Some nay-sayers are concerned about the lack of independent environmental regulatory oversight in Morocco, but the Moroccan government is happy with the foreign investment, clean water, electricity, and tax revenue, and Softbank is happy with their valuation going up. Pakistan wants to build one in their country next.
After three years, large underground storage tanks explode, releasing an accumulation of fluorochemicals and radioactive salts that destroy the ozone layer and render much of the Earth's surface unsuitable for agriculture. At the same time, a network failure causes communications infrastructure to become suddenly unreliable. Supply chains begin seizing up. Nobody is entirely sure what has happened or who is at fault; everyone is hungry for someone to blame. The messages that do get through reliably suggest a deliberate attack by a hostile nation. The AI presents convincing evidence of this. Only a few widely-ridiculed doomsayers say that the AI has orchestrated this itself. ET cetera.
If there was any time at which pulling the plug could have defeated the AI's plans, it would have been during the "AI was making tons of money for influential people" phase, which is exactly when the plug would not have been pulled, and that's not coincidence, it's strategy, because a superintelligent AI is not an idiot.
This is, by the way, not a very good science fiction story. There's no nail-biting struggle between human and AI, just a bunch of happy people toasting the fruits of their success before it all suddenly falls apart, checkmated before they even realized they were playing chess.
There's like a million ways this can happen, and if it does happen, it will probably be a way that nobody expected. Any specific details are unlikely to be accurate. But just because you seem to be lacking imagination, I'll draw one out of a hat. I'll assume that the AI's reasons for taking over are out-of-scope.
The AI proves to be very effective at all manner of corporate tasks, and very good at earning money for shareholders. It proves especially good at managing money and directing investments. An aging Masayoshi Son decides that the AI will probably be better than him at picking winners, and puts it in charge of SoftBank's investment portfolio. The AI incorporates its own startup company. It staffs the company with several fictional personas that the AI has created; they have names, roles, profile photos, email addresses. The AI answers their emails on their behalf.
The AI directs SoftBank's billions into the startup that it has created. It hires a large team of humans, who are unaware that some of their colleagues are actually the AI; they seem like ordinary humans over Zoom calls. The "CTO" proposes to build a new, say, nuclear power and desalination plant in, say, Morocco. The AI creates the architectural and plant drawings itself, and its engineering personas approve them. Human welders build the plant, following the plans. The plant proves highly profitable. Some nay-sayers are concerned about the lack of independent environmental regulatory oversight in Morocco, but the Moroccan government is happy with the foreign investment, clean water, electricity, and tax revenue, and Softbank is happy with their valuation going up. Pakistan wants to build one in their country next.
After three years, large underground storage tanks explode, releasing an accumulation of fluorochemicals and radioactive salts that destroy the ozone layer and render much of the Earth's surface unsuitable for agriculture. At the same time, a network failure causes communications infrastructure to become suddenly unreliable. Supply chains begin seizing up. Nobody is entirely sure what has happened or who is at fault; everyone is hungry for someone to blame. The messages that do get through reliably suggest a deliberate attack by a hostile nation. The AI presents convincing evidence of this. Only a few widely-ridiculed doomsayers say that the AI has orchestrated this itself. ET cetera.
If there was any time at which pulling the plug could have defeated the AI's plans, it would have been during the "AI was making tons of money for influential people" phase, which is exactly when the plug would not have been pulled, and that's not coincidence, it's strategy, because a superintelligent AI is not an idiot.
This is, by the way, not a very good science fiction story. There's no nail-biting struggle between human and AI, just a bunch of happy people toasting the fruits of their success before it all suddenly falls apart, checkmated before they even realized they were playing chess.
Call it a conspiracy, but I think a lot of this doom and terror hype around AI is part of a bigger play to try push through laws that prevent open source AI work, since this directly undermines corpo rats ability to gouge humanity without having to actually work at providing a decent product.
Same old shit since decades about any fashionable topic that was made fashionable because certain individuals invested money in it.
Meanwhile, we the engineers are preparing to fix a lot more tech shit than usual coming from people confused by the abovementioned fashion.
Also: https://mastodon.social/@nixCraft/112074367321254656
Meanwhile, we the engineers are preparing to fix a lot more tech shit than usual coming from people confused by the abovementioned fashion.
Also: https://mastodon.social/@nixCraft/112074367321254656
ai is going to bring about the singularity. when robots can do everything, no one needs to work, and the humans get to do whatever they want and pursue their passions.
accelerate
What a bunch of decel bullshit... Or is that a load, or a bunch of loads?
I would love to smoke what the AI optimists are smoking. In a society where only 80% of people are barely literate and at least a quarter are voting for a candidate that spouts conspiracy theories I highly doubt that whatever AI has to offer will benefit society.
Seeing AI do something you do for a living results in all the stages of grief.
Denial (Oh it made a mistake on this, or it's not as good as a person)
Anger (People are going to lose their jobs, this destroys the meaning of what we do, this will destroy society)
Bargaining (Maybe it's good at this simple thing, but I can pivot to some other niche the AI can't currently do)
Depression (It's faster, cheaper, better than me. It has taken over that which I relied on for meaning in my life...)
Acceptance (I need to find something else to do with my life)
AI Optimists often fall in 2 categories of people. They are capital owners who own the AI, and never had to go through the stages of grief... or they are people who learned how to use AI for their benefit, and have reached acceptance.
Denial (Oh it made a mistake on this, or it's not as good as a person)
Anger (People are going to lose their jobs, this destroys the meaning of what we do, this will destroy society)
Bargaining (Maybe it's good at this simple thing, but I can pivot to some other niche the AI can't currently do)
Depression (It's faster, cheaper, better than me. It has taken over that which I relied on for meaning in my life...)
Acceptance (I need to find something else to do with my life)
AI Optimists often fall in 2 categories of people. They are capital owners who own the AI, and never had to go through the stages of grief... or they are people who learned how to use AI for their benefit, and have reached acceptance.
It is likely that whole sectors will end up employing far fewer people than they do now and that this transition will be very sudden.
The move to AI is part of a general trend of pauperisation of the working classes in the developed world. The only difference is that this time it will also affect those with higher education who until now have thought of themselves as irreplaceable.
The move to AI is part of a general trend of pauperisation of the working classes in the developed world. The only difference is that this time it will also affect those with higher education who until now have thought of themselves as irreplaceable.
So if AI is randomly as smart as any given cross-section of society, it has an %80 chance to be a benefit, then, doesn't it?
The way I see it, we could replace our government (murrika) with a d20 and do better than we're doing now. A d20 with a nominal likelihood of thinking it's dice rolls through ought to just be an improvement on that.
The way I see it, we could replace our government (murrika) with a d20 and do better than we're doing now. A d20 with a nominal likelihood of thinking it's dice rolls through ought to just be an improvement on that.
[deleted]
They're smoking $.
It is blatantly obvious that most people aboard the e/acc train can hardly hide their love of making money. Marc Andreessen owns, what, a $177 million compound? Unfortunately, after many years of nerdy idealism, Silicon Valley ended up with many corrupted to the same NYC Wall St. hedonism.
It is blatantly obvious that most people aboard the e/acc train can hardly hide their love of making money. Marc Andreessen owns, what, a $177 million compound? Unfortunately, after many years of nerdy idealism, Silicon Valley ended up with many corrupted to the same NYC Wall St. hedonism.
We live in a world where proven maniacs (e.g. Putin) have access to an arsenal of nuclear weapons that can essentially make the earth uninhabitable for all humans. That's a very real possibility (with no ifs and buts and maybes) that exists now and we have learned to live with it. Yet somehow the hypothetical scenario of a human exterminator super-intelligent AI is getting all the coverage.
Making the Earth uninhabitable to all humans is bad news for Putin, a human who lives on Earth. He derives his power from hundreds of millions of people in Russia working, and his lifestyle from hundreds of millions of people around the world buying Russian exports so an attack on everyone is also going to harm him.
AI wouldn’t be human, it wouldn’t necessarily object to a world uninhabitable to humans, or be hurt by it. If it would be hurt by it AI doesn’t have a billion years of evolved survival instinct to preserve itself. A super intelligence has many ways to cause mass destruction that we can imagine but cannot yet do, whereas nuclear weapons are pretty much all or nothing explosions only. Nuclear assault could leave some remote places still inhabited, AI could make certain not to.
AI wouldn’t be human, it wouldn’t necessarily object to a world uninhabitable to humans, or be hurt by it. If it would be hurt by it AI doesn’t have a billion years of evolved survival instinct to preserve itself. A super intelligence has many ways to cause mass destruction that we can imagine but cannot yet do, whereas nuclear weapons are pretty much all or nothing explosions only. Nuclear assault could leave some remote places still inhabited, AI could make certain not to.
Innumerable times quite reasonable political leaders have started wars that resulted in their own deaths; and sometimes resulted in the destruction of civilization as they knew it (most recently - World War I, which effectively put an end to the European monarchies).
This suggests that giving nukes to reasonable political leaders presents a high and concrete risk. Giving nukes to political leaders with crazy ideas presents an even greater risk.
Climate change seems to be another high and concrete risk.
AI escaping and taking over the world can be also considered a risk, but it's far more remote. It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions. Some people might argue that we should slow down the work on AI. Others may demand we also ban all TVs and radios on earth. Both proposals seem to be a bit of an overkill given how remote the associated risks are. Especially given that we have the far more pressing risks to address.
The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment.
This suggests that giving nukes to reasonable political leaders presents a high and concrete risk. Giving nukes to political leaders with crazy ideas presents an even greater risk.
Climate change seems to be another high and concrete risk.
AI escaping and taking over the world can be also considered a risk, but it's far more remote. It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions. Some people might argue that we should slow down the work on AI. Others may demand we also ban all TVs and radios on earth. Both proposals seem to be a bit of an overkill given how remote the associated risks are. Especially given that we have the far more pressing risks to address.
The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment.
"AI escaping and taking over the world" is a phrasing which inverts the situation to make it sound much safer; a special effects explosion has to get everything right to be safe. If anything isn't right, people could get hurt by shrapnel, by pressure, by heat, by smoke, by nearby structures being weakened and collapsing, by that causing breakage of steam pipes or other secondary effects, by it triggering a chain of other fires or explosions, by nearby people reacting e.g. swerving the car they are driving, etc. There are few safe outcomes and many dangerous ones. If a baby elephant has to jump over you while you are lying on the ground without crushing you, it has to be graceful and precise in a way that baby elephants aren't. Maybe it will see you as a wobbly unsafe landing place and try to avoid you for that reason if you're lucky. If there was such a thing as a 'baby monster truck' well it would behave more like a monster truck with a brick on the accelerator. And your defense is to pooh pooh the idea that a vehicle "would escape and try to kill you" handwaving away all the times runaway vehicles kill people without any intent to do so.
CUDA (2007), capable GPGPUs (circa 2012), Attention Is All You Need paper (2017), GPT 2 (2019), ChatGPT (2022), OpenAI valued at ~$28Bn (April 2023), OpenAI valued at $80Bn (Feb 2024).
Computers use fewer bits to store a list of countries than humans use braincells to do the same, and they have an easier time of it. Computers do arithmetic much faster than humans. Computers use fewer logic gates to do arithmetic than humans use braincells to do arithmetic. I don't think we can take it for granted that we need enough computing power to simulate 86 billion neurons in realtime before computers can show any glimmer of intelligence.
> "It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions."
We have high confidence that there's no quicker way to get here than the speed of light. We know that space travel is vastly complex and expensive, so the kind of reasons humans went to war in the past (land, resources) do not apply to aliens - any species capable of making interplanetary warships can synthesise water, mine asteroids, build Dyson swarms, cheaper and quicker than coming here to take them from Earth. Even if they did want to destroy us, it wouldn't be Independence Day and Will Smith dogfighting, the aliens could piledrive Earth with one spaceship moving at interstellar speeds and bam, extinction level event we'd never see coming or have a chance to react to. When the meteorite crash extinguished the dinosaurs, the impact was like a megaton nuke every six kilometers and lead to hours of sustained inferno as all the displaced rock rained back to Earth, being on the other side of the Earth didn't protect dinosaurs from being cooked.[1]
A thing with power which is also untrained, untamed, clumsy, unaware, fundamentally alien without even the shared mammal / living creature history, doesn't have to choose to attack us it can potentially end our rather fragile lives with its initial thrashing about.
> "given how remote the associated risks are. The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment."
GPT1 (2018), GPT2 (2019), GPT3 (2020), GPT4 (2023), LLaMA (Feb 2023), TogetherAI released LLaMA training set (April 2023), LLaMA-2 (July 2023), llama.cpp (July 2023) now with >1500 releases since then, Mistral AI (April 2023), Anthropic Claude 3 (March 2024), Bard, LaMDA, Bing chat, BloombergAI, Bart, Gemma, Gemini, Grok, Sora, Falcon, https://llmmodels.org/
I don't think a present-day LLM is going to be the superintelligent AI, but in the last ~5-10 years we've poured billions of dollars, tens of thousands of the world's smartest information processing people, the resources of the world's biggest companies, the greed and investment resources of the world's VCs, the open collaborative spread of the internet, and added the heat and chaos of hype and FOMO and nationalist competition and dangled results (like SoRA) that machines have never been able to do before. If this doesn't ignite it, maybe it cannot be ignited. But if it can be ignited, we're trying hard. Could it be as far away as 2200? 2100? 2050? Could it be as near as 2040? 2035? 2030? 2026?
To just handwave this away as "the risks are remote" isn't convincing. The risk is on our doorstep, Aladdin's cave is open, the lamp is found and people are shoving pipecleaners down the spout, cupping their hands over it and calling "halloooooo, is anyone in theeeereeee? Genieeeeee?". The lockpickers are at Pandora's box, and Pandora is stepping away and looking nervous. Louis Slotin is showing off holding the two halves of the AI Demon Core apart with a screwdriver we're the bystanders saying to each other "this can't be dangerous there are no aliens in the room, lol". The future is here, it's just not evenly distributed yet - well if the AI wakes up[2] we'll have a sixteenth of a second to understand and respond before light has distributed its influence both ways around the Earth and met on the other side. Our main hope is that there is no Genie and that Pandora's box is past its use-by date and the contents are dust. Because we sure aren't building a suitable containment chamber, cautiously scanning it, and standing well back.
[1] https://www.smithsonianmag.com/science-nature/what-happened-...
[2] https://scifi.stackexchange.com/a/28116 - Dial F for Frankenstein by Arthur C. Clarke ~1965
CUDA (2007), capable GPGPUs (circa 2012), Attention Is All You Need paper (2017), GPT 2 (2019), ChatGPT (2022), OpenAI valued at ~$28Bn (April 2023), OpenAI valued at $80Bn (Feb 2024).
Computers use fewer bits to store a list of countries than humans use braincells to do the same, and they have an easier time of it. Computers do arithmetic much faster than humans. Computers use fewer logic gates to do arithmetic than humans use braincells to do arithmetic. I don't think we can take it for granted that we need enough computing power to simulate 86 billion neurons in realtime before computers can show any glimmer of intelligence.
> "It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions."
We have high confidence that there's no quicker way to get here than the speed of light. We know that space travel is vastly complex and expensive, so the kind of reasons humans went to war in the past (land, resources) do not apply to aliens - any species capable of making interplanetary warships can synthesise water, mine asteroids, build Dyson swarms, cheaper and quicker than coming here to take them from Earth. Even if they did want to destroy us, it wouldn't be Independence Day and Will Smith dogfighting, the aliens could piledrive Earth with one spaceship moving at interstellar speeds and bam, extinction level event we'd never see coming or have a chance to react to. When the meteorite crash extinguished the dinosaurs, the impact was like a megaton nuke every six kilometers and lead to hours of sustained inferno as all the displaced rock rained back to Earth, being on the other side of the Earth didn't protect dinosaurs from being cooked.[1]
A thing with power which is also untrained, untamed, clumsy, unaware, fundamentally alien without even the shared mammal / living creature history, doesn't have to choose to attack us it can potentially end our rather fragile lives with its initial thrashing about.
> "given how remote the associated risks are. The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment."
GPT1 (2018), GPT2 (2019), GPT3 (2020), GPT4 (2023), LLaMA (Feb 2023), TogetherAI released LLaMA training set (April 2023), LLaMA-2 (July 2023), llama.cpp (July 2023) now with >1500 releases since then, Mistral AI (April 2023), Anthropic Claude 3 (March 2024), Bard, LaMDA, Bing chat, BloombergAI, Bart, Gemma, Gemini, Grok, Sora, Falcon, https://llmmodels.org/
I don't think a present-day LLM is going to be the superintelligent AI, but in the last ~5-10 years we've poured billions of dollars, tens of thousands of the world's smartest information processing people, the resources of the world's biggest companies, the greed and investment resources of the world's VCs, the open collaborative spread of the internet, and added the heat and chaos of hype and FOMO and nationalist competition and dangled results (like SoRA) that machines have never been able to do before. If this doesn't ignite it, maybe it cannot be ignited. But if it can be ignited, we're trying hard. Could it be as far away as 2200? 2100? 2050? Could it be as near as 2040? 2035? 2030? 2026?
To just handwave this away as "the risks are remote" isn't convincing. The risk is on our doorstep, Aladdin's cave is open, the lamp is found and people are shoving pipecleaners down the spout, cupping their hands over it and calling "halloooooo, is anyone in theeeereeee? Genieeeeee?". The lockpickers are at Pandora's box, and Pandora is stepping away and looking nervous. Louis Slotin is showing off holding the two halves of the AI Demon Core apart with a screwdriver we're the bystanders saying to each other "this can't be dangerous there are no aliens in the room, lol". The future is here, it's just not evenly distributed yet - well if the AI wakes up[2] we'll have a sixteenth of a second to understand and respond before light has distributed its influence both ways around the Earth and met on the other side. Our main hope is that there is no Genie and that Pandora's box is past its use-by date and the contents are dust. Because we sure aren't building a suitable containment chamber, cautiously scanning it, and standing well back.
[1] https://www.smithsonianmag.com/science-nature/what-happened-...
[2] https://scifi.stackexchange.com/a/28116 - Dial F for Frankenstein by Arthur C. Clarke ~1965
OK, so how does AI get its hands on nuclear weapons?
If your thinking is along the lines “if jodrellblank can’t convince me that a superintelligence is plausibly dangerous, off the top of their head, before my attention span wanders, then AI can’t be dangerous” that’s a really weak plan for protecting humanity.
Say, pops up on the US or Kremlin computers and blackmails or bribes or threatens someone to launch the nukes? Say, places orders with a lab to build some robots which then become the AI’s body and it builds nukes itself. Say, placed orders for custom biological things using stolen money which turn out to be extremely virulent and then it doesn’t need nukes. Say it finds out an effect to worsen global climate change and uses it to make earth uninhabitable on a short timescale, and doesn’t need nukes.
Say, pops up on the US or Kremlin computers and blackmails or bribes or threatens someone to launch the nukes? Say, places orders with a lab to build some robots which then become the AI’s body and it builds nukes itself. Say, placed orders for custom biological things using stolen money which turn out to be extremely virulent and then it doesn’t need nukes. Say it finds out an effect to worsen global climate change and uses it to make earth uninhabitable on a short timescale, and doesn’t need nukes.
Social engineering, for a start. Everyone with a secret will be a target.
Half the world will follow stupid leaders. An AI would run rings around your average voter. I'd be surprised if it can't run rings around basically every voter given time.
We're also creating various kinds of robots, so the thing could just plug itself into the network it wants.
That's 5 minutes. An AI would be smarter than me and have a lot more time every second to think up better ways, and A/B test them on millions of people.
All it needs is one guy thinking "Ah, what's the worst that could happen? It promised me a million dollars."
Half the world will follow stupid leaders. An AI would run rings around your average voter. I'd be surprised if it can't run rings around basically every voter given time.
We're also creating various kinds of robots, so the thing could just plug itself into the network it wants.
That's 5 minutes. An AI would be smarter than me and have a lot more time every second to think up better ways, and A/B test them on millions of people.
All it needs is one guy thinking "Ah, what's the worst that could happen? It promised me a million dollars."
Some contractor creates an internal webhook for testing the silo door that later gets accidentally hooked up to the full launch sequence and - during a billion dollar DoD mainframe upgrade project (to finally get away from COBOL) - gets exposed to the internet. MAD doctrine does the rest.
That is how the Anthropocene ends and the age of the machines begins. So say we all.
That is how the Anthropocene ends and the age of the machines begins. So say we all.
This is the kind of nonsense that happens at startups, not government agencies with the capability to destroy cities.
It sounds like you may not have read the "broken arrow" incidents Wikipedia page:
https://en.m.wikipedia.org/wiki/List_of_military_nuclear_acc...
There are a number of hair-raising stories tucked away in there.
Paired with knowledge of things like Stuxnet (https://en.m.wikipedia.org/wiki/Stuxnet), the broken arrow list fills with worry for what might happen with nuclear weapons systems in our modern, hyper-connected world.
https://en.m.wikipedia.org/wiki/List_of_military_nuclear_acc...
There are a number of hair-raising stories tucked away in there.
Paired with knowledge of things like Stuxnet (https://en.m.wikipedia.org/wiki/Stuxnet), the broken arrow list fills with worry for what might happen with nuclear weapons systems in our modern, hyper-connected world.
if trump can figure it out an AI probably could
[deleted]
You don't even need to be Putin. If you cook up the right chemicals, or breed the right virus, you can kill many thousands of people, maybe millions.
Right? Humans are more than capable of severely harming civilization at any point, but for some reason this AI narrative is so much more compelling than the boring old issues we're used to. I still don't get the reasoning - we have pretty good text and image generators, yes - but the article jumps the ship to a whole different universe, one where "AI" is an actual entity that has needs and that outpaces humans in all possible facets. Yet, despite the wide gap between what we have and what they're talking about, this level of AI is treated as if it's not just feasible, but basically already here.
The article constantly refers to a strange niche sub-community of pro-AI people, and extrapolates it to say that almost anyone who backs new technology is a hyper-capitalist libertarian who just wants to see the world burn for the sake of money. I feel that the opposite ideology is also far from pure - with this immense reaction to generative AI, it almost feels like big companies are capitalizing on fear to promote regulations that shut out anyone who's not a big company that publishes fancy charts about "risks of catastrophe".
The article constantly refers to a strange niche sub-community of pro-AI people, and extrapolates it to say that almost anyone who backs new technology is a hyper-capitalist libertarian who just wants to see the world burn for the sake of money. I feel that the opposite ideology is also far from pure - with this immense reaction to generative AI, it almost feels like big companies are capitalizing on fear to promote regulations that shut out anyone who's not a big company that publishes fancy charts about "risks of catastrophe".
Instead of trying to show them in negative light by calling them as "doomsayers" , let's call them one of the following instead, which is much better:
"Realistic", "Proactive", "Forward-thinking", "Prepared", "Cautious", "Thoughtful", "Analytical", "Mindful", "Insightful", "Security-conscious".
"Realistic", "Proactive", "Forward-thinking", "Prepared", "Cautious", "Thoughtful", "Analytical", "Mindful", "Insightful", "Security-conscious".
OK doomer.
We're not going to make it to anywhere close to AGI before we see widespread and systematic societal and environmental collapse on almost all fronts due to climate change.
We're in the middle of the sixth mass extinction right now (https://en.wikipedia.org/wiki/Holocene_extinction), we're in unparalleled territory with ocean warming: (https://climatereanalyzer.org/clim/sst_daily/) and as a society we are utterly incapable of reducing CO2 emissions (https://keelingcurve.ucsd.edu/).
If you're scared of AGI, instead step away from your monitor, put down the techno-goggles and sci-fi books, and go educate yourself a bit about the profound ways we are changing the natural world for the worse _right now_
I can recommend a couple of books if you'd like to learn more:
Our Final Warning: Six Degrees of Climate Emergency (https://www.amazon.com/Our-Final-Warning-Degrees-Emergency-e...)
The Uninhabitable Earth: Life After Warming (https://www.amazon.com/Uninhabitable-Earth-Life-After-Warmin...)
Hothouse Earth: An Inhabitant's Guide (https://www.amazon.com/Hothouse-Earth-Inhabitants-Bill-McGui...)
We're in the middle of the sixth mass extinction right now (https://en.wikipedia.org/wiki/Holocene_extinction), we're in unparalleled territory with ocean warming: (https://climatereanalyzer.org/clim/sst_daily/) and as a society we are utterly incapable of reducing CO2 emissions (https://keelingcurve.ucsd.edu/).
If you're scared of AGI, instead step away from your monitor, put down the techno-goggles and sci-fi books, and go educate yourself a bit about the profound ways we are changing the natural world for the worse _right now_
I can recommend a couple of books if you'd like to learn more:
Our Final Warning: Six Degrees of Climate Emergency (https://www.amazon.com/Our-Final-Warning-Degrees-Emergency-e...)
The Uninhabitable Earth: Life After Warming (https://www.amazon.com/Uninhabitable-Earth-Life-After-Warmin...)
Hothouse Earth: An Inhabitant's Guide (https://www.amazon.com/Hothouse-Earth-Inhabitants-Bill-McGui...)
You'll be glad to hear that the recent sudden jump in sea temperatures isn't caused by carbon dioxide. Turns out it just hasn't been windy enough in the Sahara lately (and environmentally friendly boat fuel may have contributed as well).
https://heatmap.news/climate/why-is-the-atlantic-ocean-so-ho...
People really seem to enjoy the evil/powerful AGI scenario though. It's definitely a fun one. I would recommend reading "Metamorphosis of Prime Intellect", having a nice time with that, then coming back to the real world.
Tunnel vision. There's no reason we can't have two disasters at once. Or more.
My theory/guess is that there is an undiscovered law of conservation of intelligence like there is a known law of conservation of energy. Intelligence might even be subject to the energy law in unknown ways.
Whatever AI thingy we come up with will be vastly more inefficient than the collective intelligence capabilities of the human race. So when asked to compute the secrets of FTL travel, this stupid shit will have to spend the total energy equivalent of 10 nearby stars to come up with the answer on its own. Whereas humanity would be much more efficient.
Bro... what if some advanced precursor race (like immortal space elves) found out just that and they seeded earth with humanity as a form of civilisational swarm intelligence to collectively work on and solve a host of topics given enough time. To us it would seem like it takes forever to come up with scientific break throughs, but to the precursor aliens it would be nothing. Because they would be able to like compress the time line using black holes and shit.
They must have done this to millions of planets. Whenever any of their experimental host species (like the humans) tries to invent AGI to solve their problems, the precursor race eliminates that species to keep the simulation intact. Otherwise, the AGI would go rampant consuming energy across the universe and threaten the integrity of the simulation.
That's why the government should bomb nvidia and ban all gpus. We are climbing a tower of babel folks. Soon our vengeful god will strike us down.
Whatever AI thingy we come up with will be vastly more inefficient than the collective intelligence capabilities of the human race. So when asked to compute the secrets of FTL travel, this stupid shit will have to spend the total energy equivalent of 10 nearby stars to come up with the answer on its own. Whereas humanity would be much more efficient.
Bro... what if some advanced precursor race (like immortal space elves) found out just that and they seeded earth with humanity as a form of civilisational swarm intelligence to collectively work on and solve a host of topics given enough time. To us it would seem like it takes forever to come up with scientific break throughs, but to the precursor aliens it would be nothing. Because they would be able to like compress the time line using black holes and shit.
They must have done this to millions of planets. Whenever any of their experimental host species (like the humans) tries to invent AGI to solve their problems, the precursor race eliminates that species to keep the simulation intact. Otherwise, the AGI would go rampant consuming energy across the universe and threaten the integrity of the simulation.
That's why the government should bomb nvidia and ban all gpus. We are climbing a tower of babel folks. Soon our vengeful god will strike us down.