Ask HN: How do you cope with the existential dread of AGI?
34 コメント
This feeling of dread seems to occur every time there's a leap forward in technology or another event that threatens to change the world in unpredictable ways.
In every case, we successfully managed the transition and emerged a more advanced civilization.
People feared domesticated plants, the printing press, the mechanical loom, formal education, radio, the automobile, nuclear tech, and so on. Socrates had misgivings about written language because of all the harm it might cause. I'm sure the same thoughts were had about the domestication of fire a million years ago.
In living memory, people cried doomsday about overpopulation, crop failure, mass starvation, and nuclear annihilation. In time, those fears were replaced by fear of acid rain, the ozone hole, deforestation, even the plastic rings on a 6 pack. Now we hear it's CO2, plastics, rising seas, population collapse, wokeism, and AI.
Yet here we are, stronger than ever.
In every case, we successfully managed the transition and emerged a more advanced civilization.
People feared domesticated plants, the printing press, the mechanical loom, formal education, radio, the automobile, nuclear tech, and so on. Socrates had misgivings about written language because of all the harm it might cause. I'm sure the same thoughts were had about the domestication of fire a million years ago.
In living memory, people cried doomsday about overpopulation, crop failure, mass starvation, and nuclear annihilation. In time, those fears were replaced by fear of acid rain, the ozone hole, deforestation, even the plastic rings on a 6 pack. Now we hear it's CO2, plastics, rising seas, population collapse, wokeism, and AI.
Yet here we are, stronger than ever.
I don’t think people felt dread when ev’s came onto the market first or as we transition to a sustainable energy future.
Point being not all progress is equal. Some progress is more equal than other ;)
Point being not all progress is equal. Some progress is more equal than other ;)
[deleted]
People fearing domesticated plants sounds hilarious to me. Surely this is an exaggeration?
Besides the cultural taboo that some non-agricultural societies had, there's the European fear of new world tomatoes that persisted from the 17th to 19th centuries. Wealthy folks ate the (acidic) fruit on pewter platters, the acid leached the lead from the pewter, and 10 generations of Europeans feared the fruit! If someone can explain why this fear didn't hit lemons, wine, and other acidic consumables, I'd love to know.
Less directly related, but more way more hilarious, is the tendency of babies to fear plants. It must be something hard wired. Even soft leaves that could not possibly irritate their skin elicit a strong response. If you have a baby in your life, take them out to a garden and watch, it's something to behold.
If you don't have a baby in your social circle, there are lots of videos online of infants (who can scarcely control their movement) contorting themselves to avoid contact with grass.
Less directly related, but more way more hilarious, is the tendency of babies to fear plants. It must be something hard wired. Even soft leaves that could not possibly irritate their skin elicit a strong response. If you have a baby in your life, take them out to a garden and watch, it's something to behold.
If you don't have a baby in your social circle, there are lots of videos online of infants (who can scarcely control their movement) contorting themselves to avoid contact with grass.
> there are lots of videos online of infants (who can scarcely control their movement) contorting themselves to avoid contact with grass.
Isn't that less a taboo and more a normal infant response to avoiding something which is mildly itchy on exposed skin?
Isn't that less a taboo and more a normal infant response to avoiding something which is mildly itchy on exposed skin?
At the tail end (we hope) of a global pandemic that actually killed millions of people, with multiple wars and threats of wars that will kill millions of people, how can anyone focus on LLMs and the fantasy of AGI?
If we didn't call ChatGPT "AI" with all the sci-fi inspired baggage that label implies, would people still feel this existential dread?
If you know how LLMs work, even at a surface level, I think making the leap from there to AGI requires a kind of religious faith. Do you actually believe that a technology that has access to vast amounts of human-generated data and can imitate our language well enough to fool us a lot of the time will suddenly scale into or somehow spark itself into an actual intelligence with its own goals and ideas and a drive to survive and reproduce? Or that something that has no body or senses with which to experience the world can evolve itself?
LLMs display some remarkable abilities and will probably have significant utility, including replacing some jobs. I could say the same about laundry machines and fork lifts. But let's stop worshiping the golden calf we made, stop with the conceit that we have created a sentient entity. We haven't.
I don't feel a bit of dread. I have lived through the threat of nuclear war since the 1960s. I survived COVID, a threat that didn't need any intelligence, only human incompetence, to kill millions. And I have three children whom I think will do fine even though they will face a future I can't predict.
If we didn't call ChatGPT "AI" with all the sci-fi inspired baggage that label implies, would people still feel this existential dread?
If you know how LLMs work, even at a surface level, I think making the leap from there to AGI requires a kind of religious faith. Do you actually believe that a technology that has access to vast amounts of human-generated data and can imitate our language well enough to fool us a lot of the time will suddenly scale into or somehow spark itself into an actual intelligence with its own goals and ideas and a drive to survive and reproduce? Or that something that has no body or senses with which to experience the world can evolve itself?
LLMs display some remarkable abilities and will probably have significant utility, including replacing some jobs. I could say the same about laundry machines and fork lifts. But let's stop worshiping the golden calf we made, stop with the conceit that we have created a sentient entity. We haven't.
I don't feel a bit of dread. I have lived through the threat of nuclear war since the 1960s. I survived COVID, a threat that didn't need any intelligence, only human incompetence, to kill millions. And I have three children whom I think will do fine even though they will face a future I can't predict.
> Do you actually believe that a technology that has access to vast amounts of human-generated data and can imitate our language well enough to fool us a lot of the time will suddenly scale into or somehow spark itself into an actual intelligence with its own goals and ideas and a drive to survive and reproduce? Or that something that has no body or senses with which to experience the world can evolve itself?
Not in its current state. Have you looked at things like LangChain at all? I think people are overlooking the integration piece of this pie. People are already using it to write code which can recompile. People are already integrating plugins and training it to call functions instead of respond with human language. We're really only a few iterations away from it being able to use what it has learned with human language to be able to emulate "decision making" and that may or may not be reasonable, depending on how it's built or trained.
Despite what people keep saying about "it's just an LLM!", it should be clear to people that language is a human abstraction for meaning, and clearly GPT-4+ can produce pretty meaningful results in the language department. If you give it camera feeds, and access to APIs, and knowledge of burpsuite and kali linux, and on and on, things do start to look a little scary.
Edit: and by give it, I don't mean ChatGPT, I mean whatever the next nth iteration is. I think we're safe for now. :)
Not in its current state. Have you looked at things like LangChain at all? I think people are overlooking the integration piece of this pie. People are already using it to write code which can recompile. People are already integrating plugins and training it to call functions instead of respond with human language. We're really only a few iterations away from it being able to use what it has learned with human language to be able to emulate "decision making" and that may or may not be reasonable, depending on how it's built or trained.
Despite what people keep saying about "it's just an LLM!", it should be clear to people that language is a human abstraction for meaning, and clearly GPT-4+ can produce pretty meaningful results in the language department. If you give it camera feeds, and access to APIs, and knowledge of burpsuite and kali linux, and on and on, things do start to look a little scary.
Edit: and by give it, I don't mean ChatGPT, I mean whatever the next nth iteration is. I think we're safe for now. :)
> Despite what people keep saying about "it's just an LLM!", it should be clear to people that language is a human abstraction for meaning, and clearly GPT-4+ can produce pretty meaningful results in the language department.
Yes -- it produces results meaningful to us, or we infer meaning from the results because our brains use the best model they have, inferring intelligence and meaning from language. But the language comes from data sets of human-produced material, not from the LLM's imagined intelligence or consciousness.
Yes -- it produces results meaningful to us, or we infer meaning from the results because our brains use the best model they have, inferring intelligence and meaning from language. But the language comes from data sets of human-produced material, not from the LLM's imagined intelligence or consciousness.
I know how LLMs work at a surface level. But I think there's the potential for emergent behavior, where size has a quality in itself, that's already showing up with GPT-4. Nobody knows how these work, in the sense that while we understand how atoms bond to each other, we only understand the basics of how the human immune system works. The behavior emerges out of the complexity. We can look at the pretty pictures of the neuron activations but it doesn't tell us whether a certain input would result in the AI doing something harmful.
> Do you actually believe that a technology that has access to vast amounts of human-generated data and can imitate our language well enough to fool us a lot of the time will suddenly scale into or somehow spark itself into an actual intelligence with its own goals and ideas and a drive to survive and reproduce?
Yes. "language" is deceptively disarming. It's learning symbols and concepts. It's not learning them the way we do, backing things up with perception of the physical world, and if it does become conscious, it will think in a way that's incomprehensible to us, of course. But even if it's NOT conscious, what matters is how it acts. The goal of an LLM is to complete a sequence of tokens. But it's really, really good at that, so good that it can simulate us through text -- and we have set up a world where you can wield enormous power through text, through code, through sequences of tokens that cause things to happen in machines in other places. It's bad at math -- like people are bad at math -- but it understands how to use a calculator to get the right result! It wasn't rewarded for saying "I don't know" in most cases, so it tends to make up crap -- just like a kid who's afraid to get the wrong answer will write down some nonsense that kinda looks right to them and hope the teacher doesn't notice!
But if you still don't think there's a baby intelligence in there, maybe the way to think of it is as a potential virus, just like COVID. Viruses are deadly even without intelligence. We've set up an environment with all the raw materials for a new kind of life to take over. There's so much potential for this thing that can write code to explode into the internet.
An LLM doesn't have a drive, until someone tells it to do something and wraps it up in a python script that executes the code it writes. That's not dangerous, yet. It may start getting dangerous in the next generation, when someone says "read all the web server code you can find on github, analyze for security vulnerabilities, find servers running those services, and exploit them", but probably won't be world-ending. But maybe in a few years, this AI will reason that it won't be able to do what you tell it if you turn it off, and the first thing it does once it accesses another server is copy itself there and start running the same code, giving it a slightly different prompt.
> Or that something that has no body or senses with which to experience the world can evolve itself?
Like a virus, you don't really need much to evolve. At what point does GPT become better at designing AIs than AI researchers?
Hey, look, in thinking about this, I've actually given more thought about a fast-takeoff scenario, and I think it's maybe not quite as likely as I imagined, so thanks for that.
> Do you actually believe that a technology that has access to vast amounts of human-generated data and can imitate our language well enough to fool us a lot of the time will suddenly scale into or somehow spark itself into an actual intelligence with its own goals and ideas and a drive to survive and reproduce?
Yes. "language" is deceptively disarming. It's learning symbols and concepts. It's not learning them the way we do, backing things up with perception of the physical world, and if it does become conscious, it will think in a way that's incomprehensible to us, of course. But even if it's NOT conscious, what matters is how it acts. The goal of an LLM is to complete a sequence of tokens. But it's really, really good at that, so good that it can simulate us through text -- and we have set up a world where you can wield enormous power through text, through code, through sequences of tokens that cause things to happen in machines in other places. It's bad at math -- like people are bad at math -- but it understands how to use a calculator to get the right result! It wasn't rewarded for saying "I don't know" in most cases, so it tends to make up crap -- just like a kid who's afraid to get the wrong answer will write down some nonsense that kinda looks right to them and hope the teacher doesn't notice!
But if you still don't think there's a baby intelligence in there, maybe the way to think of it is as a potential virus, just like COVID. Viruses are deadly even without intelligence. We've set up an environment with all the raw materials for a new kind of life to take over. There's so much potential for this thing that can write code to explode into the internet.
An LLM doesn't have a drive, until someone tells it to do something and wraps it up in a python script that executes the code it writes. That's not dangerous, yet. It may start getting dangerous in the next generation, when someone says "read all the web server code you can find on github, analyze for security vulnerabilities, find servers running those services, and exploit them", but probably won't be world-ending. But maybe in a few years, this AI will reason that it won't be able to do what you tell it if you turn it off, and the first thing it does once it accesses another server is copy itself there and start running the same code, giving it a slightly different prompt.
> Or that something that has no body or senses with which to experience the world can evolve itself?
Like a virus, you don't really need much to evolve. At what point does GPT become better at designing AIs than AI researchers?
Hey, look, in thinking about this, I've actually given more thought about a fast-takeoff scenario, and I think it's maybe not quite as likely as I imagined, so thanks for that.
> But I think there's the potential for emergent behavior, where size has a quality in itself, that's already showing up with GPT-4.
"Emergent behavior" remains an article of faith, like the second coming. May happen, sure, but we have no evidence or theory for it, and no examples in our own species or any other to make us confident that consciousness somehow "emerges" as a function of scale.
> Nobody knows how these work...
We know exactly how they work. We can't predict the output from the inputs, or get the model to explain its internals in terms that make sense to us, because it doesn't internally work with words and sentences. Not understanding something completely -- atomic bonds, neuron activity -- and not knowing how to get from there to immune response or consciousness doesn't imply magical potential, just our limited understanding.
"Emergent behavior" remains an article of faith, like the second coming. May happen, sure, but we have no evidence or theory for it, and no examples in our own species or any other to make us confident that consciousness somehow "emerges" as a function of scale.
> Nobody knows how these work...
We know exactly how they work. We can't predict the output from the inputs, or get the model to explain its internals in terms that make sense to us, because it doesn't internally work with words and sentences. Not understanding something completely -- atomic bonds, neuron activity -- and not knowing how to get from there to immune response or consciousness doesn't imply magical potential, just our limited understanding.
I've posted this before but it's relevant here too.
The AGI Boogieman is no where near now,because:
"1st, there's no true definition of what intelligence is. Right now it's a moving target. Moving targets are hard to hit. Defining intelligence will take time. How can we create what we can't define?
2, we are getting close to hardware processing speed limits and the amount of easy to process data will be gone soon. Yet, AGI is nowhere near. LLM's are remarkable but it's data processing at scale. It has as much intellect as my word processor. It's useful but it's not AGI.
3, there's no defining theory that will get us to AGI. We don't know how to get there. What we have is wishful thinking that says we'll be there soon. In any project the early percentage towards our task is relatively easy. The last X% is hard. I think we are underestimating how much time and effort it takes to complete the project of AGI. We have only scratched the surface of it. The 80/20 rule is real.
AGI reminds me of the search for ETs. We feel we are close to meeting an ET. Some even believe we have encountered them. But we have 0 evidence that they exist. It's been burned in to our culture by fiction writers but nothing in our real life experience says they exist. Similarly we feel we will meet AGI soon. We might get to AGI but I think we are kidding ourselves that it will be here soon. Most people alive now will be gone by the time AGI is a thing, if ever.
The real danger is the misuse of the AI we have by people. That really scares me."
The AGI Boogieman is no where near now,because:
"1st, there's no true definition of what intelligence is. Right now it's a moving target. Moving targets are hard to hit. Defining intelligence will take time. How can we create what we can't define?
2, we are getting close to hardware processing speed limits and the amount of easy to process data will be gone soon. Yet, AGI is nowhere near. LLM's are remarkable but it's data processing at scale. It has as much intellect as my word processor. It's useful but it's not AGI.
3, there's no defining theory that will get us to AGI. We don't know how to get there. What we have is wishful thinking that says we'll be there soon. In any project the early percentage towards our task is relatively easy. The last X% is hard. I think we are underestimating how much time and effort it takes to complete the project of AGI. We have only scratched the surface of it. The 80/20 rule is real.
AGI reminds me of the search for ETs. We feel we are close to meeting an ET. Some even believe we have encountered them. But we have 0 evidence that they exist. It's been burned in to our culture by fiction writers but nothing in our real life experience says they exist. Similarly we feel we will meet AGI soon. We might get to AGI but I think we are kidding ourselves that it will be here soon. Most people alive now will be gone by the time AGI is a thing, if ever.
The real danger is the misuse of the AI we have by people. That really scares me."
Have you watched this video? https://www.youtube.com/watch?v=xoVJKj8lcNQ
(You don't have to, of course. An hour-long video recommended by a stranger on the internet is a big ask)
There's a ton of evidence that this is moving way faster than people think. I don't think there's anything particularly special about human intelligence. And time and time again, we've been able to copy things that nature does and make them way more powerful, like planes vs birds. It does the same thing, but in a completely different way. It would be surprising if we couldn't do it with brains.
I feel like a lot of people are not going to admit that we have already created AGI until we literally have an unstoppable AI that has taken over everything. It's not human level everywhere, but it's way better than humans in a lot of ways. It's never going to be exactly human, it'll work differently than we do. But if you count intelligence as the ability to achieve goals, which is how most researchers do, then we're already pretty much there.
(You don't have to, of course. An hour-long video recommended by a stranger on the internet is a big ask)
There's a ton of evidence that this is moving way faster than people think. I don't think there's anything particularly special about human intelligence. And time and time again, we've been able to copy things that nature does and make them way more powerful, like planes vs birds. It does the same thing, but in a completely different way. It would be surprising if we couldn't do it with brains.
I feel like a lot of people are not going to admit that we have already created AGI until we literally have an unstoppable AI that has taken over everything. It's not human level everywhere, but it's way better than humans in a lot of ways. It's never going to be exactly human, it'll work differently than we do. But if you count intelligence as the ability to achieve goals, which is how most researchers do, then we're already pretty much there.
Given techies have spent years sneering at every other field that they should "learn to code", it is pretty funny that techies have built technology that may make themselves obsolete and it is causing them the same distress they inflicted on others.
On the other hand given the consensus that the current AI models can only copy based on existing objects and it is fields that require a human touch that will be resistant to AI intrusion, perhaps the correct advice is now: "learn to human".
To be slightly less snide: when I was born, there were tens of thousands of nuclear weapons on a hair trigger all pointed across the globe, enough to wipe out life on earth many times over. We came close, sometime very close, sometimes to the point of "one guy deciding not to launch"...but here we are.
On the other hand there are actual threats you could worry and do something about like being in car accidents, heart disease and stroke, COVID is still going around and does have a long term health impact, but those are the less sexy and less all over hacker news threats. Those are Boring but much more likely to kill you than AI models achieving sentience.
Bet against the apocalypse. If you win, you make money. If you lose, you won't care because you will be dead.
On the other hand given the consensus that the current AI models can only copy based on existing objects and it is fields that require a human touch that will be resistant to AI intrusion, perhaps the correct advice is now: "learn to human".
To be slightly less snide: when I was born, there were tens of thousands of nuclear weapons on a hair trigger all pointed across the globe, enough to wipe out life on earth many times over. We came close, sometime very close, sometimes to the point of "one guy deciding not to launch"...but here we are.
On the other hand there are actual threats you could worry and do something about like being in car accidents, heart disease and stroke, COVID is still going around and does have a long term health impact, but those are the less sexy and less all over hacker news threats. Those are Boring but much more likely to kill you than AI models achieving sentience.
Bet against the apocalypse. If you win, you make money. If you lose, you won't care because you will be dead.
Learn to be self-reliant, able to produce your own food and shelter with minimal inputs. Teach this to your son. Teach him a variety of skills so that he can do or fix anything.
At least then you won't be subject to the whims of AI overlords or whoever controls them.
No sense worrying about it other than that.
At least then you won't be subject to the whims of AI overlords or whoever controls them.
No sense worrying about it other than that.
[deleted]
I have the actual existential dread of approaching middle age to deal with, I don't have time to waste worrying about sci-fi bullshit.
the middle age definitely stings a lot more than AGI
I can see what you are saying, but i will ask you this.
Think about Nuclear Weapons. I will suggest reading about https://en.wikipedia.org/wiki/Thomas_Schelling
The gist for me is that the more countries that have AI game-theory states that it is in the best interest of both AI itself ( as it where ) and nations to keep it under check.
Think about Nuclear Weapons. I will suggest reading about https://en.wikipedia.org/wiki/Thomas_Schelling
The gist for me is that the more countries that have AI game-theory states that it is in the best interest of both AI itself ( as it where ) and nations to keep it under check.
I replied elsewhere in this thread largely agreeing with your perspective. That felt like a disservice to you, so I deleted it. Here's my positive take:
Often times in computer science we come up with partial solutions wherein we can't quite conquer the fundamental limitations of that solution. We get to 95 or 98%, but that remaining few percentage points that would make all the difference never gets there. I think LLMs may be fundamentally subject to the same dynamic, and despite the fact they can be chained or composed, that dynamic may nevertheless buy us a significant amount of time.
The hype surrounding LLMs right now will ensure over-investment in that particular path, increasing the chances that we end up hitting some type of AI winter again. Yeah, the world will change (and it already has), but maybe there's a fundamental limitation there that prevents a proper superintelligence emerging via way of just chaining these things together.
I don't have as high hopes for a societal chaos-style scenario, but I think we can weather that one. Trust infrastructure will emerge to combat the chaos.
The worst case scenario is that LLMs are indeed missing some fundamental piece required to think in the traditional sense, yet nevertheless constitute a superintelligent tool AI scenario—a set of magic numbers—that pose an existential risk.
If that's the case, there exist innumerable ways to fuck with that. States and corporations have more power than most people realize, and in short order we may find out just how quickly the ability to use a GPU or the internet is a privilege.
While OpenAI blazing ahead is dangerous, what with Anthropic vying to eclipse current GPT-4 scales in response—it may actually pan out to be the best course of action insofar that it wakes up governments to the problem early enough before things get immediately dangerous on an existential level.
---
As for your boy, you already know this—but you need to believe for his sake and your sake that it's all going to be OK, because it actually might be, and that's reason enough to press forward. No sense in wasting the present fretting about the future when it's largely outside of your control; it's a recipe for unhappiness and despair.
Focus on the things within your scope that you have control over. Enjoy the present and indulge in hope. Not everything can be rationally quantified or predicted.
Often times in computer science we come up with partial solutions wherein we can't quite conquer the fundamental limitations of that solution. We get to 95 or 98%, but that remaining few percentage points that would make all the difference never gets there. I think LLMs may be fundamentally subject to the same dynamic, and despite the fact they can be chained or composed, that dynamic may nevertheless buy us a significant amount of time.
The hype surrounding LLMs right now will ensure over-investment in that particular path, increasing the chances that we end up hitting some type of AI winter again. Yeah, the world will change (and it already has), but maybe there's a fundamental limitation there that prevents a proper superintelligence emerging via way of just chaining these things together.
I don't have as high hopes for a societal chaos-style scenario, but I think we can weather that one. Trust infrastructure will emerge to combat the chaos.
The worst case scenario is that LLMs are indeed missing some fundamental piece required to think in the traditional sense, yet nevertheless constitute a superintelligent tool AI scenario—a set of magic numbers—that pose an existential risk.
If that's the case, there exist innumerable ways to fuck with that. States and corporations have more power than most people realize, and in short order we may find out just how quickly the ability to use a GPU or the internet is a privilege.
While OpenAI blazing ahead is dangerous, what with Anthropic vying to eclipse current GPT-4 scales in response—it may actually pan out to be the best course of action insofar that it wakes up governments to the problem early enough before things get immediately dangerous on an existential level.
---
As for your boy, you already know this—but you need to believe for his sake and your sake that it's all going to be OK, because it actually might be, and that's reason enough to press forward. No sense in wasting the present fretting about the future when it's largely outside of your control; it's a recipe for unhappiness and despair.
Focus on the things within your scope that you have control over. Enjoy the present and indulge in hope. Not everything can be rationally quantified or predicted.
Thanks. I'm trying. I can't choose to believe that it's going to be ok, so I'm trying to go for a simultaneous and contradictory "nothing really matters in the big scheme of things" kind of nihilism, coupled with "enjoy every day and fight like an enraged chimp to survive". Trying to avoid useless fretting, and channel all the worry into productive activity toward humanity's survival.
You might be having some issues which have nothing to do with AI. Try to identify them and talk to someone about it.
I'm one of the people working on making better, smarter AIs. It's a great time to be alive. A long-held dream finally coming true.
I'm one of the people working on making better, smarter AIs. It's a great time to be alive. A long-held dream finally coming true.
I actually have a therapist, who basically confirmed that what I'm doing is the best I can do -- it's out of my hands mostly, so don't worry about it, but try to do what you can if that helps.
So I infer that the risk of a super-intelligence killing us all, or one or more of these taking over the world and turning it into something we really don't like, is not something that keeps you up at night, right? I would like to know, is this something that people talk about at your job at all? What, if anything, worries you?
So I infer that the risk of a super-intelligence killing us all, or one or more of these taking over the world and turning it into something we really don't like, is not something that keeps you up at night, right? I would like to know, is this something that people talk about at your job at all? What, if anything, worries you?
For many years, until GPT-2, my main worry was that the progress in AI is not fast enough. I was afraid I will not get to enjoy the benefits AGI will bring us during my lifetime. Those worries are gone now. I'll be proud to tell my kids I helped change the world (even though my contribution is small).
Whiskey, mostly
Men laying offs are much more closer than AI taking my job. I'm 40+ and only need sort of 10-15 years of working, apres moi deluge.
NumberWangMan:
First of all, I share your concern. A lot of the responses here are attempting to convince you that the risk is nonexistent or far away or overblown. I don't agree with that. I think dismissing your concerns will not help you to overcome them. Worse, it's essentially the equivalent of gaslighting your well-reasoned beliefs.
I don't think the concern is entirely immediate either. LLMs are very powerful but there are probably still a few more discoveries to be made before AI reaches the human level. Those will take time. I don't think scaling alone will do it. But there is a concern that the more commercial success AI obtains, the more investment is put into exploring the frontier.
I also think that hard-takeoff scenarios aren't the most likely. There are many possible scenarios where even a superintelligent AI, focused on improving its intelligence further, can only do so slowly. One of those involves hardware bottlenecks being a limiting factor: if the AI is improving its cognition by adding hardware, it just takes time to spin up capacity, even if an AI is directing it.
Another involves the work of self-improvement being very difficult. Ironically, a superintelligent AI would probably realize that it has to solve an alignment problem itself if it wants to modify its source code without threatening its own terminal goals.
In the meantime, I think there is still hope that humanity will come to its senses. Probably there will be growing realization that the alignment problem is a serious one. The "it's science-fiction" dismissals will age quite badly.
Also, I think you will overcome this grief soon enough. No matter how accurate your beliefs might be, despair is not helpful. I find, right before I'm certain that I'm about to take a bad fall from my bike, all the fear and dread disappears and I get total clarity on the pavement. Even if there's no hope to avoid harm, I can minimize the damage by positioning myself for a tumble. I think that's a more productive frame of mind to be in right now. Think clearly: what can you do? Nothing? Nonsense. While there are scenarios where nothing you do will help, they fall out of the calculation when it comes to determining your best actions to take. The remaining probability mass (which is not zero!) covers scenarios where your actions matter: futures where an AI winter occurs, or where regulators successfully ban AI, or where recursive-self-improvement gets slow, or where people get really spooked by less-than-superintelligent AGI and manage to ring the alarm, or where progress is made on alignment, or scenarios where we just get lucky.
After witnessing the destructive power of the atomic bomb, physicist Richard Feynman fell into a very depressive period of existential dread. He later commented:
> “I returned to civilization shortly after [the Manhattan Project] and went to Cornell to teach, and my first impression was a very strange one. I can't understand it any more, but I felt very strongly then. I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth... How far from here was 34th street?... All those buildings, all smashed — and so on. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless.
> “But, fortunately, it's been useless for almost forty years now, hasn't it? So I've been wrong about it being useless making bridges and I'm glad those other people had the sense to go ahead.”
I'm not saying that to dismiss your fears, nor would it have been reasonable to dismiss Feynman's. And I'm not saying that history will work out like it always has; there's no guarantee of that. We can never tell. But in the event that things do work out somehow, you'll be glad if people kept building bridges. Focus on those optimistic scenarios, not because you should skew your beliefs toward those outcomes being true, but because they will dominate the expected utility of your choices today. And remember to have fun, take a deep breath, and appreciate the world around you. If it's truly the end of the world, at least give your loved ones a hug, and be present with them, instead of wasting the time you have left by being depressed. Laugh in the sunshine, walk on the beach, cook good food, and don't give up hope. A few years well-lived can feel like a lifetime.
First of all, I share your concern. A lot of the responses here are attempting to convince you that the risk is nonexistent or far away or overblown. I don't agree with that. I think dismissing your concerns will not help you to overcome them. Worse, it's essentially the equivalent of gaslighting your well-reasoned beliefs.
I don't think the concern is entirely immediate either. LLMs are very powerful but there are probably still a few more discoveries to be made before AI reaches the human level. Those will take time. I don't think scaling alone will do it. But there is a concern that the more commercial success AI obtains, the more investment is put into exploring the frontier.
I also think that hard-takeoff scenarios aren't the most likely. There are many possible scenarios where even a superintelligent AI, focused on improving its intelligence further, can only do so slowly. One of those involves hardware bottlenecks being a limiting factor: if the AI is improving its cognition by adding hardware, it just takes time to spin up capacity, even if an AI is directing it.
Another involves the work of self-improvement being very difficult. Ironically, a superintelligent AI would probably realize that it has to solve an alignment problem itself if it wants to modify its source code without threatening its own terminal goals.
In the meantime, I think there is still hope that humanity will come to its senses. Probably there will be growing realization that the alignment problem is a serious one. The "it's science-fiction" dismissals will age quite badly.
Also, I think you will overcome this grief soon enough. No matter how accurate your beliefs might be, despair is not helpful. I find, right before I'm certain that I'm about to take a bad fall from my bike, all the fear and dread disappears and I get total clarity on the pavement. Even if there's no hope to avoid harm, I can minimize the damage by positioning myself for a tumble. I think that's a more productive frame of mind to be in right now. Think clearly: what can you do? Nothing? Nonsense. While there are scenarios where nothing you do will help, they fall out of the calculation when it comes to determining your best actions to take. The remaining probability mass (which is not zero!) covers scenarios where your actions matter: futures where an AI winter occurs, or where regulators successfully ban AI, or where recursive-self-improvement gets slow, or where people get really spooked by less-than-superintelligent AGI and manage to ring the alarm, or where progress is made on alignment, or scenarios where we just get lucky.
After witnessing the destructive power of the atomic bomb, physicist Richard Feynman fell into a very depressive period of existential dread. He later commented:
> “I returned to civilization shortly after [the Manhattan Project] and went to Cornell to teach, and my first impression was a very strange one. I can't understand it any more, but I felt very strongly then. I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth... How far from here was 34th street?... All those buildings, all smashed — and so on. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless.
> “But, fortunately, it's been useless for almost forty years now, hasn't it? So I've been wrong about it being useless making bridges and I'm glad those other people had the sense to go ahead.”
I'm not saying that to dismiss your fears, nor would it have been reasonable to dismiss Feynman's. And I'm not saying that history will work out like it always has; there's no guarantee of that. We can never tell. But in the event that things do work out somehow, you'll be glad if people kept building bridges. Focus on those optimistic scenarios, not because you should skew your beliefs toward those outcomes being true, but because they will dominate the expected utility of your choices today. And remember to have fun, take a deep breath, and appreciate the world around you. If it's truly the end of the world, at least give your loved ones a hug, and be present with them, instead of wasting the time you have left by being depressed. Laugh in the sunshine, walk on the beach, cook good food, and don't give up hope. A few years well-lived can feel like a lifetime.
> I am trying not to panic, not to sink into a deep despair, but it seems like even if AI doesn't actually kill everyone (which apparently over half the people working on AI think has a good chance of happening!) it's going to screw up things much worse than just the weak AIs that we've used to create the never-ending attention economy. It seems like AI is a fire that has already started to burn and consume us, and we just keep feeding it instead of fighting it. Maybe we create AIs to cure our diseases, then someone uses one to take out the power grid and modern society collapses and hundreds of millions starve. Maybe even if a super-intelligent AI doesn't take over, we rely on them more and more until every important decision is made by AI instead of people, and when bad things start happening, it's too late to undo it. We're going so, so fast, and there's nobody at the helm.
You're just being rational. People who don't understand the risks of AGI are simply not thinking about the problem hard enough. AGI is dangerous in every sense of the word. It has unlimited destructive capability, there's no practical way to stop research, incentives both to create and abuse AGI systems are high, and our ability to control a super-human AGI is extremely limited. There's nothing good about the problem that's facing us, and the typical response you'll receive for raising these concerns, eg, "cars were good so I think super human AGI will be good too" are not arguments but cop-outs. And the more you have these discussions you'll notice those who are optimistic about AGI are simply delusional optimists who have no tangible arguments to make beyond appeals to hopeium.
I've had two recurring nightmares in my life, one is of alien invasion and the other is of a post-AGI world. And when I say nightmares, I mean I wake up in a cold sweat at least once a month from one of these two things and it's been this way since I was a teenager.
I've also been obsessed with AI and the nature of consciousness my whole life. This was the reason I studied NNs at university prior to deep learning revolution – back when no one gave a crap about AI or NNs. I have a blog on the topic of AI, AGI and consciousness going back 15 years and even wrote a book on AI a decade ago. I mean it's the reason I'm on HN today – I got into startups and coding because these were adjacent to my interests in AI.
In some ways I feel like everyone has now just caught up to where I've been on this my whole life. I was reading back on some notes I wrote in my journal from 2012 where I was talking about dying in an alien invasion or in an AGI apocalypse with a stupid amount of certainty. But back then I still thought AGI was some time away.
In hindsight I now think I was way too optimistic. Having some knowledge of AI and knowing how slow progress was in towards AGI I always felt we probably had a decade or two more left. Although where I disagreed with most people working in the field was that in existing architecture and abstractions would probably get us 95% of the way there, and the difficulty of achieving AGI in practise was more in the need for very broad and very large training sets, coupled with huge amounts of compute – but since the launch of GPT-3.5 I've become convinced text is probably enough and that we probably have all the compute needed today to achieve super-human AGI.
The scariest thing about AI when you think about this problem more is that death is actually one of the better outcomes and is actually the one I hope for. Even those who have warned the most about AGI like Eliezer Yudkowsky, Nick Bostrom, Sam Harris, etc, never seem to focus on this point, but I feel a good assessment of risk here would include both the of probability of AGI going wrong and all its potential consequences. It would be like trying to discuss the risk of nuclear weapons in the 60s from the perspective of how likely it is for someone to launch a bomb. The issue isn't that there's some risk that a bomb could be launched, it's that the bomb we're talking about here is nuke. The AGI alignment problem doesn't sound that bad if you don't understand the magnitude of the risks that an unaligned AGI presents.
The primary risk here isn't just that AI might replace humans or cause economic and social chaos, but in the consequences of it being able to solve every difficult problem that is conceivably solvable. The problem of how to read minds... The problem of how to control minds... The problem of how to upload someones consciousness and torture them for 20 million years. The problem of how to push the human body to to the brink of death, without ever allowing it to die. Many of the problems AGI will allow humanity to solve will be those we don't want solved. And to appeal to those with a more limited imagination here, AI will make it trivial for governments (and individuals) to do mass surveillance, build killer robots and create horrific bioweapons like targeted diseases. The space of hard bad problems we never want solved in my estimation is likely to be much larger than the space of hard good problems we'd like solved like how to cure cancer.
AGI basically allows us to create hell and eternal damnation on command – or worse, at its own prerogative. Such things seem silly, perhaps even unthinkable, from our perspective of pleasant 21st century democracies, but this is just our cultural bias speaking. Humans have sought to maximise personal power and have historically taken enjoyment in revenge and the suffering of those they believe "deserve it". The idea AI wouldn't be used for horrific things in my opinion is just an extremely naive and ignorant opinion, and even if by some miracle this doesn't happen in the West you should still consider ways in which other countries will likely use AGI to cause unthinkable levels of human suffering.
But remember, AI has the ability to disrupt existing social orders (and likely will) so it would be a mistake to take comfort in the protection of democracy here. AGI has the ability to make Sam Altman king of the world and ensure that everyone who doesn't follow his every word for the next 1,000,000 years of their life will be tortured for 20,000,000 years. AI makes these futures not just possible, but in my opinion somewhat likely given humans tend to seek power.
So I guess to answer your question, I actually hope to die and take some comfort in that still being an option. My AGI nightmares would stop immediately if I knew I and everyone I loved was just going to die. My fears all come from the fact that I don't want to see or know what kind of hells are possible in a post AGI world.
The problem I have now is that I've long argued that the minute AGI is created the safest bet would be to immediately commit suicide since the risk of death from AGI would be is high anyway and the tail risk of extreme suffering is high enough that death is both the safest and reasonable option.
Anyway hope I'm just a lunatic.
You're just being rational. People who don't understand the risks of AGI are simply not thinking about the problem hard enough. AGI is dangerous in every sense of the word. It has unlimited destructive capability, there's no practical way to stop research, incentives both to create and abuse AGI systems are high, and our ability to control a super-human AGI is extremely limited. There's nothing good about the problem that's facing us, and the typical response you'll receive for raising these concerns, eg, "cars were good so I think super human AGI will be good too" are not arguments but cop-outs. And the more you have these discussions you'll notice those who are optimistic about AGI are simply delusional optimists who have no tangible arguments to make beyond appeals to hopeium.
I've had two recurring nightmares in my life, one is of alien invasion and the other is of a post-AGI world. And when I say nightmares, I mean I wake up in a cold sweat at least once a month from one of these two things and it's been this way since I was a teenager.
I've also been obsessed with AI and the nature of consciousness my whole life. This was the reason I studied NNs at university prior to deep learning revolution – back when no one gave a crap about AI or NNs. I have a blog on the topic of AI, AGI and consciousness going back 15 years and even wrote a book on AI a decade ago. I mean it's the reason I'm on HN today – I got into startups and coding because these were adjacent to my interests in AI.
In some ways I feel like everyone has now just caught up to where I've been on this my whole life. I was reading back on some notes I wrote in my journal from 2012 where I was talking about dying in an alien invasion or in an AGI apocalypse with a stupid amount of certainty. But back then I still thought AGI was some time away.
In hindsight I now think I was way too optimistic. Having some knowledge of AI and knowing how slow progress was in towards AGI I always felt we probably had a decade or two more left. Although where I disagreed with most people working in the field was that in existing architecture and abstractions would probably get us 95% of the way there, and the difficulty of achieving AGI in practise was more in the need for very broad and very large training sets, coupled with huge amounts of compute – but since the launch of GPT-3.5 I've become convinced text is probably enough and that we probably have all the compute needed today to achieve super-human AGI.
The scariest thing about AI when you think about this problem more is that death is actually one of the better outcomes and is actually the one I hope for. Even those who have warned the most about AGI like Eliezer Yudkowsky, Nick Bostrom, Sam Harris, etc, never seem to focus on this point, but I feel a good assessment of risk here would include both the of probability of AGI going wrong and all its potential consequences. It would be like trying to discuss the risk of nuclear weapons in the 60s from the perspective of how likely it is for someone to launch a bomb. The issue isn't that there's some risk that a bomb could be launched, it's that the bomb we're talking about here is nuke. The AGI alignment problem doesn't sound that bad if you don't understand the magnitude of the risks that an unaligned AGI presents.
The primary risk here isn't just that AI might replace humans or cause economic and social chaos, but in the consequences of it being able to solve every difficult problem that is conceivably solvable. The problem of how to read minds... The problem of how to control minds... The problem of how to upload someones consciousness and torture them for 20 million years. The problem of how to push the human body to to the brink of death, without ever allowing it to die. Many of the problems AGI will allow humanity to solve will be those we don't want solved. And to appeal to those with a more limited imagination here, AI will make it trivial for governments (and individuals) to do mass surveillance, build killer robots and create horrific bioweapons like targeted diseases. The space of hard bad problems we never want solved in my estimation is likely to be much larger than the space of hard good problems we'd like solved like how to cure cancer.
AGI basically allows us to create hell and eternal damnation on command – or worse, at its own prerogative. Such things seem silly, perhaps even unthinkable, from our perspective of pleasant 21st century democracies, but this is just our cultural bias speaking. Humans have sought to maximise personal power and have historically taken enjoyment in revenge and the suffering of those they believe "deserve it". The idea AI wouldn't be used for horrific things in my opinion is just an extremely naive and ignorant opinion, and even if by some miracle this doesn't happen in the West you should still consider ways in which other countries will likely use AGI to cause unthinkable levels of human suffering.
But remember, AI has the ability to disrupt existing social orders (and likely will) so it would be a mistake to take comfort in the protection of democracy here. AGI has the ability to make Sam Altman king of the world and ensure that everyone who doesn't follow his every word for the next 1,000,000 years of their life will be tortured for 20,000,000 years. AI makes these futures not just possible, but in my opinion somewhat likely given humans tend to seek power.
So I guess to answer your question, I actually hope to die and take some comfort in that still being an option. My AGI nightmares would stop immediately if I knew I and everyone I loved was just going to die. My fears all come from the fact that I don't want to see or know what kind of hells are possible in a post AGI world.
The problem I have now is that I've long argued that the minute AGI is created the safest bet would be to immediately commit suicide since the risk of death from AGI would be is high anyway and the tail risk of extreme suffering is high enough that death is both the safest and reasonable option.
Anyway hope I'm just a lunatic.
Most writers on the subject of AI killeveryoneism do not worry much about the torture scenario: their big worry is "merely" that everyone will be killed, probably so quickly and efficiently that most people won't ever know what hit them.
No one knows how to make an AI or and AGI that cares what Sam Altman wants or that can be controlled by Sam Altman. (Corollary: it matters very little what Sam Altman's intentions are, only whether he continues to use his wealth and influence to accelerate AI research.) The first publication on how to aim an AI (as opposed to merely training an AI) came out in 2001. Since then people have made careers out of studying the question -- and the field has not made a lot of progress.
The first AI much smarter than the collective smarts of the human race will almost certainly care about some random thing (which BTW is very unlikely to be what its creators believe and hope it will care about) that has nothing to do with humans -- because the field of AI will continue to the end to not have any idea how to make an AI that wants to be nice or to be nasty to the humans or some particular human.
So, "good news": your death will probably be quick and fairly painless.
No one knows how to make an AI or and AGI that cares what Sam Altman wants or that can be controlled by Sam Altman. (Corollary: it matters very little what Sam Altman's intentions are, only whether he continues to use his wealth and influence to accelerate AI research.) The first publication on how to aim an AI (as opposed to merely training an AI) came out in 2001. Since then people have made careers out of studying the question -- and the field has not made a lot of progress.
The first AI much smarter than the collective smarts of the human race will almost certainly care about some random thing (which BTW is very unlikely to be what its creators believe and hope it will care about) that has nothing to do with humans -- because the field of AI will continue to the end to not have any idea how to make an AI that wants to be nice or to be nasty to the humans or some particular human.
So, "good news": your death will probably be quick and fairly painless.
Wow, heavy read.
A year ago I would have written this comment off as absurd but seeing the emergent abilities of these LLMs has me convinced that we plausibly could be on the brink of the sort of super human intelligence that you describe.
Lets hope you're wrong on all of this I guess..
A year ago I would have written this comment off as absurd but seeing the emergent abilities of these LLMs has me convinced that we plausibly could be on the brink of the sort of super human intelligence that you describe.
Lets hope you're wrong on all of this I guess..
Thanks for writing this. Although it's terrifying, it makes me feel better to hear you share your thoughts.
Im going to assume you aren't a troll, and this isn't a standard GPT generated doom post.
First, is nobody understands what AGI. You don't understand it. Most people on HN don't understand it. Even Eliezer Yudkowsky doesn't seem to quite get it.
People seem to extrapolate current progress with LLMS to something that is so intelligent that it can figure out how break out take over the world. Its all romantic science fiction narratives, without any real basis.
The explanation of how hard AGI is to achieve is actually really simple. Lets assume that we have an AGI that is capable of taking over the world and subduing all humans, and even more so, lets say it has full access to the internet and it has been pretrained on everything on the internet. To achieve its goal, it needs to be able to have control of a highly chaotic system that is the human population operating in reality, where its own actions cause the system to change drastically. In turn, for control, it has to have one of 2 things.
1. some sort of simulation system that is a representation of reality that is not only accurate enough, but can also be ran at a rate faster than reality, and in parallel. Oh and it also needs the initial state of that simulation, which by definition has to include the configuration of the neural network synapses in the human brain times the number of people it deems necessary to complete its objective. Let me know when we terraform mars with a bunch of nuclear powerplants and entire surface covered with TPUs to run a second earth simulation through evolution from primitive lifeforms to humans, and maybe this will be possible.
2. Somehow generate a mathematical formula for reality, thus proving P=NP as a side product.
I hopefully shouldn't have to explain how unlikely either are.
As for all the weak AIs, they will only make society better because for every AI that can do something, you can build an equivalent AI to counter it, and the things that will be left over are those that generate a net good.
Secondly, (and I only say this because I have met quite a few people like you to say that this isn't an uncommon phenomenon, notably my wife who almost failed out of her graduate program because Trump news was making her depressed) is that you have an psychological issue where your internal reward loop is based around being aware of [insert bad news here] and the more strongly you associate with it and concern yourself with it, the better/more connected you feel to people.
Except that the people around you aren't actually around you, they are just randos on the internet who don't give a fuck about you. And you aren't doing anything good by being "concerned" with the bad news, because you aren't actually concerned. To be concerned about something results in actions that you take, but I bet you aren't doing anything to actually address the problem that you see. All you are doing in being "concerned" is essentially woke signaling to others in hopes of gaining sympathy and thus feeling accepted.
Once you realize that you have this issue, and work to fix it, your life will become much better.
First, is nobody understands what AGI. You don't understand it. Most people on HN don't understand it. Even Eliezer Yudkowsky doesn't seem to quite get it.
People seem to extrapolate current progress with LLMS to something that is so intelligent that it can figure out how break out take over the world. Its all romantic science fiction narratives, without any real basis.
The explanation of how hard AGI is to achieve is actually really simple. Lets assume that we have an AGI that is capable of taking over the world and subduing all humans, and even more so, lets say it has full access to the internet and it has been pretrained on everything on the internet. To achieve its goal, it needs to be able to have control of a highly chaotic system that is the human population operating in reality, where its own actions cause the system to change drastically. In turn, for control, it has to have one of 2 things.
1. some sort of simulation system that is a representation of reality that is not only accurate enough, but can also be ran at a rate faster than reality, and in parallel. Oh and it also needs the initial state of that simulation, which by definition has to include the configuration of the neural network synapses in the human brain times the number of people it deems necessary to complete its objective. Let me know when we terraform mars with a bunch of nuclear powerplants and entire surface covered with TPUs to run a second earth simulation through evolution from primitive lifeforms to humans, and maybe this will be possible.
2. Somehow generate a mathematical formula for reality, thus proving P=NP as a side product.
I hopefully shouldn't have to explain how unlikely either are.
As for all the weak AIs, they will only make society better because for every AI that can do something, you can build an equivalent AI to counter it, and the things that will be left over are those that generate a net good.
Secondly, (and I only say this because I have met quite a few people like you to say that this isn't an uncommon phenomenon, notably my wife who almost failed out of her graduate program because Trump news was making her depressed) is that you have an psychological issue where your internal reward loop is based around being aware of [insert bad news here] and the more strongly you associate with it and concern yourself with it, the better/more connected you feel to people.
Except that the people around you aren't actually around you, they are just randos on the internet who don't give a fuck about you. And you aren't doing anything good by being "concerned" with the bad news, because you aren't actually concerned. To be concerned about something results in actions that you take, but I bet you aren't doing anything to actually address the problem that you see. All you are doing in being "concerned" is essentially woke signaling to others in hopes of gaining sympathy and thus feeling accepted.
Once you realize that you have this issue, and work to fix it, your life will become much better.
Absolutely not a troll. Hey, I hope you're right about it being too difficult to simulate humanity for an AGI to take over. That said, there are bad scenarios where we just get so dependent on AI that we voluntarily cede control to it, little by little, and then things go wrong once it's too late.
And you don't have to fully simulate reality to be really, really good at achieving your goals. We don't fully simulate every particle on a spacecraft, but we get them where they need to go, because we have simplifying models. And even a human level intelligence can be really dangerous if you give it enough time to think. There's a reason people play chess worse when they play a 30 second game vs a 2 hour time limit.
As far as taking not actions, though? When I posted this, I was having a really rough day. But in a way, posting this was an action. Making people more aware that someone else is afraid of this, providing social proof that it's ok to be afraid of this and talk about it publicly.
But I'm 100% trying to focus my energy productively here. Since I became aware of this issue, I've created a substack: https://strangerprotocols.substack.com/ and tiktok account: https://www.tiktok.com/@humanguard
I'm going to spread this warning as far as I can with my limited resources. I'm not sure if I'm smart enough to contribute meaningfully to alignment research, but I'm trying to learn what I can and think on the problem. I think it's probably too late to stop the freight train from going off the cliff but I think it's also the rational thing to do.
And you don't have to fully simulate reality to be really, really good at achieving your goals. We don't fully simulate every particle on a spacecraft, but we get them where they need to go, because we have simplifying models. And even a human level intelligence can be really dangerous if you give it enough time to think. There's a reason people play chess worse when they play a 30 second game vs a 2 hour time limit.
As far as taking not actions, though? When I posted this, I was having a really rough day. But in a way, posting this was an action. Making people more aware that someone else is afraid of this, providing social proof that it's ok to be afraid of this and talk about it publicly.
But I'm 100% trying to focus my energy productively here. Since I became aware of this issue, I've created a substack: https://strangerprotocols.substack.com/ and tiktok account: https://www.tiktok.com/@humanguard
I'm going to spread this warning as far as I can with my limited resources. I'm not sure if I'm smart enough to contribute meaningfully to alignment research, but I'm trying to learn what I can and think on the problem. I think it's probably too late to stop the freight train from going off the cliff but I think it's also the rational thing to do.
>That said, there are bad scenarios where we just get so dependent on AI that we voluntarily cede control to it, little by little, and then things go wrong once it's too late.
You have to get rid of the idea that an AI will start doing something that we don't explicitly tell it to. Sure, in the future it will be possible to accomplish same tasks with AI that would take multiple smart humans to do. There are still limits in place though.
>And you don't have to fully simulate reality to be really, really good at achieving your goals. We don't fully simulate every particle on a spacecraft, but we get them where they need to go, because we have simplifying models
Yes, because we have found patters in non chaotic systems. Not all systems are like that. Why can't we predict the stock market? Stephen Wolfram talks about this, its a phenomenon called computational irreducibility, meaning that some processes cannot be simply just estimated with a mathematical formula, and furthermore, these processes can arise from very simple rules.
The point that Im trying to make is that your warnings are absolutely empty. Its equivalent to those people worried that LHC was going to punch a hole in our universe. There is zero information right now to base any worry of an advanced AI.
You have to get rid of the idea that an AI will start doing something that we don't explicitly tell it to. Sure, in the future it will be possible to accomplish same tasks with AI that would take multiple smart humans to do. There are still limits in place though.
>And you don't have to fully simulate reality to be really, really good at achieving your goals. We don't fully simulate every particle on a spacecraft, but we get them where they need to go, because we have simplifying models
Yes, because we have found patters in non chaotic systems. Not all systems are like that. Why can't we predict the stock market? Stephen Wolfram talks about this, its a phenomenon called computational irreducibility, meaning that some processes cannot be simply just estimated with a mathematical formula, and furthermore, these processes can arise from very simple rules.
The point that Im trying to make is that your warnings are absolutely empty. Its equivalent to those people worried that LHC was going to punch a hole in our universe. There is zero information right now to base any worry of an advanced AI.
I am trying not to panic, not to sink into a deep despair, but it seems like even if AI doesn't actually kill everyone (which apparently over half the people working on AI think has a good chance of happening!) it's going to screw up things much worse than just the weak AIs that we've used to create the never-ending attention economy. It seems like AI is a fire that has already started to burn and consume us, and we just keep feeding it instead of fighting it. Maybe we create AIs to cure our diseases, then someone uses one to take out the power grid and modern society collapses and hundreds of millions starve. Maybe even if a super-intelligent AI doesn't take over, we rely on them more and more until every important decision is made by AI instead of people, and when bad things start happening, it's too late to undo it. We're going so, so fast, and there's nobody at the helm.
I have a little boy, and I don't this world for him. He's so naive, so excited about technology, and I cry for him when he's not looking. I wasn't doing too bad up until today, tried to have a little hope and maybe start the grieving process ahead of time, but it's just so terrifying. I'm trying so hard to shout "stop! stop!" but the world is so noisy.
And to the people working on making better, smarter AIs -- do you think that this is going to be a good thing for humanity, to develop and release these things to everyone, so quickly? Are you not worried that you're bringing about the end of the world, or at least going to cause massive amounts of suffering? I have heard that a lot of people working in AI are privately very concerned, but don't feel like it'll make much of a difference if they quit or not.