AI 2040: Plan A(ai-2040.com)
ai-2040.com
AI 2040: Plan A
https://ai-2040.com/
168 コメント
I'm eagerly awaiting "AI 2100" from the same people.
please bro just one more decade bro i promise AGI is right around the corner no the hockeystick graph isn't made up bullshit we just need to wait a few more years bro please please bro
This seems way more accurate. Up until early 2026.
Though it has bizzare fixation on geopolitics and China which it severely understimated. It's pretty obvious that China is going to outinnovate and outcompute US companies quite soon. Even if just because they care about higher education, providing enough electricity and letting smart people do smart things instead of randomly muzzling them with bans and export controls and coddling them with financial protectionism.
Though it has bizzare fixation on geopolitics and China which it severely understimated. It's pretty obvious that China is going to outinnovate and outcompute US companies quite soon. Even if just because they care about higher education, providing enough electricity and letting smart people do smart things instead of randomly muzzling them with bans and export controls and coddling them with financial protectionism.
Hm, China is also beginning to invoke export controls to restrict homegrown models: https://www.reuters.com/world/beijing-is-looking-curbing-ove...
I wonder how much of what China is currently able to do it because USA is such a mess.
The whole article is kind of ridiculous of course, and is also heavily fixated on OpenBrain, whatever that is.
I also wonder about the economics of running an AI lab attached to an existing large tech company (such as Meta or Tencent) instead of a dedicated company like OpenAI. It's starting to seem like it's not possible to charge enough for current-gen AI usage, with current-gen inference technology, in order to turn a profit, i.e. nobody is able or willing to pay at least marginal cost for tokens.
The whole article is kind of ridiculous of course, and is also heavily fixated on OpenBrain, whatever that is.
I also wonder about the economics of running an AI lab attached to an existing large tech company (such as Meta or Tencent) instead of a dedicated company like OpenAI. It's starting to seem like it's not possible to charge enough for current-gen AI usage, with current-gen inference technology, in order to turn a profit, i.e. nobody is able or willing to pay at least marginal cost for tokens.
I'm sure some people will have issue with my phrasing but, honest question:
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
This is a just so story. The main issue today is the lack of democracy in the country and the use of technology to surveil and govern a restive population as the government has less and less legitimacy. The narrative you are telling is the heroic tale of computing and the internet c. 1990-2010.
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
http://yashalevine.com/surveillance-valley
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
http://yashalevine.com/surveillance-valley
Yes, and RMS was correct in his Right to Read and so many other things - we're seeing the slow death of the never-enshrined-in-law right to compute. Luckily open-source is big enough to slow this down; we should all be pretty amazed and appreciative that there even are open-weights models at all, out there, because it is a profoundly democratizing thing.
> appreciative that there even are open-weights models at all
thank, mr 习
thank, mr 习
I don't think parent commenter means tech in the modern sense. Seems like they're describing a transition to democracy which started centuries ago, not decades.
No one can run a nuclear reactor on their phone but can run an AI
We were discussing AI in the 90s and it's been discussed before that.
The answer was always the same; hardware can't hang.
Now it can and will get even better.
The SaaS era fueled by ZIRP and ignorant Congress was a fluke that from an engineering perspective didn't produce anything but hype and same old
The generation enriched and empowered by it is just as temporary as Boomers. Little point in enabling their appeals at the expense of scientific progress that helps all of humanity.
China won't. Russia won't.
It's ridiculous to me the level QQing coming from Americans exploiting child sweatshop labor so they are free to ignore their own biological needs and keep a "knowledge work" job (talk about first world privilege) handing them wealth to go tour the poor villages they exploit.
Those workers never had a choice between college or the mines. So sorry 300 million Americans in a world of 8 billion.
We don't even want these jobs given how much bitching I have listened to the last 10-15. IMO the job creators and Congress saw how Millennials liked to be on the computer and went way too far into enabling such banal output.
Make healthcare and housing the economic tentpole. Both still need jobs and technology. But at least the outcome isn't a generational Ponzi scheme engineered by Boomers to enrich them and then let it all collapse when the majority realize those stocks were never real.
We were discussing AI in the 90s and it's been discussed before that.
The answer was always the same; hardware can't hang.
Now it can and will get even better.
The SaaS era fueled by ZIRP and ignorant Congress was a fluke that from an engineering perspective didn't produce anything but hype and same old
The generation enriched and empowered by it is just as temporary as Boomers. Little point in enabling their appeals at the expense of scientific progress that helps all of humanity.
China won't. Russia won't.
It's ridiculous to me the level QQing coming from Americans exploiting child sweatshop labor so they are free to ignore their own biological needs and keep a "knowledge work" job (talk about first world privilege) handing them wealth to go tour the poor villages they exploit.
Those workers never had a choice between college or the mines. So sorry 300 million Americans in a world of 8 billion.
We don't even want these jobs given how much bitching I have listened to the last 10-15. IMO the job creators and Congress saw how Millennials liked to be on the computer and went way too far into enabling such banal output.
Make healthcare and housing the economic tentpole. Both still need jobs and technology. But at least the outcome isn't a generational Ponzi scheme engineered by Boomers to enrich them and then let it all collapse when the majority realize those stocks were never real.
No, of course not. That would be an insane trust fall. Even relatively small advances in technology give a country world dominating power. Fun fact: India was militarily superior to Britain in the 1600s—a gunpowder empire with a million soldiers—but was taken over by it in the 1700s. Britain’s edge was small: lighter, more maneuverable cannons, standardized ammunition, better military and political organization. Not a first world country versus a third world country—more like the dynamic US versus a sclerotic EU. And that modest edge led to 200 years of colonization.
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
You overstate the advantages of technology. Mughal India was fragmented and on a sharp decline. The British used politics, finance, and treachery to divide and conquer what was remaining.
> we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial.
An easy choice to make if the alternative is everyone dying instead.
An easy choice to make if the alternative is everyone dying instead.
The alternative isn’t “everyone dying.” It’s us holding all the cards.
Why? Is human extinction not permitted by the laws of physics?
That’s a possibility, but not the only one. The two most realistic ones are: we race ahead and maintain our status, or we slow down and open ourselves up to colonization.
Using an ASI to subjugate humans in any capacity is a terrible idea.
without sharing tech to make the ASI, you'd hope humanity could work together to determine how to align an AI for our common benefit.
without sharing tech to make the ASI, you'd hope humanity could work together to determine how to align an AI for our common benefit.
I think the question “would China cooperate” needs much more investigation. Everyone online pundit seems to think “obviously not”, but they’re people too with clear positive and negative incentives. It’s possible they’ve found a very similar calculus that we have.
> “Politics is the art of the possible”
> “Politics is the art of the possible”
Worth noting that it is the Europeans and Americans that have been colonial. Asian peoples have, with the prominent exception of the Mongols and Japanese Empire, pretty much not done that. In particular, China shut down its exploration program.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
The Quing era boundaries are quite a bit larger than the Han boundaries. That did not happen by peaceful means.
But is it? Is there any realistic world where we need ASI for human survival?
Yes, this one. Look at our governance; look at our coordination-at-scale; look at our collective problem solving. It's abysmal, beyond hope. If we have global scale problems, we are not capable of solving them effectively. We are literally not intelligent enough to handle the problems we are creating. Between rivers of garbage and CO2 levels and war, we have proven ourselves to be woefully unintelligent at the scale needed. If we are lucky, our thin window of survival depends on getting a hell of a lot smarter, real quick.
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
we should never make ASI, what I am saying is strongly sane. ASI = not good. AGI, ~human brainpower, can be made safe enough.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?
The decision to not go with the development of extremely large thermonuclear weapons might count - the US Sundial Project was supposed to be about 10 gigatons of TNT. Not the most practical weapons but once you get to a certain size delivery arguably stops being a problem - its going to kill everyone anyway so doesn't matter where you let it off!
https://en.wikipedia.org/wiki/Sundial_(weapon)
Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
https://en.wikipedia.org/wiki/Sundial_(weapon)
Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
If there were examples, their example status would drop the odds that I know about them.
The 1972 Biological Weapons Convention? Probably partly successful.
The Soviet Union systematically ignored the BWC. There's tons of evidence.
Isn't the US famous for not even signing a lot of world treaties like climate accords and others?
This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.
Genomics is what finally broke the barrier, especially in the last decade or so.
Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.
Genomics is what finally broke the barrier, especially in the last decade or so.
Yes, there are examples of where we have collective decided not to pursue a particular technology tree.
For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)
And there have been successful world-wide bans.
For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)
And there have been successful world-wide bans.
For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
I completely can see why we'd want to, for ethical reasons, ban germline editing, and I want to be clear that I agree doing so cannot be done ethically, but there is a part of me that is wistful for what could have been. Same with things like CRISPR but it's probably just fun to dream and the reality would be a nightmare.
Taking the approach with AI that we took with the atomic bomb would be catastrophic. If the only people who are allowed to use this technology are governments, intelligence agencies, and a select few anointed companies, then the risk of authoritarian misuse will skyrocket.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
I agree. I guess I should have said something like:
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
I found the AI 2027 paper to be overly optimistic, but not wholly fantastical. This paper feels wildly speculative, and relies on premises I am not confident even pass surface reasoning. Even under optimistic conditions, we are not going to see robots "capable of 95% of all cognitive and physical tasks" by 2035. Nor do I think a 74% unemployment rate is even remotely possible. Economic collapse would implode AI development long before those figures were plausible.
If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
The thing about exponentials is if you admit 60%, it's pretty easy to admit 95%.
The "and physical" is the part I'm particularly skeptical of. Sure, drones are scary, but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly.
A lot of it relies on what is effectively "the AI will be so smart it can solve anything" magic.
The book Sentient is not about AI but abount the most amazing physical senses some other animals have.
The theme of the scientific findings is that while humans excel with none of our physical sensors, we do very well across the board in making use of them thanks to our relatively huge brains.
And fantastical amounts of compute power is exactly what are handing over to AI. The fact that their training data isn't perfect may matter less.
The theme of the scientific findings is that while humans excel with none of our physical sensors, we do very well across the board in making use of them thanks to our relatively huge brains.
And fantastical amounts of compute power is exactly what are handing over to AI. The fact that their training data isn't perfect may matter less.
Do autonomous systems need to solve humanoid robotics to exert power over the physical world? Seems like a lot can be done with drones.
Military power, sure. In Ukraine they hit everything they can see.
But during peacetime, you don't make money running a delivery service that way, so it's not going to replace those jobs.
But during peacetime, you don't make money running a delivery service that way, so it's not going to replace those jobs.
but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly
If you don't care about getting the drone back, it does simplify the problem somewhat.
If you don't care about getting the drone back, it does simplify the problem somewhat.
> Nor do I think a 74% unemployment rate is even remotely possible
250 years of constant automation has never produced large scale unemployment, despite obsoleting everyone's jobs several times over.
250 years of constant automation has never produced large scale unemployment, despite obsoleting everyone's jobs several times over.
Don’t you think the speed at which obsolescence occurs matters? There’s a bit of survivorship bias here, in the sense of
“I’ve been pulling my sled across this lake for 50 winters even when the temperature went above freezing. Never fell through!”
“I’ve been pulling my sled across this lake for 50 winters even when the temperature went above freezing. Never fell through!”
It seems to me
we’re already at the top of the S curve, not at the toe of an exponential curve. At least with LLMs. Better training data will make small improvements, better architecture will make it less compute intensive, and all these “hyper-scale” data centers will make it cheap and ubiquitous. But none of that is it getting exponentially more intelligent.
I completely agree, but I also think that an industry disrupting architecture tweak akin to the “Attention is all you need” is VERY possible to emerge at any moment.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
What leads you to believe that?
Things like https://www.tobyord.com/writing/hourly-costs-for-ai-agents and https://www.tobyord.com/writing/mostly-inference-scaling seem in line with other accounts like https://www.youtube.com/watch?v=aR20FWCCjAs ?
Is chatgpt 5.6 that much smarter than chatgpt 5.0?
If you’ve done any software development at all, certainly.
Is that true or does it only feel true because they nerf the old models just before every major release?
I remember being blown away by o1-o3 family of models finally stringing together coherent agentic tool calls to write and execute scripts semi-reliably for workloads in the several minutes before they would start hallucinating/flailing. GPT 5 was a bit ahead of that, but barely
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
Not only this, but in the GPT 5.0 era, agents had 0 taste. Nothing looked good. It was the agentic version of the twitter bootstrap era, but worse somehow. Now, I would argue the average agent frontend beats the average human frontend. This isn't even getting into 3D applications in the GPT 5 era
Anyway, the models now reliably execute more than a human can fit into their own context. It's magic
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
Not only this, but in the GPT 5.0 era, agents had 0 taste. Nothing looked good. It was the agentic version of the twitter bootstrap era, but worse somehow. Now, I would argue the average agent frontend beats the average human frontend. This isn't even getting into 3D applications in the GPT 5 era
Anyway, the models now reliably execute more than a human can fit into their own context. It's magic
Yes, and we haven't even really begun to nail down computer-use agents yet (can you believe they're still basically just OCR'ing screenshots?)
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
How do you think humans experience desktop interfaces? “Basically just OCR'ing screenshots” is exactly what humans do.
I was also under the impression modern AI agents have moved on from just OCR'ing screenshots to leveraging native vision model capabilities.
I always thought bootstrap was pretty good. All the gradients and sparkles don't do much for me.
You can compare benches of the old models against the new models. So yeah, you can see the difference.
Even then, you can just compare the progress in open models. Leaps and bounds from where they were 6 months ago.
Even then, you can just compare the progress in open models. Leaps and bounds from where they were 6 months ago.
5.6 to 5.0 is a big enough of a jump to say yes. if it was 5.4 to 5.6 it would be a bit easier to say it only feels true because of that, but 5.6 is definitely better than 5.0. I don't have anything empirical to point at though, which is your point, but August 2025 for 5.0 vs July 2026 is almost a year later, and it's not just vibes that it's better, despite not having an objective metric to point at. It would be more scientifical to have numbers and shit to point at and there are some benchmarks out there, but you have to dig into them and really understand them in order to believe in exactly what they're testing, and I'm betting you haven't.
I note that they use the Sam Altman quote.
"AI will most likely lead to the end of the world, but in the meantime there will be great companies.”
Is there any serious journalistic source suggesting that this was anything other than an offhand joke? This article links to a youtube clip of the comment with context removed, but hair raising comments.
Taking the most uncharitable view of any person, you could imagine someone who was evil enough to cause the end of the world after their own lifespan where they faced no inconvenience, but not the circumstances from the quote
The quote as it stands is preposterous enough that I don't think a human capable of functioning in society would seriously say such a thing.
Are people wilfully misinterpreting the comment, or do they truly believe this an actually held opinion? If so, can they explain how they think someone could hold an opinion like that?
"AI will most likely lead to the end of the world, but in the meantime there will be great companies.”
Is there any serious journalistic source suggesting that this was anything other than an offhand joke? This article links to a youtube clip of the comment with context removed, but hair raising comments.
Taking the most uncharitable view of any person, you could imagine someone who was evil enough to cause the end of the world after their own lifespan where they faced no inconvenience, but not the circumstances from the quote
The quote as it stands is preposterous enough that I don't think a human capable of functioning in society would seriously say such a thing.
Are people wilfully misinterpreting the comment, or do they truly believe this an actually held opinion? If so, can they explain how they think someone could hold an opinion like that?
I think he means the end of the world as we know it. Which is probably true, but the timeline could be 20 years, 2000 years, or more.
NYT reported today that Russia and China are funding anti-datacenter and anti-ai hysteria on western social media.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
Which then gives more ammunition to folks like Kevin O'Leary et al to pretend any objections to data centers or AI are Chinese plots, or "hysteria" as you put it.
While I can admit some of the anti-data center arguments are overblown, many are more than valid in my opinion. Data centers are fundamentally extractive technologies. They are enormous, windowless boxes that take resources from one location to make someone else in a far off location enormously rich and powerful, with extremely few benefits to the local community.
Plus, as another commenter mentioned, it's not exactly like the Chinese and Russians have been fanning the flames that AI is going to take all of our jobs - it's the leaders of the frontier AI companies in the US saying that. Remind me again why I think putting up a giant data center in my state, that was proposed to use more electricity than my state already currently uses, is a good thing for the average joe where I live??
Honestly, I feel like many commenters here are in their own bubble and don't understand how much AI and tech generally is widely viewed as a net negative for society by huge swaths of the the population, and I don't really think it's an unwarranted perception.
While I can admit some of the anti-data center arguments are overblown, many are more than valid in my opinion. Data centers are fundamentally extractive technologies. They are enormous, windowless boxes that take resources from one location to make someone else in a far off location enormously rich and powerful, with extremely few benefits to the local community.
Plus, as another commenter mentioned, it's not exactly like the Chinese and Russians have been fanning the flames that AI is going to take all of our jobs - it's the leaders of the frontier AI companies in the US saying that. Remind me again why I think putting up a giant data center in my state, that was proposed to use more electricity than my state already currently uses, is a good thing for the average joe where I live??
Honestly, I feel like many commenters here are in their own bubble and don't understand how much AI and tech generally is widely viewed as a net negative for society by huge swaths of the the population, and I don't really think it's an unwarranted perception.
These campaigns have historically amplified conflict. They do not care what the conflict is about.
But unlike some of the others, I’m hearing anti-AI sentiment from a wide range of people who don’t even use social media.
But unlike some of the others, I’m hearing anti-AI sentiment from a wide range of people who don’t even use social media.
All news is influenced by what’s popular social media these days. And that becomes part of what people talk about through the grapevine.
But no doubt there’s plenty of organic NIMBYism, anti tech growth stuff, and run of the mill fear of change and loss of control as society grows more abstract/centralized.
But no doubt there’s plenty of organic NIMBYism, anti tech growth stuff, and run of the mill fear of change and loss of control as society grows more abstract/centralized.
Applying the term NIMBYism to anti-AI and anti-DC sentiment is a gross abuse of terminology. Datacenters don't need to be in anybody's residential neighborhood.
These developments are always facing local resistance, HN has featured multiple articles about towns resisting. I previously lived in a place where they were building a quarry 30min outside of town and tons of houses had anti-quarry signs on their lawns. It was a big deal to appear anti-quarry even though it had little to do with their specific neighbourhood (except maybe increased highway traffic).
Most industrial development face local protests like this and it's often has large crossover with those who resist housing developments, and show up at town/city councils.
Most industrial development face local protests like this and it's often has large crossover with those who resist housing developments, and show up at town/city councils.
It’s also possible that the endless slew of doomer provocateur rhetoric coming from the labs and the boosters is just deeply unpopular.
Everyone lives in a world deeply affected by social media. Even if you've never looked at a screen your entire life, you have spent thousands of hours talking to and being informed by people who did.
There are 22 golf courses within a 30 minute drive from me, and people here a losing their minds over datacenter water usage...
People have a dozen reasons to refuse data centers being built in their communities and zero reasons to encourage it. They're little more than post-industrial mines that take limited resources (power, land, water, quiet) from a community, sell them for profit as compute, and siphon those profits away onto the books of far-flung megacorps with no community reward.
Meanwhile, golf courses are a traditional green space where people in a community gather for both work and leisure. They're not ideal themselves, but they at least provide some benefit against which their negatives can be weighed.
If all you hear from critics of data center building is water use complaints, that's strictly because you've chosen not to listen to people.
Meanwhile, golf courses are a traditional green space where people in a community gather for both work and leisure. They're not ideal themselves, but they at least provide some benefit against which their negatives can be weighed.
If all you hear from critics of data center building is water use complaints, that's strictly because you've chosen not to listen to people.
> data centers being built in their communities
> golf courses are a traditional green space where people in a community
I have a feeling those two sets of communities are disjoint
> golf courses are a traditional green space where people in a community
I have a feeling those two sets of communities are disjoint
How is this different from farming?
It takes the limited resources of land and water from a community and sells the result for profit as food or fuel. The vast majority of profit is made downstream and outside the community.
Golf courses being a traditional green place where people gather seems a bit far fetched to me when most of them are elite private clubs.
It takes the limited resources of land and water from a community and sells the result for profit as food or fuel. The vast majority of profit is made downstream and outside the community.
Golf courses being a traditional green place where people gather seems a bit far fetched to me when most of them are elite private clubs.
Most golf courses around me are open and anyone can go play for a cheap greens fee. The clubhouse has normal low end restaurant prices for a hot dog or a burger.
If the clubhouse is your idea of the valuable public good we can provide without 18 holes of grass.
As far as "valuable", the golf course is private property that the owners presumably bought and can do with as they see fit. If you want to develop it into something else, make an offer to them?
The golf courses are a business like any other (although we do have some publicly-owned golf courses around here too). The cost to play 9 holes on a weekday is $10 at one of them. I'm not really sure what you're asking for here.
The golf courses are a business like any other (although we do have some publicly-owned golf courses around here too). The cost to play 9 holes on a weekday is $10 at one of them. I'm not really sure what you're asking for here.
I associate golf with WASP-y business types who hobnob with their boss in an electric cart to get ahead, and can't stop talking about their hobby of chasing a little white ball.
Is that fair? Probably not. But I don't think golf is a particularly inclusive sport, unless you live in a golf course gated community... in which case everyone is included.
Is that fair? Probably not. But I don't think golf is a particularly inclusive sport, unless you live in a golf course gated community... in which case everyone is included.
In farming, the result is food that can be eaten.
Depends. Sometimes it's just biofuels.
Golf courses don't do that either...
When mines pay a sizeable share of their profits as local taxes, and obey environmental regulations, people suddenly start to like them.
Can you point to some examples of where that's happening?
These data centers are specifically being scouted for communities whose governance is too weak to negotiate for some "sizeable share of their profits" and too ill-prepared to have suitable environmental regulations on the books already. The Ivy League sharks planning these buildout initiatives are sharp people who are looking out for the interests of their employers and know how to pick locations where they have the best opportunity to exploit locals unprepared for their kind of esoteric deal-making, political lobbying, and lawfare. They'd be failing at their job if they did what you're suggesting, and that's why we don't really see that happen.
For any benefit to national or global society AI data centers might provide to someone, the buildout looks a lot like the dirty and exploitative stories of rail expansion in the 19th century. That rail infrastructure proved a good thing for the US, but that doesn't mean the process of making it happen was honest or good for the people immediately affected.
These data centers are specifically being scouted for communities whose governance is too weak to negotiate for some "sizeable share of their profits" and too ill-prepared to have suitable environmental regulations on the books already. The Ivy League sharks planning these buildout initiatives are sharp people who are looking out for the interests of their employers and know how to pick locations where they have the best opportunity to exploit locals unprepared for their kind of esoteric deal-making, political lobbying, and lawfare. They'd be failing at their job if they did what you're suggesting, and that's why we don't really see that happen.
For any benefit to national or global society AI data centers might provide to someone, the buildout looks a lot like the dirty and exploitative stories of rail expansion in the 19th century. That rail infrastructure proved a good thing for the US, but that doesn't mean the process of making it happen was honest or good for the people immediately affected.
States give the tax incentives, local municipalities are cheap to buy out, that's why all these towns quickly pass approval. $30 million annually is pennies for a datacenter and a windfall for a town.
This may sound crazy, but I’m glad the data centre being built near me (the new us-east-3) is being built by Amazon who pays lip service to local government and the community, as opposed to cartoon villain levels of saving a few pennies by forcing noise pollution on everyone else and everything else undesirable the other builders are doing.
Never thought I’d say that.
Never thought I’d say that.
> Can you point to some examples of where that's happening?
You mean the AI datacenters doing it? No, they are not doing it.
They seem to be doing the opposite. They being loud is really hard to accept, decorrelating the fans cooling them would probably pay for itself in less than an year. It's like it's some Capitan Planet villain building those things.
You mean the AI datacenters doing it? No, they are not doing it.
They seem to be doing the opposite. They being loud is really hard to accept, decorrelating the fans cooling them would probably pay for itself in less than an year. It's like it's some Capitan Planet villain building those things.
Got it. I misunderstood you. Agreed.
People are scared about the personal impact from AI, then backfill in justifications without even realizing they're doing it.
If the equivalent numbers for electricity and water usage were being being used for streaming video, I seriously doubt people would be demanding no more Netflix data centers. The news story would immediately die.
If the equivalent numbers for electricity and water usage were being being used for streaming video, I seriously doubt people would be demanding no more Netflix data centers. The news story would immediately die.
Nobody wants their electric rates to go up, the local water utility to have to raise rates to build a bigger plant, all in exchange for also losing good white collar jobs. That’s currently what AI data centre builders are selling.
Not exactly. There are some data centers being built in places that don't have the power and water to support them, and obviously it's rational for the locals to oppose them.
But I live in a place where we have plenty of water and relatively cheap power (lots of renewables). There's not much risk to data center construction, but people are opposing it here, too. Because for most people, it's not actually about that.
But I live in a place where we have plenty of water and relatively cheap power (lots of renewables). There's not much risk to data center construction, but people are opposing it here, too. Because for most people, it's not actually about that.
An obvious question is if the cheap power is going to stay cheap after a large power-user comes in who has a proven track record of trying to make everything cheap for themselves with no regard for anyone else.
Or another question to ask is - how does this data centre benefit the people who live there? If it doesn't, there's no reason they should want one to be built. Rubbish tips are necessary. I still don't want one built next to my house and would fight such a thing tooth and nail.
Or another question to ask is - how does this data centre benefit the people who live there? If it doesn't, there's no reason they should want one to be built. Rubbish tips are necessary. I still don't want one built next to my house and would fight such a thing tooth and nail.
Golf courses don’t have backup generators running 24/7, with humming you can hear from a meaningful distance away. They also don’t pollute the air.
This is a poor comparison, but I do get what you’re attempting here. It’s also absurd that we are leveling land everywhere around me to build warehouses. No one is really complaining about that, either.
This is a poor comparison, but I do get what you’re attempting here. It’s also absurd that we are leveling land everywhere around me to build warehouses. No one is really complaining about that, either.
People can care about more than one thing
Personally I would happily close down all golf courses and put them to better use as literally anything else.
Even just making them public parks would be great
Personally I would happily close down all golf courses and put them to better use as literally anything else.
Even just making them public parks would be great
I attended an auction of a golf course a few years ago. It went for a few thousand an acre. You could have shown up and bid on it.
The winner ended up just choosing to keep the current employees and keep operating it. Nobody, I mean nobody, wanted the land for development. It was in an era with basically no zoning either.
The winner ended up just choosing to keep the current employees and keep operating it. Nobody, I mean nobody, wanted the land for development. It was in an era with basically no zoning either.
No, in fact, most of us could not have just shown up and bid on it with any expectation of a meaningful outcome.
Coming in for a landing on a flight recently, I was amazed at the contrast I saw between a housing development of 50+ homes and a golf course across the road that took up the same amount of space.
I don’t get the appeal of golf. Hitting a ball into a hole with a crooked stick? But hey, that is just my personal preference. People can play whatever sport they want, or do whatever they want and call it a sport.
That said, it would be nice if the so called sport didn’t take so much land, water etc. Especially in prime locations
That said, it would be nice if the so called sport didn’t take so much land, water etc. Especially in prime locations
People can pretend to care about more than one thing.
Whether they actually actively oppose those things to the point of impacting building permits, that's a completely different matter. It really doesn't take much legislation to make golf courses economically unviable and force them to close, especially if you've got enough population within 30 minutes to support 22 of them (I speak from experience, I helped write a water reclamation ordinance that shut down at least one in my SoCal city)
If anyone actually bothered to talk to their local reps instead of posting internet comments about how much they "care", they'd get something done. If they don't, their care is just a fart in the wind for all the good it will do.
Whether they actually actively oppose those things to the point of impacting building permits, that's a completely different matter. It really doesn't take much legislation to make golf courses economically unviable and force them to close, especially if you've got enough population within 30 minutes to support 22 of them (I speak from experience, I helped write a water reclamation ordinance that shut down at least one in my SoCal city)
If anyone actually bothered to talk to their local reps instead of posting internet comments about how much they "care", they'd get something done. If they don't, their care is just a fart in the wind for all the good it will do.
Russia and China aren’t the ones constantly telling us AI will put us all out of a job, here’s why that’s a good thing, and why the government should dedicate billions of dollars to incumbent AI providers.
source needed. I'm not saying ur wrong, but source needed
Source or link?
What choice does that rag have? Of course they're going to backpedal and try to launder blame.
Surprisingly enough the us constitution specifies a right to free speech. Meanwhile who is funding the past two decades of pie in the sky bullshit from silicon valley in the media? Is promoting and unpopular opinion illegal? Since when do we judge the merits of an argument based on who articulates it?
[deleted]
This is by far the most realistic optimistic AI takeoff scenario I've seen, and more specifically it's the first one I've read that deals with both the AI alignment and power concentration issues in a sufficient way, even in a world where hard alignment is assumed (in this scenario the AIs are assumed to be misaligned until ~2038-39).
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
The thing that continues to irk me about these sorts of "papers" is how they refuse to remotely acknowledge the possibility that LLMs and Diffusion models won't lead to AGI, ASI, or whatever acronym they're foisting upon the populace.
If you won't even so much as acknowledge the possibility of error, your argument is hollow and empty. All the "choices" presume these labs are being completely honest and acting in some degree of good faith (relative to the systemic incentives of society in its present form), while in reality we're still just building and refining probability models with increasing accuracy of output and flexibility of processing (namely agents) but still lack actual "intelligence" of any real sort.
Show me a paper that doesn't merely presume inevitability of LLM-based AGI/ASI, and instead actually lays out the core paths that history suggests we're likely to encounter with any "world changing technology":
* In the best case, that the technology really will revolutionize the world and do everything promised by its biggest boosters (papers like this one)
* In the middle case, that it becomes just another tool in our collective toolkit, and the consequences of a revolution built on external investment fizzling out
* In the worst case, that the tool itself is so niche in its utility that investment collapses rather than fizzles out; what do we do with all this compute, now? Who owns the debt? Who foots the bill? How can we mitigate those existential risks?
I'm just rather nauseated by the continued trot of inevitablism masquerading as academia rather than an actual, neutral, bias-controlled-and-disclosed study that paints potentialities instead.
---
Having finished skimming through it, another comment springs to mind: Jesus Christ these things continue to be jingoist as absolute fuck. It's a fancier set of makeup for the same shitty western chauvinism worldview of American excellence and Manifest Supremacy.
Nah, I'm done with this trite garbage. Go proselytize to idiots, I'm not one of them.
If you won't even so much as acknowledge the possibility of error, your argument is hollow and empty. All the "choices" presume these labs are being completely honest and acting in some degree of good faith (relative to the systemic incentives of society in its present form), while in reality we're still just building and refining probability models with increasing accuracy of output and flexibility of processing (namely agents) but still lack actual "intelligence" of any real sort.
Show me a paper that doesn't merely presume inevitability of LLM-based AGI/ASI, and instead actually lays out the core paths that history suggests we're likely to encounter with any "world changing technology":
* In the best case, that the technology really will revolutionize the world and do everything promised by its biggest boosters (papers like this one)
* In the middle case, that it becomes just another tool in our collective toolkit, and the consequences of a revolution built on external investment fizzling out
* In the worst case, that the tool itself is so niche in its utility that investment collapses rather than fizzles out; what do we do with all this compute, now? Who owns the debt? Who foots the bill? How can we mitigate those existential risks?
I'm just rather nauseated by the continued trot of inevitablism masquerading as academia rather than an actual, neutral, bias-controlled-and-disclosed study that paints potentialities instead.
---
Having finished skimming through it, another comment springs to mind: Jesus Christ these things continue to be jingoist as absolute fuck. It's a fancier set of makeup for the same shitty western chauvinism worldview of American excellence and Manifest Supremacy.
Nah, I'm done with this trite garbage. Go proselytize to idiots, I'm not one of them.
One of the authors talks about this here: https://www.astralcodexten.com/p/introducing-plan-a
> It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.
> ...
> Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.
> ...
> If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.
I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".
The jingoism is off putting. I think Daniel says it's a political necessity: https://x.com/DKokotajlo/status/2075261194978640096
> It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.
> ...
> Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.
> ...
> If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.
I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".
The jingoism is off putting. I think Daniel says it's a political necessity: https://x.com/DKokotajlo/status/2075261194978640096
So it's a fan-fiction?
If carbon taxes are already a lethal policy for an political campaign, it's absurd to think that fears of ASI will create any real movement around pausing AI.
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
If that is true and one cares about a moratorium on progress in the US then it seems like the number one way is to meet people where they are: so water use misinformation, degrowth, power supply constraints. That does place all the people who push for these things in a different light. They may well be attempting to do what the AI safety labs are ostensibly trying to do.
As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.
As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.
If there's going to be any pause, I'm sure it will come from a populist movement. I just can't imagine misplaced worries about AI water use will translate into the kinds of policy the authors want to see.
Yeah it’s like shoving the top of a double pendulum. You will get some movement in one direction, but where it will precisely land is hard to predict. The water-use argument is already earning refinement by differentiating “AI datacenters” from “normal datacenters” in an effort to control the movement.
I imagine any populism movement will require rampant fearmongering to get a result. Considering the rough present alignments, presumably blue tribe focused propaganda will involve climate and inequality focused fear and red tribe focused propaganda will involve job loss. Grey tribe positioning is the P(doom) meme where everyone is rewarded for a high-P(doom) estimate.
I imagine any populism movement will require rampant fearmongering to get a result. Considering the rough present alignments, presumably blue tribe focused propaganda will involve climate and inequality focused fear and red tribe focused propaganda will involve job loss. Grey tribe positioning is the P(doom) meme where everyone is rewarded for a high-P(doom) estimate.
You should consider why the best ally to your position is misinformation.
Did anyone else catch the logical inconsistency between Plan C and A?
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
By legislative design? If a nuclear bomb could be made with hardware store finds the world would already be over. Big collaborative works raise the stakes and the observability for surveillance. Apply for a job at a defense supplier or even and energy company.
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
In Plan C the government essentially misses the opportunity to implement the multinational deal while the threat of covert projects is still low (fewer latest-gen chips unaccounted by tracking measures, worse models/algorithms to use for RSI). That's why it says the probability of a deal is lower and lower each month rather than outright zero.
Firstly, let me say that "AI 2040: Plan A" is a nicely done presentation. If feels like reading an interactive graphic novel with a narrator to boot. Can't wait for the book, series or movie :)
Plan A seems like a good start and am glad an effort is actually being made to address any potential dangers. The only weak link I see is that there is no way for inaccessible, third-world countries, non-aligned states, and malicious wealthy rogue agents to be regulated. All I hear is a way for regulating companies that, themselves, legally have to answer or are bound to their host nations. Basically, I don't see a way to hold non-aligned states accountable.
I see a lot of focus on the well-being and protection of AI, which is important to country economies but, the folks that have been affected by layoffs, not necessarily due to AI, plus the workforce that are now feeling the negative consequences of the AI burn are justified in being worried. Anyone that feels the need to criticize them, clearly is not being affected by AI in the same way. Job loss will lead to economic and population destabilization, far worse than anything that has transpired in modern times. Hopefully, those being squeezed now won't be ignored.
Plan A seems like a good start and am glad an effort is actually being made to address any potential dangers. The only weak link I see is that there is no way for inaccessible, third-world countries, non-aligned states, and malicious wealthy rogue agents to be regulated. All I hear is a way for regulating companies that, themselves, legally have to answer or are bound to their host nations. Basically, I don't see a way to hold non-aligned states accountable.
I see a lot of focus on the well-being and protection of AI, which is important to country economies but, the folks that have been affected by layoffs, not necessarily due to AI, plus the workforce that are now feeling the negative consequences of the AI burn are justified in being worried. Anyone that feels the need to criticize them, clearly is not being affected by AI in the same way. Job loss will lead to economic and population destabilization, far worse than anything that has transpired in modern times. Hopefully, those being squeezed now won't be ignored.
This is follows the classic AI policy influence playbook: enumerate a small handful of potential outcomes that make your preferred policy seem obviously correct, ignoring the fact that there are literally infinite more "plans" that could be reasonably predicted that would support wildly different policy prescriptions.
This is whole a slippery slope. Always building on assumption of infinite exponential growth. But every exponential is at a certain point a sigmoid.
Whenever I encounter these people I'm reminded of the meme about the baby who has doubled his weight in the three months since birth. At that growth rate, he'll weigh trillions of pounds by age 10.
Nothing follows from this empty platitude though, right? It can't inform you choices or decisions? It's just a disempowering thought?
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
AI is just a tool. It will be a tool as we constantly push the requirements of what qualifies as consciousness. Therefore I'm not worried about it. Almost all the predictions about future growth of technology were incorrect. We don't have flying cars, hoverboards, fusion, the matrix and so on. The goal of each of us is to live a better life. It always be and forcing people to give up luxuries for some external agenda based on slippery slopes is cruel and selfish.
We should have been under water, hunted by AI, overpopulated, killed by terrorist, smitten by god for our sins and so on. Luckily all it took was our privacy and a lot of tax money to survive.
We should have been under water, hunted by AI, overpopulated, killed by terrorist, smitten by god for our sins and so on. Luckily all it took was our privacy and a lot of tax money to survive.
Degrowth is a deeply unpopular policy around the world. Where would one even go to find a degrowth rally? I have to imagine everyone there would be there ironically.
> * they net shutting down grade schools or building them in your neck of the woods?*
My area (rural Iowa) has had several new schools built in the last 10 years. Net gain for sure.
> * they net shutting down grade schools or building them in your neck of the woods?*
My area (rural Iowa) has had several new schools built in the last 10 years. Net gain for sure.
> But every exponential is at a certain point a sigmoid.
strongly, no. its just hard to distinguish them. for example, radioactive decay. cmon
strongly, no. its just hard to distinguish them. for example, radioactive decay. cmon
Point taken. But it's interesting that the example you give is one of exponential decay rather than exponential growth, which is the context of this discussion. Where in nature has anything ever grown exponentially in perpetuity? Bacteria in a petri dish multiply exponentially. Until they don't.
> Why exactly will AI never be able to do my job?
Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities. Furthermore, AI agents lack any notion of an identity, so certainly are not capable of attaining legal personhood or being sued or fired, or owning property. I think slop burnout, cybersecurity, loss of privacy, even environment issues are far more concerning and real issues arising from AI than alignment or the prospect of mass labor displacement due to AI.
Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities. Furthermore, AI agents lack any notion of an identity, so certainly are not capable of attaining legal personhood or being sued or fired, or owning property. I think slop burnout, cybersecurity, loss of privacy, even environment issues are far more concerning and real issues arising from AI than alignment or the prospect of mass labor displacement due to AI.
> Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities.
There are multiple teams working on adding long-term memory to AI. This is not a fundamental problem.
Fine-tuning to store memory in the weights is something that almost works right now, btw.
There are multiple teams working on adding long-term memory to AI. This is not a fundamental problem.
Fine-tuning to store memory in the weights is something that almost works right now, btw.
Sounds like another Chinese Op to me; Ensuring Chinese compliance would be incredible hard to enforce or to check.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
> Ensuring Chinese compliance would be incredible hard to enforce or to check
consider SALT and other treaties from USSR vs USA cold war. they checked
consider SALT and other treaties from USSR vs USA cold war. they checked
I recommend actually reading their recommendation, because they get into the weeds about precisely how the US and China could address this in a trustless/auditable way. The TL;DR is that basically all of the relevant compute can be tracked.
Edit: Also, definitely not a Chinese op. The authors are prominent Americans, and are the folks responsible for the AI 2027 forecast that has pretty accurately predicted the current state of affairs today: https://ai-2027.com/
Edit: Also, definitely not a Chinese op. The authors are prominent Americans, and are the folks responsible for the AI 2027 forecast that has pretty accurately predicted the current state of affairs today: https://ai-2027.com/
Pretty accurately what? So now OpenBrain has an Agent-1 that makes their algorithmic progress 50% faster than other companies? If it was 50% more CVEs, that would be something, but I doubt any meaningful self improvement is achieved which the competitors are slower due that, which is the core of the prediction for 2026.
I wonder. There has been some headway in getting decentralized training runs to work.
Yeah, Cognition's work is interesting in that regard, but it still doesn't obviate the need for the chips--it just enables training on them when they're spread across multiple data centers.
The Plan A proposal estimates that the ownership of ~96% of AI relevant compute hardware can have its ownership traced, since the companies selling are very few.
The Plan A proposal estimates that the ownership of ~96% of AI relevant compute hardware can have its ownership traced, since the companies selling are very few.
I doubt we can even track all of the chip production capacity.
Forecasting that the GPU build-out will reach 100 trillion USD in 2034 is wild (that's triple the US GDP in 8 years). And another 10x within 2 years.
I am not sure where they believe that amount of capital could come from. It would require central bank level money printing never seen before.
https://ai-2040.com/supplements/compute-supplement
I am not sure where they believe that amount of capital could come from. It would require central bank level money printing never seen before.
https://ai-2040.com/supplements/compute-supplement
The article reads like a compilation of online chatter, social media post, etc. on AI. Not surprising to find unsubstantiated imaginary numbers there.
I feel like we will end up with a future that equally disappoints everyone and is somehow not covered in any plan here.
People overestimate progress in physical world.
2035:
robot population will soon be larger than the human one
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
I mean, what is a robot? If you add up all the vacuum cleaners, 3d printers, and dishwashers, that’s probably close or more than the human population.
This is dangerously naive and misguided. They claim to want to avoid centralization of control but propose a world police state of AI regulation. Governments exerting this much control will only end in war and tyranny.
> Governments exerting this much control will only end in war
elaborate. regulation -> war, how?
elaborate. regulation -> war, how?
If the production, and distribution of the tech is confined to one or two companies, then those two companies and the resulting power that comes to them, makes them the de facto world government. (Democratic choice may be a farce but it is a useful friction on net.)
If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.
Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.
Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?
If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?
If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.
Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.
Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?
If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?
> If the production, and distribution of the tech is confined to one or two companies
We already have at least 5 companies only in the US. Your whole premise is false.
We already have at least 5 companies only in the US. Your whole premise is false.
> Then, in the mid-2030s, they pause at AIs around the level of top human geniuses.
They being the US and China and by agreement.
It would be ideal, but there’s far too much money on the table to overcome human nature.
So my hope is we hit some kind of limits naturally.. Wishful thinking?
They being the US and China and by agreement.
It would be ideal, but there’s far too much money on the table to overcome human nature.
So my hope is we hit some kind of limits naturally.. Wishful thinking?
... and in the meantime people are looking at their AI bills and realizing tokens aren't worth what they cost. The frontier is getting the cost down, not getting intelligence up. In a cage match between this guy and "Ed", Ed wins.
Why did Scott Alexander (one of the authors of the original AI 2027 paper) not join/contribute to this one?
Hey, author here! Scott did contribute, but less than before.
On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"
On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"
I wonder if they are double-counting Anthropic's leased capacity from SpaceX under SpaceX again.
Associated post: Introducing Plan A
https://www.astralcodexten.com/p/introducing-plan-a
https://www.astralcodexten.com/p/introducing-plan-a
Stop trying to cram your "P" into "AI".
Nah, screw that. I won't be waiting that long. I'll be 61 in 2040. I'd love humanity to take a shot at clinical immortality way sooner.
Kind of feels like fusion power at this point, always just around the corner.
What predictions about the technology are the authors making that you do not believe?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
Self improvement of agent-1 is not achieved. Sure, people in AI labs write python code with AI, but I doubt it resulted in 50% algorithmic effiecency in training. Writing python code never was the bottleneck, if it was, AI labs could hire more people to do it. And this is core of the prediction.
LLM adoption is 30% in the charts I saw googling for "ai adoption". An example of capibility: I have had Claude one shot an RL agent that learns connect four in 30 minutes. That's PhD level stuff.
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
More wild speculation, now with wishful thinking spread on top.
Excellent work by Daniel and the authors. 47,000 words plus supplements is a huge read (and re-read), and an even bigger think-and-write.
My early analysis of the analysis:
https://lifearchitect.substack.com/p/the-memo-special-editio...
My early analysis of the analysis:
https://lifearchitect.substack.com/p/the-memo-special-editio...
[1] https://ai-2027.com/