"So I'd like to engage AI risk from both these perspectives. I think the arguments for superintelligence are somewhat silly, and full of unwarranted assumptions.
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
"John wore a T-shirt that featured a smiley face with a bullet hole in the forehead from which trickled a few drops of blood"
Sounds like a Watchmen Comedian logo t-shirt. It could be construed as a bold choice but was probably just what was on the top of his t-shirt stack that day.
"but I can't help but see parallels between today and the Industrial Revolution"
You're not the only one.
The current Pope Leo XIV explicitly named himself after the the previous Leo, Pope Leo XIII, who was pope during the Industrial Revolution (1878-1903) and issued the influential Encyclical Rerum novarum (Rights and Duties of Capital and Labor) in response to the upheaval.
“Pope Leo XIII, with the historic Encyclical Rerum novarum, addressed the social question in the context of the first great industrial revolution,” Pope Leo recalled. “Today, the Church offers to all her treasure of social teaching in response to another industrial revolution and the developments of artificial intelligence.” A name, then, not only rooted in tradition, but one that looks firmly ahead to the challenges of a rapidly changing world and the perennial call to protect those most vulnerable within it.”
Imagine hearing pre-attention-is-all-you-need that "AI" could do something that Donald Knuth could not (quickly solve the stated problem in collaboration with his friend).
The idea that this (Putnam perfect, IMO gold, etc) is all just "statistical parrot" stuff is wearing a little thin.
“The laws of nature should be expressed in beautiful equations.”
- Paul Dirac
“It is, indeed, an incredible fact that what the human mind, at its deepest and most profound, perceives as beautiful finds its realisation in external nature. What is intelligible is also beautiful. We may well ask: how does it happen that beauty in the exact sciences becomes recognizable even before it is understood in detail and before it can be rationally demonstrated? In what does this power of illumination consist?”
- Subrahmanyan Chandrasekhar
“I often follow Plato’s strategy, proposing objects of mathematical beauty as models for Nature.”
“It was beauty and symmetry that guided Maxwell and his followers.”
- Frank Wilczek
“Beauty, is bound up with symmetry.”
- Herman Weyl
"Still twice in the history of exact natural science has this shining-up of the great interconnection become the decisive signal for significant progress. I am thinking here of two events in the physics of our century: the rise of the theory of relativity and that of the quantum theory. In both cases, after yearlong unsuccessful striving for understanding, a bewildering abundance of details was almost suddenly ordered. This took place when an interconnection emerged which, thought largely unvisualizable, was finally simple in its substance. It convinced through its compactness and abstract beauty – it convinced all those who can understand and speak such an abstract language."
- Werner Heisenberg
Maybe (just maybe) these things (whatever you want to call them) will (somehow) gain access to some "compact", beautiful, "largely unvisualizable" "interconnection" which will be the self-evident solution. And if they do, many will be sure to label it a statistical accident from a stochastic parrot. And they'll right, for some definitions of "statistical", "accident", "stochastic", and "parrot".
Donald Knuth is an extremal outlier human and the problem is squarely in his field of expertise.
Claude, guided by Filip Stappers, a friend of Knuth, solved a problem that Knuth and Stappers had been working on for several weeks. Unfortunately, it doesn't seem (from my quick scan) to have been stated how long (or how many tokens or $) it took for Claude + Stappers to complete the proof.
In response, Knuth said: "It seems that I’ll have to revise my opinions about “generative AI” one of these days."
Seems like good advice. From reading elsewhere in this comment section, the goalposts seem to be approaching the infrared and will soon disappear from the extreme redshift due to rate at which they are receding with each new achievement.
It’s time. According to my doctors. All further treatments are pointless. So, please donate so my kids can create a funeral worthy of my keyboard, Pixelbreaker! So I can make a worthy entrance for reuniting with my one true love, Jennell Jaquays.
My daughter Cynthia Elizabeth Heineman, will be making the arrangements
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
[0] https://www.nature.com/articles/s41467-024-53759-4