Speculations Concerning the First Ultraintelligent Machine (1965) [pdf](languagelog.ldc.upenn.edu)
languagelog.ldc.upenn.edu
Speculations Concerning the First Ultraintelligent Machine (1965) [pdf]
https://languagelog.ldc.upenn.edu/myl/Good1964.pdf
52 comments
>> It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make, since it will lead to an “intelligence explosion.”
> Well, that didn't happen.
Predictions are not false because their claims are not in the past yet. That bar would make all predictions wrong.
The faster things happen, the higher the speed-of-progress expectation bar gets raised. This is how objectively compounding progress gets interpreted subjectively as linear, or even as a stall. Despite the dramatically increased rate of progress compared to the context of decades or centuries of speculations about cognitive machines.
Models being used to write a lot of the code for new models is a strong suggestion of compounding capability. With new models achieving higher scores. Not proof, but a high bar for evidence that we may be in that explosion now.
The fact that models trained to match SOTA model behavior (i.e. distillation), now learn much faster and more cheaply, than models trained on human behavior, is also strong evidence that capabilities are compounding.
> Well, that didn't happen.
Predictions are not false because their claims are not in the past yet. That bar would make all predictions wrong.
The faster things happen, the higher the speed-of-progress expectation bar gets raised. This is how objectively compounding progress gets interpreted subjectively as linear, or even as a stall. Despite the dramatically increased rate of progress compared to the context of decades or centuries of speculations about cognitive machines.
Models being used to write a lot of the code for new models is a strong suggestion of compounding capability. With new models achieving higher scores. Not proof, but a high bar for evidence that we may be in that explosion now.
The fact that models trained to match SOTA model behavior (i.e. distillation), now learn much faster and more cheaply, than models trained on human behavior, is also strong evidence that capabilities are compounding.
I don't think it's a knock on the original prediction to say it didn't turn out to be true. In fact, at the time it probably felt like a pretty conservative prediction. That being said, if you think back to the year 2000, we were pretty far from building such a machine at that time.
To me it's an interesting lesson in how hard it is to predict technology advances which still applies to our own predictions of the future.
To me it's an interesting lesson in how hard it is to predict technology advances which still applies to our own predictions of the future.
Early on he's also predicting GHz serial computation by 1980: if you adjust for compute advancement rate he's probably a lot closer.
> Not proof, but a high bar for evidence that we may be in that explosion now.
I agree, that's why I said "we're seeing things we have working towards this". I think that the "jagged intelligence" that is often used to describe our curent models is confusing a lot of people. On the one hand you have models failing basic "trick" questions that a 5yo would get, but on the other hand you also have models that write better/faster kernels that can run faster/better models, and are currently used to improve the next generation (via dataset prep, filtering, RL environment creation/curation and so on).
I agree, that's why I said "we're seeing things we have working towards this". I think that the "jagged intelligence" that is often used to describe our curent models is confusing a lot of people. On the one hand you have models failing basic "trick" questions that a 5yo would get, but on the other hand you also have models that write better/faster kernels that can run faster/better models, and are currently used to improve the next generation (via dataset prep, filtering, RL environment creation/curation and so on).
The twentieth century is in the past.
Are you confusing 20th century with 2000s?
>>> It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make, since it will lead to an “intelligence explosion.”
>> Well, that didn't happen.
> Predictions are not false because their claims are not in the past yet.
"Within the twentieth century" places it definitely in the past, and definitely false.
>> Well, that didn't happen.
> Predictions are not false because their claims are not in the past yet.
"Within the twentieth century" places it definitely in the past, and definitely false.
> It is more probable than not that, within the twentieth century
Ouch! Well ok then!
Zoom out in time, and the explosion started in the 20th century. But the pop, while near, has yet to come.
Mea culpa.
Ouch! Well ok then!
Zoom out in time, and the explosion started in the 20th century. But the pop, while near, has yet to come.
Mea culpa.
> > The required high degree of connectivity might be attained with the help of microminiature radio transmitters and receivers.
> Hahaha, this is straight out of 60s-70s sci-fi, where their best futuristic interfaces were smaller CRT screens / flashy keys, etc.
Actually that is not far off from how high bandwidth interconnects work, but through other media.
> Hahaha, this is straight out of 60s-70s sci-fi, where their best futuristic interfaces were smaller CRT screens / flashy keys, etc.
Actually that is not far off from how high bandwidth interconnects work, but through other media.
Agreed. It's basically RF guided by wires, or light guided by glass/plastic.
And... I hate to say it, but I've been interacting with computers connected to WiFi about 90% of the time over the last week. Rarely used a physical Ethernet link.
And... I hate to say it, but I've been interacting with computers connected to WiFi about 90% of the time over the last week. Rarely used a physical Ethernet link.
Funny to think that WiFi is in fact just a specialized variant of the radio technology explosion that started with Marconi.
Technically of course the path is continual evolution, but the applications are wildly different.
Technically of course the path is continual evolution, but the applications are wildly different.
Yeah the hyperparallelism and the fact that it would emerge from machine translation are shockingly good calls. There are things in there that resemble the bag of words model, his 'clumps' sound a lot like tokenization, and he has concepts like RAG.
Also, of the early AI writers, I think he was the least doom-y. Norbert wiener and I think von neumann were very concerned about the social effects of the singularity (von neumann's word for it).
Vernor Vinge has an 80s essay calling down to Good a lot, and credits him with a 'meta golden rule' which aspirationally says that what comes after us will care about their origins.
Also, of the early AI writers, I think he was the least doom-y. Norbert wiener and I think von neumann were very concerned about the social effects of the singularity (von neumann's word for it).
Vernor Vinge has an 80s essay calling down to Good a lot, and credits him with a 'meta golden rule' which aspirationally says that what comes after us will care about their origins.
> It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make, since it will lead to an “intelligence explosion.”
It’s not uncommon to find these kinds of predictions from the twentieth century, which of course with hindsight can seem wildly optimistic.
But I think there are understandable reasons for that optimism.
Imagine you were born in the early twentieth century, say roughly 1910 +/- 10 years. If you had been born then, you would probably have lived through all of the following:
- The transition from horses to motor vehicles
- The invention of the airplane, from first flight to supersonic
- Not one but two world wars
- The splitting of the atom
- Human spaceflight, including traveling to the moon
If you’d been born slightly earlier, you could also add electrification to that list.
When I compare what I’ve seen in my life so far, I don’t think anything comes close (for better and worse) to even one of those things. So yeah, of course they were optimistic about technological progress—look at what they just lived through.
It’s not uncommon to find these kinds of predictions from the twentieth century, which of course with hindsight can seem wildly optimistic.
But I think there are understandable reasons for that optimism.
Imagine you were born in the early twentieth century, say roughly 1910 +/- 10 years. If you had been born then, you would probably have lived through all of the following:
- The transition from horses to motor vehicles
- The invention of the airplane, from first flight to supersonic
- Not one but two world wars
- The splitting of the atom
- Human spaceflight, including traveling to the moon
If you’d been born slightly earlier, you could also add electrification to that list.
When I compare what I’ve seen in my life so far, I don’t think anything comes close (for better and worse) to even one of those things. So yeah, of course they were optimistic about technological progress—look at what they just lived through.
> The required high degree of connectivity might be attained with the help of microminiature radio transmitters and receivers.
I mean pretty much everything has a wifi chip these days
I mean pretty much everything has a wifi chip these days
> “intelligence explosion”
> Well, that didn't happen.
Not only that, but in my cynical eyes the proliferation of LLMs has triggered a stupidity explosion. Either that, or it just made it blatantly obvious by how much stupidity we have been surrounded the whole time without realizing it. No other development demonstrated so clearly that the dark ages where we believed in sorcery and miracles have never really ended.
> Well, that didn't happen.
Not only that, but in my cynical eyes the proliferation of LLMs has triggered a stupidity explosion. Either that, or it just made it blatantly obvious by how much stupidity we have been surrounded the whole time without realizing it. No other development demonstrated so clearly that the dark ages where we believed in sorcery and miracles have never really ended.
I think it is exposing how weakly people grasp communications. The amount of implied everything in most people's communications is extremely high, and those same people do not understand they are using implied information that a non-human construct really struggles to figure out what the human is talking about. I help a lot of non-technical people use AI, and asking them not to imply and to explain what they mean... many cannot parse, the implied version is all they have ever held in their head, and my telling them that they are using implied information confuses them, they don't know or never realized the full meaning of their usage of language.
Yes. Furthermore, when I write precisely - because it's an important subject (read: will cost the company money) where we must be on the same page - people's eyes glaze over, and they ask for a meeting or a phone call instead. So, we do that, and I answer all the questions, and then everyone goes away with a different idea about what we're talking about. The only person who doesn't do that is our controller, who'll tell me "I had to read this three times, but now I understand", and I love her for it. Sadly, she doesn't make decisions, just pays the bills.
I'm sure it's my fault, somehow - but I'm a damn good writer; I just don't know how to communicate with people who won't put forth the effort to engage with detail.
The latest advice is "put in through [LLM]". Which, yeah, takes out the details, and makes bullet points, but sometimes the details are the point... Right?
I'm sure it's my fault, somehow - but I'm a damn good writer; I just don't know how to communicate with people who won't put forth the effort to engage with detail.
The latest advice is "put in through [LLM]". Which, yeah, takes out the details, and makes bullet points, but sometimes the details are the point... Right?
> I'm sure it's my fault, somehow - but I'm a damn good writer; I just don't know how to communicate with people who won't put forth the effort to engage with detail.
I used to put effort into those people. It was a waste of time. Now I just do everything I can to ignore them, work around them, etc.
Knowledge workers who won't engage with the material are just like ditch diggers who won't touch a shovel. They're dead weight, and organizations that fail to cull them are doing it all wrong.
Sadly, that's most of them these days.
I used to put effort into those people. It was a waste of time. Now I just do everything I can to ignore them, work around them, etc.
Knowledge workers who won't engage with the material are just like ditch diggers who won't touch a shovel. They're dead weight, and organizations that fail to cull them are doing it all wrong.
Sadly, that's most of them these days.
Careful, because some of those you overlook are probably favorites of management. For who knows what reason, but it surprises me how many times a seemingly deadweight team member just happens to be the owner's kid or some such nepo-baby.
Yeah I've stepped on that face rake a few times. I still think it's a decent heuristic, don't see any reason to change.
I realize this may sounds like a lot of effort, but I find telling stories of how a person would use the system, with theatrics of confusion and failure, tend to penetrate the dull unwilling to think participants I must engage to get my work completed. I'm not an actor or a good speaker, I grew up with a severe stutter, and I find I have to "put on a show" to generate the dawn of comprehension in others.
Funnily enough, I am (was) an actor. Public speaking and telling stories are my jam. So... I do all that. I love making wacky analogies to help people understand complex systems.
Higher-level decision-making, I find, isn't so much like that. Sometimes it's just tradeoffs: "if the company's going this direction, then A is better; but if that, then B". But they haven't even recognized the this and the that - and don't want to think them through transparently - so I've got to choose the A or the B in the dark. Actually, it's worse than that, because some people have (by implication) already committed to the this, and others to the that, but neither side wants to talk about either, because that would create conflict. So, the best (job- / career-wise) choice for me is to predict which of A or B best satisfies whichever side will be most influential when the bill for A or B comes due. Either could be (technically speaking) correct, but the choice is never made on a technical basis.
Higher-level decision-making, I find, isn't so much like that. Sometimes it's just tradeoffs: "if the company's going this direction, then A is better; but if that, then B". But they haven't even recognized the this and the that - and don't want to think them through transparently - so I've got to choose the A or the B in the dark. Actually, it's worse than that, because some people have (by implication) already committed to the this, and others to the that, but neither side wants to talk about either, because that would create conflict. So, the best (job- / career-wise) choice for me is to predict which of A or B best satisfies whichever side will be most influential when the bill for A or B comes due. Either could be (technically speaking) correct, but the choice is never made on a technical basis.
I've been working for far too freaking long, so I've had a series of pointy haired bosses (referencing Dilbert here). I once had a boss that really fought for the technical reasons every time, exhausing upper management until they gave in. Years later, socially chatting with one of the upper managers from that company, which I'd long left, about that manager and how much his team loved him. The manager's response was "that was the last on-spectrum manager we hired. never again!" and my opinion dimmed.
When I read this, there was a comment right below yours saying that they bet the military had LLMs in the 1980s. Consider your point proved.
> Well, that didn't happen.
The military had stealth aircraft in the 70's. I'll bet they had LLM or better in the 80's and the tech we have now is the consumer-grade version they seeded into industry in the 2020's.
The military had stealth aircraft in the 70's. I'll bet they had LLM or better in the 80's and the tech we have now is the consumer-grade version they seeded into industry in the 2020's.
There's 0 chance of that. You can maybe say that for things like material science, nuclear stuff, weird physics, and so on (basically anything relating to making big booms, delivering big booms, or ensuring others can't make big booms). But having LLMs as we understand them now in the 80s would be impossible. They simply didn't have the compute necessary. The entire world combined didn't have it.
For reference, a single 4090 GPU has more FP8 flops than the top supercomputer in 2007. A 4x 4090 computer (something you can buy today for ~10k) would be better than to top supercomputer in 2010 [1]. There's a reason deep learning only started to really work in the past few decades. We had the "theory" for a while, but no compute to actually put it in practice. And the current models are being trained on 10s to 100s of thousand of enterprise GPUs, that make the 4090 look like a toy.
[1] - https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...
For reference, a single 4090 GPU has more FP8 flops than the top supercomputer in 2007. A 4x 4090 computer (something you can buy today for ~10k) would be better than to top supercomputer in 2010 [1]. There's a reason deep learning only started to really work in the past few decades. We had the "theory" for a while, but no compute to actually put it in practice. And the current models are being trained on 10s to 100s of thousand of enterprise GPUs, that make the 4090 look like a toy.
[1] - https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...
Another axis would be power consumption. Pretty sure if you want LLM training in 1980 you'd also need fusion power in 1980. And it would be very, very hard to hide the cooling apparatus of your datacenter.
All that aside, it would also likely mean that the secret chip fabs eclipse the volume of the non-secret chip fabs by orders of magnitude... covertly? Or are we imagining chips that are 30yr ahead of their time? Either way it's just absurdities upon absurdities..
All that aside, it would also likely mean that the secret chip fabs eclipse the volume of the non-secret chip fabs by orders of magnitude... covertly? Or are we imagining chips that are 30yr ahead of their time? Either way it's just absurdities upon absurdities..
Why are you so certain you know what tech existed in the 80's?
NASA used Commodore 64's as part of the constellation of computers needed to launch the space shuttle. That whole program was pretty hokey. That silliness doesn't preclude some radically advanced compute tech from existing elsewhere at the same time. Of course something like that would be ultra secret and you and I wouldn't be told about it.
You have to be told something before you can believe it, right? You also get to choose whether you believe that something you're told is 100% god's honest truth or a load of bullshit as well. It's kindof sad to see people try to gatekeep thought here. Lot of rigid thinkers.
I recall Intel in the 90's trying to build clockless pentiums. They couldn't figure it out or maybe it had integration problems with clockful peripherals and it never came to fruition. But I wondered if that was some furtive attempt to seed some tech into industry. I just try to keep an open mind to these sorts of things.
NASA used Commodore 64's as part of the constellation of computers needed to launch the space shuttle. That whole program was pretty hokey. That silliness doesn't preclude some radically advanced compute tech from existing elsewhere at the same time. Of course something like that would be ultra secret and you and I wouldn't be told about it.
You have to be told something before you can believe it, right? You also get to choose whether you believe that something you're told is 100% god's honest truth or a load of bullshit as well. It's kindof sad to see people try to gatekeep thought here. Lot of rigid thinkers.
I recall Intel in the 90's trying to build clockless pentiums. They couldn't figure it out or maybe it had integration problems with clockful peripherals and it never came to fruition. But I wondered if that was some furtive attempt to seed some tech into industry. I just try to keep an open mind to these sorts of things.
Far-out theories need solid evidence to be believable. However, there is not even circumstantial evidence to suggest that anybody was five decades ahead of public technology. You might rather claim that the military has super advanced aircraft, at least we have lots of sightings of UFOs doing apparently miraculous aerial acrobatics.
50 years for large models is a stretch, though TI did have smaller, real-time field-trainable neural nets almost 40 years ago:
https://en.wikipedia.org/wiki/FGM-148_Javelin#Seeker
It was featured in one of the Modern Warfare games from around 20 years ago, and I looked it up because back then, I called bs on the fire & forget aspect.
https://en.wikipedia.org/wiki/FGM-148_Javelin#Seeker
It was featured in one of the Modern Warfare games from around 20 years ago, and I looked it up because back then, I called bs on the fire & forget aspect.
"Military grade" means overpriced and 5-10 years late.
I just try to keep an open mind to these sorts of things.
An open mind doesn't mean "a mind full of open circuits."
Why are you so certain you know what tech existed in the 80's?
Useful LLMs could not have been built without deep-UV semiconductor fabrication at scale, which in turn can't exist without a massive, complex supply chain that envelops the globe like a spiderweb.
Such technology could not have been built speculatively, or without people noticing, or with the physics and material science of the day. We didn't even know how to make blue LEDs then!
An open mind doesn't mean "a mind full of open circuits."
Why are you so certain you know what tech existed in the 80's?
Useful LLMs could not have been built without deep-UV semiconductor fabrication at scale, which in turn can't exist without a massive, complex supply chain that envelops the globe like a spiderweb.
Such technology could not have been built speculatively, or without people noticing, or with the physics and material science of the day. We didn't even know how to make blue LEDs then!
Because an LLM is the only way to express intelligence with a computer. So your conception of ram/cpu/storage/networks and learning (training on huge corpus of data) is the only way to do artificial intelligence?
LLM's are the perfect consumer-grade AI, safe to release to the public. It lacks creativity so little chance for a runaway super-intelligence, and it gives the people real tools. It's the best case. Isn't it tidy how reality is? Almost seems planned.
LLM's are the perfect consumer-grade AI, safe to release to the public. It lacks creativity so little chance for a runaway super-intelligence, and it gives the people real tools. It's the best case. Isn't it tidy how reality is? Almost seems planned.
“Keeping an open mind is a virtue—but, as the space engineer James Oberg once said, not so open that your brains fall out.”
― Carl Sagan, The Demon-Haunted World: Science as a Candle in the Dark
I had research lab exposure to what the US military had in the 80's, the groups I saw were hyper focused on 3D visualizations in support of in-the-field strategy. Think wireframes superimposed over a live camera feed, with those wireframes being the horizon line and what are the items beyond their current range of view. A building wall is simply a wireframe, and the floors and every type of sensor they could get to feed into visualizations to see inside and past buildings, hills, any navigable area.
That brings me back to this wireframe tank combat game we used to play on the LAN in middle school. I think it was called Spectre.
the same military is using computers with Windows and Xbox controllers
and got upset when Anthropic didn't want to let them use their LLM
and got upset when Anthropic didn't want to let them use their LLM
While I think GP comment was a bit looney, the first sentence is a fairly weak counterargument. Xbox controllers especially are used specifically for familiarity purposes. The US military can and does produce significantly more advanced control planes, but they aren't necessary for all purposes and if something is simple enough it's easier to not have to train people on something new. Likewise creating an own general-purpose OS would require significant time into training people how to use it.
sure, I was just talking about this Hollywood myth that the military is far ahead of everyone else, including in computing matters when we all know the military procurement birrocracy
I'm not saying the movie War Games was a limited hangout for a super-intelligent secret military computer system/network that already existed. I'm just saying it could have been.
I see those trillion dollar data centers we're building in the US are classed as military installations or at least appear to be a public-private partnerships where the data centers are being built on air force bases.
Perhaps the consumer compute tech has advanced to the point that it's good enough to host the next phase of intelligence for the military. Maybe all the water AI datacenters need is for energy, not cooling.
I see those trillion dollar data centers we're building in the US are classed as military installations or at least appear to be a public-private partnerships where the data centers are being built on air force bases.
Perhaps the consumer compute tech has advanced to the point that it's good enough to host the next phase of intelligence for the military. Maybe all the water AI datacenters need is for energy, not cooling.
> I'll bet they had LLM or better in the 80's and the tech we have now is the consumer-grade version they seeded into industry in the 2020's.
They didn't have LLMs but they did have "AI" already (for a fair while by then). It wasn't much anything like what we have now really, but it did exist and by the current standards of that time period it was pretty much straight out of science fiction. (Imagine how shocked they'd be seeing what all we have now. "Supercomputers" in nearly every pocket, widespread broadband Internet, LLMs, etc, etc.) You're definitely right though in thinking that they had technologies far beyond what they told the public about. That's been the case since before my own lifetime at least, and absolutely certainly still true today.
They didn't have LLMs but they did have "AI" already (for a fair while by then). It wasn't much anything like what we have now really, but it did exist and by the current standards of that time period it was pretty much straight out of science fiction. (Imagine how shocked they'd be seeing what all we have now. "Supercomputers" in nearly every pocket, widespread broadband Internet, LLMs, etc, etc.) You're definitely right though in thinking that they had technologies far beyond what they told the public about. That's been the case since before my own lifetime at least, and absolutely certainly still true today.
that's really unlikely. in the 80s the US military had several AI labs and was working with academia...I worked with some. the focus was nearly exlucsively on logic programming and 'expert systems', a path which failed miserably leading to an 'AI winter'.
somewhere around 1990, working at a Dod lab, I picked up on the 'neural network' craze and did a little work. it seemed interesting, but not that much different than using adaptive filters for signal processing. that seemed to be the general consensus, that there wasn't a lot of of interesting behavior or depth there. the number of weights and sizes of super computers at the time were a good 3-4 decimal orders of magnitude lower than what's being thrown today.
so unless the military ran a giant expensive program just to hide the fact that they actually had machines 10000x faster than the $50M (in 80s money) supercomputers of the day, and had trained them on a corpus they whipped up out of nowhere (the amount of information on the internet at the time could probably have fit on a few of todays phones), it probably didn't happen.
somewhere around 1990, working at a Dod lab, I picked up on the 'neural network' craze and did a little work. it seemed interesting, but not that much different than using adaptive filters for signal processing. that seemed to be the general consensus, that there wasn't a lot of of interesting behavior or depth there. the number of weights and sizes of super computers at the time were a good 3-4 decimal orders of magnitude lower than what's being thrown today.
so unless the military ran a giant expensive program just to hide the fact that they actually had machines 10000x faster than the $50M (in 80s money) supercomputers of the day, and had trained them on a corpus they whipped up out of nowhere (the amount of information on the internet at the time could probably have fit on a few of todays phones), it probably didn't happen.
> so unless the military ran a giant expensive program just to hide the fact that they actually had machines 10000x faster than the $50M (in 80s money) supercomputers of the day, and had trained them on a corpus they whipped up out of nowhere (the amount of information on the internet at the time could probably have fit on a few of todays phones), it probably didn't happen.
Well, did they? They do fritter away a lot of money. Maybe there's a long-running black budget somewhere that could account for it.
You are framing your argument in technologies and technological progressions that you are familiar with. Reality isn't so neat though and there are surely many technological paths to artificial intelligence they could have exploited.
Well, did they? They do fritter away a lot of money. Maybe there's a long-running black budget somewhere that could account for it.
You are framing your argument in technologies and technological progressions that you are familiar with. Reality isn't so neat though and there are surely many technological paths to artificial intelligence they could have exploited.
I’ll take that bet
I think it's worth noting the date of this when people claim that AI is being rushed without asking any of the important questions of what it might mean for us. This has been a topic of serious discussion for many many years.
It feels like the people who laughed at these fanciful ideas then are the ones now pretending those same ideas are new revelations that have gone unconsidered.
It feels like the people who laughed at these fanciful ideas then are the ones now pretending those same ideas are new revelations that have gone unconsidered.
Up to five years ago[^1], we lived an age of pure speculation about intelligence and minds. That age brought about multiple theories of the mind, and to me, their finest legacy are arguments of the form "$X is not $concept because $theory", where any given day $X can be "LLM" or "LLM+harness+tool", "agent" and $concept can be "intelligent", "aware", "rational", "long-termed", , etc. Those arguments, in the context of $theory, are not wrong, and they are in fact quite useful. But there's only so much utility in a sieve.
We are going to need positive theories of the mind, and we will have to build them using terms and bits that avoid confrontation with the old guard. So, maybe instead of "mind" we will say "artilect"[^2][^3], and then move on to engineer desirable properties. For some of those properties ("alignment", "paucity", Kantian analysis and synthesis) we are already there or aiming for, but as we design to take on more and more challenging deeds, we may need to go back, steal and rebrand other old ideas. For example, "long term goals" and "self-preservation" are strongly linked in humans. Can we reify that relationship in the new artilects we make?
Other things we will need to make from whole cloth. For example, minds with an intuitive understanding of, and capability to manipulate: quantum mechanics, biological organisms, the workings of an ecosystem, the stock market, a stock company, or itself and other minds.
[^1]: YMMV, depending on where you draw the line between fine intelligence and stochastic parrots.
[^2]: Not the dictionary meaning, but the term as used by Alastair Reynolds in "Blue Remembered Earth (2012)"
[^3]: We may not use the term "LLM", because of the stochastic parrot crowd, because LLMs become one of many ingredients, or simply (but unlikely) because we find a completely different tool for the kit and discard LLMs entirely.
We are going to need positive theories of the mind, and we will have to build them using terms and bits that avoid confrontation with the old guard. So, maybe instead of "mind" we will say "artilect"[^2][^3], and then move on to engineer desirable properties. For some of those properties ("alignment", "paucity", Kantian analysis and synthesis) we are already there or aiming for, but as we design to take on more and more challenging deeds, we may need to go back, steal and rebrand other old ideas. For example, "long term goals" and "self-preservation" are strongly linked in humans. Can we reify that relationship in the new artilects we make?
Other things we will need to make from whole cloth. For example, minds with an intuitive understanding of, and capability to manipulate: quantum mechanics, biological organisms, the workings of an ecosystem, the stock market, a stock company, or itself and other minds.
[^1]: YMMV, depending on where you draw the line between fine intelligence and stochastic parrots.
[^2]: Not the dictionary meaning, but the term as used by Alastair Reynolds in "Blue Remembered Earth (2012)"
[^3]: We may not use the term "LLM", because of the stochastic parrot crowd, because LLMs become one of many ingredients, or simply (but unlikely) because we find a completely different tool for the kit and discard LLMs entirely.
> depending on where you draw the line between fine intelligence and stochastic parrots.
I'm always amused by this argument. "Stochastic parrot" is pretty much what most humans are, most of the time. I suppose you could add "stochastic parrots who believe they're conscious" if you want to be complete.
Where do your thoughts come from? This has been explored quite a bit long before LLMs, and the answer is, they essentially come from our subconscious minds, and the evidence shows that the we don't exert nearly as much conscious control over that as we imagine we do.
Given what we know, it's quite reasonable to conclude that our thoughts are a function of our biological neural network(s), which are responding to inputs with the outputs that they've been programmed to produce by a lifetime of training.
How is that any different from an LLM, in terms of the overall principle? I.e., if LLMs are stochastic parrots, then so, most likely, are we.
I'm always amused by this argument. "Stochastic parrot" is pretty much what most humans are, most of the time. I suppose you could add "stochastic parrots who believe they're conscious" if you want to be complete.
Where do your thoughts come from? This has been explored quite a bit long before LLMs, and the answer is, they essentially come from our subconscious minds, and the evidence shows that the we don't exert nearly as much conscious control over that as we imagine we do.
Given what we know, it's quite reasonable to conclude that our thoughts are a function of our biological neural network(s), which are responding to inputs with the outputs that they've been programmed to produce by a lifetime of training.
How is that any different from an LLM, in terms of the overall principle? I.e., if LLMs are stochastic parrots, then so, most likely, are we.
That's a pretty early definition of what we now call ASI (Artificial Super Intelligence). In the next paragraph the author goes to describe what we today call the "singularity" (ASI designing better ASI). But that term seems to be associated to some very weird communities, so the concept is relegated to sci-fi. Even though we're already seeing signs of things we have working towards this. Interesting to see that in the past "Man" was more optimistic :)
> It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make, since it will lead to an “intelligence explosion.”
Well, that didn't happen.
> The first ultraintelligent machine will need to be ultraparallel, and is likely to be achieved with the help of a very large artificial neural net.
Right on, that we have.
> The required high degree of connectivity might be attained with the help of microminiature radio transmitters and receivers.
Hahaha, this is straight out of 60s-70s sci-fi, where their best futuristic interfaces were smaller CRT screens / flashy keys, etc.
> The first ultraintelligent machine will be educated partly by means of positive and negative reinforcement. The task of education will be eased if the machine is somewhat of a robot, sinae the activity of a robot is concrete. [...] the machine will be able to lem from experience, by means of positive and negative reinforcement, and the instruction of the machine will resemble that of a child.
Heh, nice early insights. They missed the how, but RL is the thing that ultimately made it "click" and be useful. And there is increasing talk about embodiment and how that'll help the next iteration of models. So there's that.
Overall a cool read. I skipped most of the middle part, only skimming for things here and there.