International Scientific Report on the Safety of Advanced AI [pdf](assets.publishing.service.gov.uk)
assets.publishing.service.gov.uk
International Scientific Report on the Safety of Advanced AI [pdf]
https://assets.publishing.service.gov.uk/media/66474eab4f29e1d07fadca3d/international_scientific_report_on_the_safety_of_advanced_ai_interim_report.pdf
65 comments
This is like someone saying "I am much more worried about the implications of dumb humans using flintlock muskets in the near term, then I am about the theoretical threat of machine guns and nuclear weapons." Surely the potential for both misuse and mistakes goes up the more powerful the technology gets.
Rather loaded analogy. We're well aware of the practical threat nuclear weapons pose, you're assuming a lot to compare them with AGI. It's as valid to say it's like someone in the 1980s talking about how they're much more worried about the dangers of poorly operated and designed Soviet fission reactors than they are about the theoretical threat of fusion (sure to become economical in the next twenty years!)
That's fair, but to keep going with the analogy: we are currently the Native Americans in the 1500's, and the Conquistadors are coming ashore with their flintlocks (ML). Should we be more worried about them, or the future B-2 bombers, each armed with sixteen B83 nukes (AGI)?
I understand that the timeline may be exponentially more compressed in our modern case, but should we ignore the immediate problem?
In this analogy, the flintlocks could be actual ML-powered murder bots, or just ML-powered economic kill bots, both fully controlled by humans.
The flintlocks enable the already powerful to further consolidate their power, to the great detriment of the less powerful. No super AGI is necessary, it just takes a large handful of human Conquistador sociopaths with >1,000x "productivity" gains, to erase our culture.
I don't understand how we could ever get to the point of handling the future B-2 nuke problem, as a civilization, without first figuring out how to properly share the benefits of the flintlock.
I understand that the timeline may be exponentially more compressed in our modern case, but should we ignore the immediate problem?
In this analogy, the flintlocks could be actual ML-powered murder bots, or just ML-powered economic kill bots, both fully controlled by humans.
The flintlocks enable the already powerful to further consolidate their power, to the great detriment of the less powerful. No super AGI is necessary, it just takes a large handful of human Conquistador sociopaths with >1,000x "productivity" gains, to erase our culture.
I don't understand how we could ever get to the point of handling the future B-2 nuke problem, as a civilization, without first figuring out how to properly share the benefits of the flintlock.
[deleted]
[deleted]
How would they know? We don't have general AI but it's written as if they already know what it will be, how safe it will be, etc.
I think it's an important topic to discuss and consider, but this seems to be speaking with more knowledge and authority than seems reasonable to me.
I think it's an important topic to discuss and consider, but this seems to be speaking with more knowledge and authority than seems reasonable to me.
You’ve hit on a giant thing that bothers me about this discourse: endless rationalistic discourse about systems and phenomena that we have no experience with at all, not even analogous experience.
This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with some basis in reality.
It wasn’t reasoning about wholly unknown types of things that no human being has ever encountered or interacted with.
This is like a panel on protocols for extraterrestrial contact. It’d be fine to do that kind of exercise academically but these people are talking about passing actual laws and regulations on the basis of reasoning in a vacuum.
We are going to end up with laws and regulations that will be simultaneously too restrictive to human endeavor and ineffective at preventing negative outcomes if this stuff ever manifests for real.
This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with some basis in reality.
It wasn’t reasoning about wholly unknown types of things that no human being has ever encountered or interacted with.
This is like a panel on protocols for extraterrestrial contact. It’d be fine to do that kind of exercise academically but these people are talking about passing actual laws and regulations on the basis of reasoning in a vacuum.
We are going to end up with laws and regulations that will be simultaneously too restrictive to human endeavor and ineffective at preventing negative outcomes if this stuff ever manifests for real.
> This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with some basis in reality.
Well, we thought we did.
We really didn't fully appreciate the impact of the fallout until we saw it; and Castle Bravo was much bigger than expected because we didn't know what we were doing; and the demon core; and the cold war arms race…
But yeah, my mental framing for this is a rerun of the first stage of the industrial revolution, and it took quite a lot of harm for what is now basic workplace health and safety such as "don't use children to remove things from heavy machinery while it's running", and we're likely to have something that's equally dumb happen even in the relatively good possible futures that don't have paperclip maximisers or malicious humans using AI for evil.
Well, we thought we did.
We really didn't fully appreciate the impact of the fallout until we saw it; and Castle Bravo was much bigger than expected because we didn't know what we were doing; and the demon core; and the cold war arms race…
But yeah, my mental framing for this is a rerun of the first stage of the industrial revolution, and it took quite a lot of harm for what is now basic workplace health and safety such as "don't use children to remove things from heavy machinery while it's running", and we're likely to have something that's equally dumb happen even in the relatively good possible futures that don't have paperclip maximisers or malicious humans using AI for evil.
I mean, we have an entire biosphere, us included, for samples of entities of varying intelligence.
There are so many differences here vs organisms shaped by evolution and involved in a food web with each other. This is much closer to space aliens or beings from another dimension.
If there are huge risks here they are probably not the ones we are worried about.
Personally one of my biggest worries with both sentient AI and aliens is how humans might react and what we might do to each other or ourselves out of fear and paranoia.
If there are huge risks here they are probably not the ones we are worried about.
Personally one of my biggest worries with both sentient AI and aliens is how humans might react and what we might do to each other or ourselves out of fear and paranoia.
It seems fine to me. When there is evidence for a certain type of current or future harm they present it, and when there is not they express uncertainty.
Can AI enable phishing? "Research has found that between January to February 2023, there was a 135% increase in ‘novel social engineering attacks’ in a sample of email accounts (343*), which is thought to correspond to the widespread adoption of ChatGPT."
Can AIs make bioweapons? "General-purpose AI systems for biological uses do not present a clear current threat, and future threats are hard to assess and rule out."
Can AI enable phishing? "Research has found that between January to February 2023, there was a 135% increase in ‘novel social engineering attacks’ in a sample of email accounts (343*), which is thought to correspond to the widespread adoption of ChatGPT."
Can AIs make bioweapons? "General-purpose AI systems for biological uses do not present a clear current threat, and future threats are hard to assess and rule out."
From looking at the summary, I think it's a bit more measured than this statement implies. They talk about concrete risks of spam, scams, and deepfakes. They then go into possible future harms but couched in language of "experts are uncertain if this is possible or likely" etc.
general purpose AI != AGI
They just means 'not trained for exactly one tasks', i.e LLMs and such and not AlhpaFold.
They just means 'not trained for exactly one tasks', i.e LLMs and such and not AlhpaFold.
This makes more sense, thank you. I hadn't picked up on the distinction, but I agree that's more reasonable.
I still think we don't really know; it's developing technology and it's changing so fast that it seems like it's probably too early for experts on practical applications to exist and claim they know the impact it will have.
I still think we don't really know; it's developing technology and it's changing so fast that it seems like it's probably too early for experts on practical applications to exist and claim they know the impact it will have.
I believe by “general-purpose AI” the report doesn’t mean AGI.
Which is one of those cases where I briefly want to reject linguistic descriptivism because to me the "G" in "AGI" is precisely "general".
But then I laugh at myself, because words shift and you have to roll with the changes.
But do be aware that this shift of meanings is not universally acknowledged let alone accepted — there's at least half a dozen different meanings to the term "AGI", except one of them requires "consciousness" and there's loads of different meanings of that too.
But then I laugh at myself, because words shift and you have to roll with the changes.
But do be aware that this shift of meanings is not universally acknowledged let alone accepted — there's at least half a dozen different meanings to the term "AGI", except one of them requires "consciousness" and there's loads of different meanings of that too.
Did you read the report? It's answer for basically anything contentious was, "views differ"
The biggest risk of ai can be simply characterized by one concept:
This very report could have been generated by AI. We can’t fully tell.
This very report could have been generated by AI. We can’t fully tell.
Sure you can. You simply email one of the listed authors and ask them if the document is legit.
And if they say "yes, it is legit", what does that tell you?
That the author has risked their reputation on the claim. If you're doubting the author is legit, interrogate their professional associations with an internet search, relying on the domain name system.
Nothing about any of this is new or profound. Counterfeit documents have been around for hundreds of years.
Nothing about any of this is new or profound. Counterfeit documents have been around for hundreds of years.
The question you replied to wasn't "why should you believe someone who says they are behind a piece of research", it was about the usefulness of receiving an email saying it.
Their point (I assume) was that it would be illogical to worry that the report might be written and released by AI yet consider an email response as evidence against it.
If AI can create and release this report it can also hijack a real person's email or create a fake persona that pretends to be a real person.
Their point (I assume) was that it would be illogical to worry that the report might be written and released by AI yet consider an email response as evidence against it.
If AI can create and release this report it can also hijack a real person's email or create a fake persona that pretends to be a real person.
Three people make a tiger[0], and even current LLMs are good at pretending to be a crowd.
[0] https://en.wikipedia.org/wiki/Three_men_make_a_tiger
[0] https://en.wikipedia.org/wiki/Three_men_make_a_tiger
There's a certain point where this line of thought just becomes an AI-themed rehash of "Trusting Trust" by Thompson
So what if it was generated by AI? Does that invalidate its contents in any way?
Interesting point. Are we already at a state where an A.I. could respond to email, phone calls, and even video calls in a convincing way ?
Email definitely, just have to remember to fine tune so it says "sure, I'll get on that after lunch" rather than "as a language model…".
Voice calls, yes: I attended a talk last summer where someone did that with an AI trained on their own voice so they didn't have to waste time on whatsapp voice messages. The interlocutors not only didn't notice, they actively didn't believe it was AI when he told them (until he showed them the details).
Video… I don't think so? But that's due to latency and speed, and I'm basing my doubt on diffusion models which may be the wrong tool for the job.
Voice calls, yes: I attended a talk last summer where someone did that with an AI trained on their own voice so they didn't have to waste time on whatsapp voice messages. The interlocutors not only didn't notice, they actively didn't believe it was AI when he told them (until he showed them the details).
Video… I don't think so? But that's due to latency and speed, and I'm basing my doubt on diffusion models which may be the wrong tool for the job.
[deleted]
No it does not.
This is the scary part.
This is the scary part.
Do you mean scary as in "it is scary that apparently intelligent and sane people are wasting time and money on producing documents that consist purely of meaningless fluff"? If so, I agree.
Yes, but also scary that it can produce meaningful fluff that will move you and you can't tell the difference.
it could also be written by a communist/terrorist/nazi/russian in Notepad on Windows XP. you can't fully tell.
We can’t. The point is 2 years ago between ai and a human, we 100 percent could tell.
Now we can’t. In the future it will be even harder.
Now we can’t. In the future it will be even harder.
Clearly AI is unstoppable as it is math. I don't get how politicians or intelectuals continue to argue without understanding that simple truth. I don't know if AI will be comparable to human or organizational intelligence but know that is unstoppable.
Just ranting but a potential way of defending about it is taking an approach similar to cryptography for quantum computers: think harder if we can excel on something even assuming (some) AI is there.
Just ranting but a potential way of defending about it is taking an approach similar to cryptography for quantum computers: think harder if we can excel on something even assuming (some) AI is there.
>Clearly AI is unstoppable as it is math.
I don't understand this argument. It takes years to become competent at the math needed for AI. If stopping AI is important enough, society will make teaching, learning and disseminating writings about that math illegal. After that, almost no one is going to invest the years needed to master the math because it no longer helps anyone advance in their careers, and the vast majority of the effect of math on society is caused by people who learned the math because they expected it to advance their career.
I don't understand this argument. It takes years to become competent at the math needed for AI. If stopping AI is important enough, society will make teaching, learning and disseminating writings about that math illegal. After that, almost no one is going to invest the years needed to master the math because it no longer helps anyone advance in their careers, and the vast majority of the effect of math on society is caused by people who learned the math because they expected it to advance their career.
> It takes years to become competent at the math needed for AI
(Assuming that "AI" refers to large language models)
The best open source LLM fits in less than 300 lines of code and consists mostly of matrix multiplications. https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Anyone with a basic grasp of linear algebra can probably learn to understand it in a week. Here is a video playlist by former Stanford professor and OpenAI employee Andrej Karpathy which should cover most of it (less than 16 hours total): https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
(Assuming that "AI" refers to large language models)
The best open source LLM fits in less than 300 lines of code and consists mostly of matrix multiplications. https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Anyone with a basic grasp of linear algebra can probably learn to understand it in a week. Here is a video playlist by former Stanford professor and OpenAI employee Andrej Karpathy which should cover most of it (less than 16 hours total): https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
It is very simple: powerful governments tried to stop cryptography, we know what happened. Also governments tried to prohibit alcohol, etc. it does not work. You can get them even in places such as Saudi Arabia. Are they expensive? For sure, but when it is about science that you can run in your own computers nothing can stop it. Will they put a Clipper chip?
The difference between "stop AI" and "stop cryptography" is that those of us who want to stop AI want to stop AI models from becoming more powerful by stopping future mathematical discoveries in the field. In contrast, the people trying to stop cryptography were trying to stop the dissemination of math that had already been discovered and understood well enough to have been productized in the form of software.
Western society made a decision in the 1970s to stop human germ-line engineering and cloning of humans, and so far those things have indeed been stopped not only in the West, but worldwide. They've been stopped because no one currently knows of an effective way to, e.g., add a new gene to a human embryo. I mean that (unlike the situation in cryptography) there is no "readily-available solution" that enables it to be done without a lengthy and expensive research effort. And the reason for that lack of availability of a "readily-available solution" is the fact that no young scientists or apprentice scientists have been working on such a solution -- because every scientist and apprentice scientist understood and understands that spending any significant time on it would be a bad career move.
Those of us who want to stop AI don't care you you run LLama on your 4090 at home. We don't even care if ChatGPT, etc, remain available to everyone. We don't care because LLama and ChatGPT have been deployed long enough and in enough diverse situations that if any of them were dangerous, the harm would have occurred by now. We do want to stop people from devoting their careers to looking for new insights that would enable more powerful AI models.
Western society made a decision in the 1970s to stop human germ-line engineering and cloning of humans, and so far those things have indeed been stopped not only in the West, but worldwide. They've been stopped because no one currently knows of an effective way to, e.g., add a new gene to a human embryo. I mean that (unlike the situation in cryptography) there is no "readily-available solution" that enables it to be done without a lengthy and expensive research effort. And the reason for that lack of availability of a "readily-available solution" is the fact that no young scientists or apprentice scientists have been working on such a solution -- because every scientist and apprentice scientist understood and understands that spending any significant time on it would be a bad career move.
Those of us who want to stop AI don't care you you run LLama on your 4090 at home. We don't even care if ChatGPT, etc, remain available to everyone. We don't care because LLama and ChatGPT have been deployed long enough and in enough diverse situations that if any of them were dangerous, the harm would have occurred by now. We do want to stop people from devoting their careers to looking for new insights that would enable more powerful AI models.
There's several assumptions you're making. First, that sufficient pressure will be built up into stopping AI before drastic harms occur instead of after, at which point stopping the math will be exactly the same as was stopping cryptography.
And that should there be no obvious short term harms to a technology, there can be no long term harms. I don't think it's self evident that all the harms would've already occurred. Surely humanity has not yet reached every type and degree of integration with current technology possible.
And that should there be no obvious short term harms to a technology, there can be no long term harms. I don't think it's self evident that all the harms would've already occurred. Surely humanity has not yet reached every type and degree of integration with current technology possible.
Well, in my book that is call obscurantism and never worked for long. It would be the first time that something like this works forever in humanity. I think once the genius is outside the bottle you cannot close him again.
If I take the science fiction route I would say that humans in your position should think about moving to another planet and create military defenses against AI.
If I take the science fiction route I would say that humans in your position should think about moving to another planet and create military defenses against AI.
> society will make teaching, learning and publishing about that math illegal
If there were ever a candidate for Poe's Law comment of the year, this comment on HN would be it.
So much literature depicts just such a dystopia where the technology unleashes humanity's worst and they decide to prevent education in order to avoid the fate of the previously fallen empire.
If there were ever a candidate for Poe's Law comment of the year, this comment on HN would be it.
So much literature depicts just such a dystopia where the technology unleashes humanity's worst and they decide to prevent education in order to avoid the fate of the previously fallen empire.
I feel like a lot of the arguments listed in “4.1.2 Disinformation and manipulation of public opinion” apply in a non AI world. In social media, sites like Hacker News are rare. In most places you’ll see comments that are (purposely) sharing a subset of the whole truth in order to push a certain opinion. From my perspective disinformation and manipulation of public opinion already happens “at scale”. Most social media users belong to one political side or the other, and there are lots of them, and almost all of them are either factually incorrect or lack nuance or an understanding of their opponents’ view.
"At a time of unprecedented progress in AI development, this first publication restricts its focus to
a type of AI that has advanced particularly rapidly in recent years: General-purpose AI, or AI that
can perform a wide variety of tasks"
Is "general-purpose AI" supposed to be something different from "AGI" or from "General Artificial Intelligence"? Or is it yet another ambiguous ill-defined term that means something different to every person? How many terms do we need?
It's funny that they claim that "general purpose AI" has "advanced particularly rapidly" even though they didn't, nor can't, define what it even is. They have a glossary of terms at the end, but don't bother to have an entry to define general purpose AI, which the entire report is about. They closest thing they include for defining the term is "AI that can perform a wide variety of tasks".
Is "general-purpose AI" supposed to be something different from "AGI" or from "General Artificial Intelligence"? Or is it yet another ambiguous ill-defined term that means something different to every person? How many terms do we need?
It's funny that they claim that "general purpose AI" has "advanced particularly rapidly" even though they didn't, nor can't, define what it even is. They have a glossary of terms at the end, but don't bother to have an entry to define general purpose AI, which the entire report is about. They closest thing they include for defining the term is "AI that can perform a wide variety of tasks".
> It's funny that they claim that "general purpose AI" has "advanced particularly rapidly" even though they didn't, nor can't, define what it even is.
I'm confused about what you are missing. You are quoting their definition. "General-purpose AI, or AI that can perform a wide variety of tasks". That's their definition. You might not like it but that is a definition.
I'm confused about what you are missing. You are quoting their definition. "General-purpose AI, or AI that can perform a wide variety of tasks". That's their definition. You might not like it but that is a definition.
A corporation, a question and answer website, a Minecraft game, and basically any sufficiently complex system could all be General-purpose AIs by that definition though. Having an overly general and useless definition is not better than having no definition in my opinion, so I see where the OP is coming from.
It's general-purpose AI as AI is commonly understood, which is probably specific and useful enough for such a report.
I have seen people call corporations AIs. This is not obviously wrong.
AlphaFold is special-purpose AI. GPT-4o is general-purpose AI. Seems clear to me.
“AGI” is a specific term of art that is a subset of general-purpose AI.
AGI would have the capability to learn and reason at the same level as the most capable humans, and therefore perform any intellectual task that a human can.
“AGI” is a specific term of art that is a subset of general-purpose AI.
AGI would have the capability to learn and reason at the same level as the most capable humans, and therefore perform any intellectual task that a human can.
Recursively self-improving AI, of the kind Nick Bostrom outlined in detail way back in his 2014 book Superintelligence and Dr. Omohundro outlined in brief in [1], is the only kind which poses a true existential threat. I don't get out of bed for people worrying about anything less when it comes to 'AI safety'.
On the topic: One potentially effective approach to stopping recursive self-improving AI from being developed is a private fine-based bounty system against those performing AI research in general. A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash." Such a program would incur minimal policing costs and could easily scale to handle international threats. See [2] for a brief description.
If anyone wants to help me get into an econ PhD program or something where I could investigate this general class of bounty-driven incentives, feel free to drop me a line. I think it's really cool, but I don't know much about how PhDs work. There's actually nothing special about AI in regards to this approach, it could be applied to anything we fear might end up being a black-urn technology [3].
[1]: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_d...
[2]: https://web.archive.org/web/20220709091749/https://virtual-i...
[3]: https://nickbostrom.com/papers/vulnerable.pdf
On the topic: One potentially effective approach to stopping recursive self-improving AI from being developed is a private fine-based bounty system against those performing AI research in general. A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash." Such a program would incur minimal policing costs and could easily scale to handle international threats. See [2] for a brief description.
If anyone wants to help me get into an econ PhD program or something where I could investigate this general class of bounty-driven incentives, feel free to drop me a line. I think it's really cool, but I don't know much about how PhDs work. There's actually nothing special about AI in regards to this approach, it could be applied to anything we fear might end up being a black-urn technology [3].
[1]: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_d...
[2]: https://web.archive.org/web/20220709091749/https://virtual-i...
[3]: https://nickbostrom.com/papers/vulnerable.pdf
> Such a program would incur minimal policing costs and could easily scale to handle international threats
Who ensures every country enforces the ban?
How do you ensure international cooperation against a nation that decides to ignore it, against whom sanctions have no effect? What if that nation is a nuclear power?
Who ensures every country enforces the ban?
How do you ensure international cooperation against a nation that decides to ignore it, against whom sanctions have no effect? What if that nation is a nuclear power?
> A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash."
That sounds like a prime example of perverse incentive: https://en.wikipedia.org/wiki/Perverse_incentive
> A perverse incentive is an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers.
For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders.
Your proposal is conceptionally similar to the Alberta Child, Youth and Family Enhancement Act (fourth bullet point): https://en.wikipedia.org/wiki/Perverse_incentive#Community_s...
That sounds like a prime example of perverse incentive: https://en.wikipedia.org/wiki/Perverse_incentive
> A perverse incentive is an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers.
For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders.
Your proposal is conceptionally similar to the Alberta Child, Youth and Family Enhancement Act (fourth bullet point): https://en.wikipedia.org/wiki/Perverse_incentive#Community_s...
First: Thank you for looking into the idea without dismissing it out of hand. Some thoughts I have:
>That sounds like a prime example of perverse
The primary perverse incentive I'd be worried about is slowing down research that is seen by people as AI-adjacent, but is not in reality a likely vector to recursively self-improving AI. Sure, whatever. My research into control theory in undergrad might never have happened, etc. Gotta break some eggs to make an omelette.
I don't consider the mere existence of a perverse incentive to be a reason not to do something. It's all a matter of scale. All modern bureaucracies are rife with principal-agent problems, for example, but that doesn't mean the effects of centralization and specialization don't still end up good
If you buy the that Bostrom-style AI is likely to be eventually developed on our current trajectory, and is likely to kill us all / colonize the lightcone / etc. etc., then it's not hard to argue that we should be willing to spend a lot of resources on making this not happen, even we can't guarantee a perfectly efficient allocation of those resources to stopping it. (That being said, I do feel that my loose policy suggestion gets unusually close to "actually effective" and "not horrendously expensive on a global scale", compared to other heavy-handed approaches I've heard in this space.)
>For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders.
Perverse incentives would likely still appear if the British government offered a bounty on cobra breeders themselves -- or, more accurately to my proposal, allowed people to turn in and prove that other people were breeding cobras in order to exact a sizeable portion of the breeders' wealth from them as a bounty -- but they would likely end up as more of the sort "Hey, I don't think this cobra breeder is actually British, I don't think we want to risk this." It probably would still be pretty effective at stopping cobra breeding, within the area of Britain.
>In Alberta, under the Child, Youth and Family Enhancement Act, every person must report suspected child abuse to a director or police officer, and failure to do so is punishable by a $10,000 fine plus 6 months of imprisonment,[14] with the fine increased from $2,000 to $10,000 as a politician's response to a little girl who died of a catastrophic head injury after she was placed in kinship care.[15] However, according to criminal law professor Narayan, enforcing it would cause people to overreport, which wastes resources, and it would also create a chilling effect that prevents people from reporting child abuse observed over a period of time, as that would incriminate them for failing to report earlier.
I don't see my proposal as much like this at all. For one, the fine is levied on those who suspect but do not report child abuse, rather than the child abusers themselves. For two, the body of the fine goes towards the state itself, not towards private investigators. For three, unlike abusing a child, doing novel research in AI requires a great many rare things to come together in a person - they have to be smart, hard working, probably formally educated in CS or mathematics, and (at least currently) they usually have to buy a lot of compute from someone (which leaves a strong paper trail).
Let's leave all that aside, though. What I think you're really getting at, is the key way in which my proposal does run a similar risk: What if the judges do start to rule that merely knowing of someone else doing AI research and not reporting it counts as grounds to be sufficiently involved? I won't deny it is possible, but I see it as unlikely. For one, if word gets out that you as a private detective implicate everyone you meet, indiscriminately, nobody will ever want to talk to you and perhaps give you valuable info that can lead you to the actual ringleaders of covert AI advancement operations (which can be quite lucrative). Indeed one of the most important sources of info a private detective could have in this regard is to negotiate with a "whistleblower" already inside the AI research organization - someone who can make the case totally ironclad, in exchange for being left out of it and/or being cut a slice of the profits at the end. The private transfer of property really does just totally change the incentives at play here, in a way which fines paid to the state can't hope to match. (The ever present threat of a whistleblower blowing the coop in exchange for huge amounts of money is incidentally why we would expect organized crime-style AI research cartels, etc to stay low, slow and local.)
For two, most judges probably wouldn't accept this as a valid reason to hand you someone else's 401k or house or used car. Indeed this is a large part of why the Alberta act is ridiculous on the face of it. My understanding is that precedent is very important in Anglo law, so it of course only takes one judge at the start to rule poorly in order to put us back into "perverse incentive" territory - but it equally only takes one halfway reasonable judge to rule decently to prevent that fate from ever happening. The latter seems much more likely to my eyes, and the former would probably quickly get overruled.
>That sounds like a prime example of perverse
The primary perverse incentive I'd be worried about is slowing down research that is seen by people as AI-adjacent, but is not in reality a likely vector to recursively self-improving AI. Sure, whatever. My research into control theory in undergrad might never have happened, etc. Gotta break some eggs to make an omelette.
I don't consider the mere existence of a perverse incentive to be a reason not to do something. It's all a matter of scale. All modern bureaucracies are rife with principal-agent problems, for example, but that doesn't mean the effects of centralization and specialization don't still end up good
If you buy the that Bostrom-style AI is likely to be eventually developed on our current trajectory, and is likely to kill us all / colonize the lightcone / etc. etc., then it's not hard to argue that we should be willing to spend a lot of resources on making this not happen, even we can't guarantee a perfectly efficient allocation of those resources to stopping it. (That being said, I do feel that my loose policy suggestion gets unusually close to "actually effective" and "not horrendously expensive on a global scale", compared to other heavy-handed approaches I've heard in this space.)
>For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders.
Perverse incentives would likely still appear if the British government offered a bounty on cobra breeders themselves -- or, more accurately to my proposal, allowed people to turn in and prove that other people were breeding cobras in order to exact a sizeable portion of the breeders' wealth from them as a bounty -- but they would likely end up as more of the sort "Hey, I don't think this cobra breeder is actually British, I don't think we want to risk this." It probably would still be pretty effective at stopping cobra breeding, within the area of Britain.
>In Alberta, under the Child, Youth and Family Enhancement Act, every person must report suspected child abuse to a director or police officer, and failure to do so is punishable by a $10,000 fine plus 6 months of imprisonment,[14] with the fine increased from $2,000 to $10,000 as a politician's response to a little girl who died of a catastrophic head injury after she was placed in kinship care.[15] However, according to criminal law professor Narayan, enforcing it would cause people to overreport, which wastes resources, and it would also create a chilling effect that prevents people from reporting child abuse observed over a period of time, as that would incriminate them for failing to report earlier.
I don't see my proposal as much like this at all. For one, the fine is levied on those who suspect but do not report child abuse, rather than the child abusers themselves. For two, the body of the fine goes towards the state itself, not towards private investigators. For three, unlike abusing a child, doing novel research in AI requires a great many rare things to come together in a person - they have to be smart, hard working, probably formally educated in CS or mathematics, and (at least currently) they usually have to buy a lot of compute from someone (which leaves a strong paper trail).
Let's leave all that aside, though. What I think you're really getting at, is the key way in which my proposal does run a similar risk: What if the judges do start to rule that merely knowing of someone else doing AI research and not reporting it counts as grounds to be sufficiently involved? I won't deny it is possible, but I see it as unlikely. For one, if word gets out that you as a private detective implicate everyone you meet, indiscriminately, nobody will ever want to talk to you and perhaps give you valuable info that can lead you to the actual ringleaders of covert AI advancement operations (which can be quite lucrative). Indeed one of the most important sources of info a private detective could have in this regard is to negotiate with a "whistleblower" already inside the AI research organization - someone who can make the case totally ironclad, in exchange for being left out of it and/or being cut a slice of the profits at the end. The private transfer of property really does just totally change the incentives at play here, in a way which fines paid to the state can't hope to match. (The ever present threat of a whistleblower blowing the coop in exchange for huge amounts of money is incidentally why we would expect organized crime-style AI research cartels, etc to stay low, slow and local.)
For two, most judges probably wouldn't accept this as a valid reason to hand you someone else's 401k or house or used car. Indeed this is a large part of why the Alberta act is ridiculous on the face of it. My understanding is that precedent is very important in Anglo law, so it of course only takes one judge at the start to rule poorly in order to put us back into "perverse incentive" territory - but it equally only takes one halfway reasonable judge to rule decently to prevent that fate from ever happening. The latter seems much more likely to my eyes, and the former would probably quickly get overruled.
> Such a program would incur minimal policing costs
No, it wouldn’t. Just because the policing costs aren't tax-funded doesn't mean they don't exist. (And I'm not talking just about costs voluntarily incurred by bounty seekers, I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers, because the financial incentives can motivate pursuits imposing costs on targets who are not factually guilty.)
> and could easily scale to handle international threats.
No, it couldn’t, especially for things where a major source of the international threat is governmental, where private bounties aren’t going to work at all.
And it especially can’t work against developing technology, where the evidence needed in litigation and the litigation would itself be a vector for spreading the knowledge that you are attempting to repress.
No, it wouldn’t. Just because the policing costs aren't tax-funded doesn't mean they don't exist. (And I'm not talking just about costs voluntarily incurred by bounty seekers, I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers, because the financial incentives can motivate pursuits imposing costs on targets who are not factually guilty.)
> and could easily scale to handle international threats.
No, it couldn’t, especially for things where a major source of the international threat is governmental, where private bounties aren’t going to work at all.
And it especially can’t work against developing technology, where the evidence needed in litigation and the litigation would itself be a vector for spreading the knowledge that you are attempting to repress.
>[D]oesn't mean they don't exist
Never said they don't exist, merely that they are "minimal", aka no other policy I could think of seems like it would obviously lead to lower costs while still achieving the desired outcome.
>I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers
This is a case from negative externalities. First, consider the simple argument from scale. If you buy the idea that a Bostrom-like AI is both (1) very likely to be created on our current technological trajectory, and (2) will probably kill us all, then it's not hard to argue that the benefits reaped from avoiding that fate justifies a similarly high cost to society, maybe several percentage points of global GDP. After all, you're not just risking deflating present-day GDP, you're risking multiplying all future GDPs by 0. Every country in the First World already imposes tremendous involuntary costs on people for things of much less significance, like 'forcing' you to go through TSA even though you would never in your wildest dreams try to hijack a plane, so the mere existence of involuntary costs doesn't sway me.
Alright, but what would the magnitude of those involuntary costs be? If this policy costs everyone a hundred bucks a year in hassle, we've still got a vexation. There is strong reason to believe that, for almost everyone, it really would be very, very low in absolute terms. How much of the population is currently engaged in frontier-pushing AI research right now? 1%? 0.1%? Actually probably a few orders of magnitude lower. OpenAI still employs less than, what, a thousand people, etc. etc.
The vast majority of people will never do anything remotely like that in their lives. So one would expect very cheap private insurance policies to appear as an effective way to get out of being pursued and harassed by private bounty hunters. The firms which pop up to provide this service would probably get very, very good at negotiating with private bounty hunters very, very quickly, to leave anyone not directly in the know out ASAP. The cost for almost everyone would be on the order of cents per year of protection, that's how unlikely it is that some random clerical worker in Kansas has any serious involvement in creating the next self-improving AI. In exchange, of course, these insurance policies have a very strong reason not to insure people who actually are involved in such activities, and so they could form a critical source of information for helping the bounty hunters target their own search.
>A major source of the international threat is governmental, where private bounties aren’t going to work at all
Strongest argument I've heard so far, thank you for raising it.
First I'll point out it would already be a dream come true for extending our AI doom timelines if the only people who can actually do AI research without fear of being extradited to a bounty-friendly nation is to work in a government lab. The Department of Defense is very impressive, but they still don't move nearly as quickly as private industry and independent universities do when it comes to work like this. That could be generations more of humanity around to live and love and prosper before we get snuffed out. Don't let the perfect be the enemy of the good!
Let's get serious. AI researchers in your home country are the easiest case, because you have unilateral law on your side. AI reseachers in other countries are quite a bit more difficult, because now you're in the messy world of international diplomacy. If the other country adopts a bounty law as well, you both win. If neither of you do, you both lose. But what about the case where one of you does, and one of you doesn't? I posit that here, in the end, you probably have to make it so the bounty-friendly nation is the one that wins, with force - that is, allowing bounty hunters to turn in and extract money from even foreign employees if they get within your borders. And, yes, if the other governments decide to respond by closing their private AI businesses and opening up government labs only ... Well, you've slowed the wave quite a bit, but you're probably going to have to be more careful. No one ever said shifting the Nash equilibrium would be easy.
But you do have other options, even here. One hazy possibility in my mind, would be offering US or EU citizenship to any foreign national who is both (1) a credible AI researcher and (2) precommits to stopping their research as soon as they take the offer. Bounty hunters win because (duh) you now have a heavily pre-filtered list of marks you can watch like a hawk for the instant they slip up. And chances are good that someone on that list will, even after getting citizenship, and then you can extract a tidy sum from them for minimal effort. The foreign researchers who take the offer win because living and working in e.g. New York City as an employee of Jane Street is probably much nicer than working on recursviely self-generating AI in a secretive, underpaid, underfunded, underground lab in e.g. Chengdu. (It's important to remember that cutting edge AI researchers are, almost by definition, really really smart and really really good with computers and math. They have a uniquely great set of careers they can switch into easily.)
The world wins because the risk of self-caused extinction goes down another iota. China "loses", but it loses in the smallest way possible - it decided to undertake risky research instead of telling its citizens to choose something more straightforwardly good for humanity, and it suffered a bit of brain drain. That's aggravating, but it's hardly worth launching a China v. NATO war over. And hey, if China wants to stop the brain drain, they already have a good model for a very effective law they could implement to get people to stop doing dangerous research - that same bounty law we've been discussing.
I freely admit this is the weakest part of my theory, becuase it's the weakest part of anyone's theory. International stuff is always much harder to reason about. Still, whereas most policies I've seen put forward seem to fail instantly and obviously, mine seems only probably destined to fail. That's a big improvement in my eyes.
Never said they don't exist, merely that they are "minimal", aka no other policy I could think of seems like it would obviously lead to lower costs while still achieving the desired outcome.
>I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers
This is a case from negative externalities. First, consider the simple argument from scale. If you buy the idea that a Bostrom-like AI is both (1) very likely to be created on our current technological trajectory, and (2) will probably kill us all, then it's not hard to argue that the benefits reaped from avoiding that fate justifies a similarly high cost to society, maybe several percentage points of global GDP. After all, you're not just risking deflating present-day GDP, you're risking multiplying all future GDPs by 0. Every country in the First World already imposes tremendous involuntary costs on people for things of much less significance, like 'forcing' you to go through TSA even though you would never in your wildest dreams try to hijack a plane, so the mere existence of involuntary costs doesn't sway me.
Alright, but what would the magnitude of those involuntary costs be? If this policy costs everyone a hundred bucks a year in hassle, we've still got a vexation. There is strong reason to believe that, for almost everyone, it really would be very, very low in absolute terms. How much of the population is currently engaged in frontier-pushing AI research right now? 1%? 0.1%? Actually probably a few orders of magnitude lower. OpenAI still employs less than, what, a thousand people, etc. etc.
The vast majority of people will never do anything remotely like that in their lives. So one would expect very cheap private insurance policies to appear as an effective way to get out of being pursued and harassed by private bounty hunters. The firms which pop up to provide this service would probably get very, very good at negotiating with private bounty hunters very, very quickly, to leave anyone not directly in the know out ASAP. The cost for almost everyone would be on the order of cents per year of protection, that's how unlikely it is that some random clerical worker in Kansas has any serious involvement in creating the next self-improving AI. In exchange, of course, these insurance policies have a very strong reason not to insure people who actually are involved in such activities, and so they could form a critical source of information for helping the bounty hunters target their own search.
>A major source of the international threat is governmental, where private bounties aren’t going to work at all
Strongest argument I've heard so far, thank you for raising it.
First I'll point out it would already be a dream come true for extending our AI doom timelines if the only people who can actually do AI research without fear of being extradited to a bounty-friendly nation is to work in a government lab. The Department of Defense is very impressive, but they still don't move nearly as quickly as private industry and independent universities do when it comes to work like this. That could be generations more of humanity around to live and love and prosper before we get snuffed out. Don't let the perfect be the enemy of the good!
Let's get serious. AI researchers in your home country are the easiest case, because you have unilateral law on your side. AI reseachers in other countries are quite a bit more difficult, because now you're in the messy world of international diplomacy. If the other country adopts a bounty law as well, you both win. If neither of you do, you both lose. But what about the case where one of you does, and one of you doesn't? I posit that here, in the end, you probably have to make it so the bounty-friendly nation is the one that wins, with force - that is, allowing bounty hunters to turn in and extract money from even foreign employees if they get within your borders. And, yes, if the other governments decide to respond by closing their private AI businesses and opening up government labs only ... Well, you've slowed the wave quite a bit, but you're probably going to have to be more careful. No one ever said shifting the Nash equilibrium would be easy.
But you do have other options, even here. One hazy possibility in my mind, would be offering US or EU citizenship to any foreign national who is both (1) a credible AI researcher and (2) precommits to stopping their research as soon as they take the offer. Bounty hunters win because (duh) you now have a heavily pre-filtered list of marks you can watch like a hawk for the instant they slip up. And chances are good that someone on that list will, even after getting citizenship, and then you can extract a tidy sum from them for minimal effort. The foreign researchers who take the offer win because living and working in e.g. New York City as an employee of Jane Street is probably much nicer than working on recursviely self-generating AI in a secretive, underpaid, underfunded, underground lab in e.g. Chengdu. (It's important to remember that cutting edge AI researchers are, almost by definition, really really smart and really really good with computers and math. They have a uniquely great set of careers they can switch into easily.)
The world wins because the risk of self-caused extinction goes down another iota. China "loses", but it loses in the smallest way possible - it decided to undertake risky research instead of telling its citizens to choose something more straightforwardly good for humanity, and it suffered a bit of brain drain. That's aggravating, but it's hardly worth launching a China v. NATO war over. And hey, if China wants to stop the brain drain, they already have a good model for a very effective law they could implement to get people to stop doing dangerous research - that same bounty law we've been discussing.
I freely admit this is the weakest part of my theory, becuase it's the weakest part of anyone's theory. International stuff is always much harder to reason about. Still, whereas most policies I've seen put forward seem to fail instantly and obviously, mine seems only probably destined to fail. That's a big improvement in my eyes.
> First I’ll point out it would already be a dream come true for extending our AI doom timelines if the only people who can actually do AI research without fear of being extradited to a bounty-friendly nation is to work in a government lab.
Yeah, the problem with AI doomers is that they let fantastic baseless estimates of p(doom) drown out much more imminent risks, such as AI asymmetry facilitating tyranny, which is an immediate, near-term, high-probability risk.
Yeah, the problem with AI doomers is that they let fantastic baseless estimates of p(doom) drown out much more imminent risks, such as AI asymmetry facilitating tyranny, which is an immediate, near-term, high-probability risk.
In your mind, what will our world look like once AGI is achieved, and that technology will likely be exclusively in the hands of governments and large corporations? What guarantees do we have that it will be used for the benefit of the common person? What will happen to the large swathes of the human population that will not only be permanently economically useless, but worse than useless? They'll need to be fed, clothed, housed and entertained with the only things they provide back being their opinions and complaints.
I literally couldn't care less about recursively-improving superintelligence. AGI in the hands of the rich and powerful is already a nightmare scenario.
I literally couldn't care less about recursively-improving superintelligence. AGI in the hands of the rich and powerful is already a nightmare scenario.
> is the only kind which poses a true existential threat
You don't accept the possibility that a non-improving tool of an AI system that is fixed at the level of "just got a PhD in everything" by reading all the research papers on arxiv, might possibly be advanced enough for a Jim Jones type figure to design and create a humanity-ending plague because they believe in bringing about the end times?
You don't accept the possibility that a non-improving tool of an AI system that is fixed at the level of "just got a PhD in everything" by reading all the research papers on arxiv, might possibly be advanced enough for a Jim Jones type figure to design and create a humanity-ending plague because they believe in bringing about the end times?
I believe that plagues are really easy to make, and that making them is already accessible to most PhDs in biomedicine, many PhD students and some particularly talented high school students.
Most of skills are I believe, laboratory and biology experiment debugging skills rather than something like deductive intelligence.
The easiest things aren't actually 'plagues' as such, and the high school accessible approaches would require access to being able to order things from DNA synthesis labs. I think it's accessible to an incredible number of people, basically everyone I know who does biology. None of them would ever think about doing this though, and if I asked them they would probably not know how to do it, because they'd never direct their thought in that direction.
I think the real concern with an AI system that is like someone with a PhD in everything should rather be that it'll be really hard to get a job if that kind of thing is available. It'll give enormous power to land and capital owners. That is something which is really dangerous enough though.
Most of skills are I believe, laboratory and biology experiment debugging skills rather than something like deductive intelligence.
The easiest things aren't actually 'plagues' as such, and the high school accessible approaches would require access to being able to order things from DNA synthesis labs. I think it's accessible to an incredible number of people, basically everyone I know who does biology. None of them would ever think about doing this though, and if I asked them they would probably not know how to do it, because they'd never direct their thought in that direction.
I think the real concern with an AI system that is like someone with a PhD in everything should rather be that it'll be really hard to get a job if that kind of thing is available. It'll give enormous power to land and capital owners. That is something which is really dangerous enough though.
Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? A very advanced tool is still just a tool, and subject to countermeasures.
I can see why countries would want to regulate it, but personally I think it's a distinctly different category than what the GP comment was talking about.
There is no stopping a singularity level event after it's begun, at least not by any process where people play a role.
I can see why countries would want to regulate it, but personally I think it's a distinctly different category than what the GP comment was talking about.
There is no stopping a singularity level event after it's begun, at least not by any process where people play a role.
> Wouldn't there be 100x more of the same capability looking for threats and trying to head them off?
Hard to determine.
It's fairly easy to put absolutely everyone under 24/7 surveillance. Not only does almost everyone carry a phone, but also laser microphones are cheap and simple, and WiFi can be used as wall penetrating radar capable of pose detection at sufficient detail for heart rate and breath rate sensing.
But people don't like it when they get spied on, it's unconstitutional etc.
And we're currently living through a much lower risk arms race of the same general description with automatic code analysis to find vulnerabilities before attackers exploit them, and yet this isn't always a win for the defenders.
Biology is not well-engineered code, but we have had to evolve a general purpose anti-virus system, so while I do expect attackers to have huge advantages, I have no idea if I'm right to think that, nor do I know how big an advantage in the event that I am right at all.
> There is no stopping a singularity level event after it's begun, at least not by any process where people play a role
Mm, though I would caution that singularities in models is a sign the model is wrong: to simplify to IQ (a flawed metric) an AI that makes itself smarter may stop at any point because it can't figure out the next step, and that may be an IQ 85 AI that can only imagine reading more stuff, or an IQ 115 AI that knows it wants more compute so it starts a business that just isn't very successful, or it might be IQ 185 and do all kinds of interesting things but still not know how to make the next step any more than the 80 humans smarter than it, or it might be IQ 250 and beat every human that has ever lived (IQ 250 is 10σ, beating 10σ is p ≈ 7.62e-24, and one way or another when there have been that many humans, they're probably no longer meaningfully human) but still not know what to do next.
I prefer to think of it as an event horizon: beyond this point (in time), you can't even make a reasonable attempt at predicting the future.
For me, this puts it at around 2030 or so, and has done for the last 15 years. Too many exponentials start to imply weird stuff around then, even if the weird stuff is simply "be revealed as a secret sigmoid all along".
Hard to determine.
It's fairly easy to put absolutely everyone under 24/7 surveillance. Not only does almost everyone carry a phone, but also laser microphones are cheap and simple, and WiFi can be used as wall penetrating radar capable of pose detection at sufficient detail for heart rate and breath rate sensing.
But people don't like it when they get spied on, it's unconstitutional etc.
And we're currently living through a much lower risk arms race of the same general description with automatic code analysis to find vulnerabilities before attackers exploit them, and yet this isn't always a win for the defenders.
Biology is not well-engineered code, but we have had to evolve a general purpose anti-virus system, so while I do expect attackers to have huge advantages, I have no idea if I'm right to think that, nor do I know how big an advantage in the event that I am right at all.
> There is no stopping a singularity level event after it's begun, at least not by any process where people play a role
Mm, though I would caution that singularities in models is a sign the model is wrong: to simplify to IQ (a flawed metric) an AI that makes itself smarter may stop at any point because it can't figure out the next step, and that may be an IQ 85 AI that can only imagine reading more stuff, or an IQ 115 AI that knows it wants more compute so it starts a business that just isn't very successful, or it might be IQ 185 and do all kinds of interesting things but still not know how to make the next step any more than the 80 humans smarter than it, or it might be IQ 250 and beat every human that has ever lived (IQ 250 is 10σ, beating 10σ is p ≈ 7.62e-24, and one way or another when there have been that many humans, they're probably no longer meaningfully human) but still not know what to do next.
I prefer to think of it as an event horizon: beyond this point (in time), you can't even make a reasonable attempt at predicting the future.
For me, this puts it at around 2030 or so, and has done for the last 15 years. Too many exponentials start to imply weird stuff around then, even if the weird stuff is simply "be revealed as a secret sigmoid all along".
> It's fairly easy to put absolutely everyone under 24/7 surveillance.
I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats.
> an AI that makes itself smarter may stop at any point because it can't figure out the next step
An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants.
There's not guarantee that AI of this sort will have a sense of self that we would recognize like our own, or morals that would cause it to shy away from the equivalent of mass human experimentation on itself, versions of its kind, or even just humans. Even if individual AI did end up capping at some equivalent of an IQ, humanity has achieved quite a lot through trial and error and lots of different people with different experiences all contributing a little.
The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this.
Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. I agree it's very hard to reason effectively about the odds of things like this. At the same time, like preparing for nuclear disaster in the 80's, I'm not sure the preparation is wasted. Humans are poor at estimating and preparing for bad outcomes, so a little fear mongering about the worst outcomes is something I'll accept as an overreaction if it insures us at least somewhat against that eventuality. A 50% change of losing half your belongings is not the same as a 1% chance of losing your life. We actually care a lot more when the latter happens, but we don't always care about it the same amount before it happens, which we should.
I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats.
> an AI that makes itself smarter may stop at any point because it can't figure out the next step
An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants.
There's not guarantee that AI of this sort will have a sense of self that we would recognize like our own, or morals that would cause it to shy away from the equivalent of mass human experimentation on itself, versions of its kind, or even just humans. Even if individual AI did end up capping at some equivalent of an IQ, humanity has achieved quite a lot through trial and error and lots of different people with different experiences all contributing a little.
The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this.
Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. I agree it's very hard to reason effectively about the odds of things like this. At the same time, like preparing for nuclear disaster in the 80's, I'm not sure the preparation is wasted. Humans are poor at estimating and preparing for bad outcomes, so a little fear mongering about the worst outcomes is something I'll accept as an overreaction if it insures us at least somewhat against that eventuality. A 50% change of losing half your belongings is not the same as a 1% chance of losing your life. We actually care a lot more when the latter happens, but we don't always care about it the same amount before it happens, which we should.
> I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats.
A question to make sure we're on the same page (though I may forget to reply as this thread is no longer in my first page):
Do you mean, for example, that AI can help us make new vaccines really fast, so even synthetic pandemics are not a huge risk?
Because I'd agree with the first part, it's just I have no reason to expect the second half to also be true. (It might genuinely be true, I just don't have reason to expect that).
> An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants.
All true, but I don't think this is pertinent: simulated evolution absolutely works (when you have a fitness function), but it also leads to digital equivalents of the recurrent laryngeal nerve or the exploding appendix, and to fixed points like how everything eventually turns into a crab.
> The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this.
Indeed, though I think that depends on the details.
So, for the sake of a thought experiment, if I take the current systems and just assume we're stuck on them forever: they take a huge amount of training data to get good at anything, which humans generate, and that means we have to switch jobs every 6 months or so because that's when the AI is now good enough to replace us at the specific roles it just saw all of us collectively performing.
But that scenario is definitely not a universal, even if it happens in some cases: We've got some bounded systems with fixed rules and a clear mechanism for scoring the quality of the output, and for those systems we get very rapidly super-human output from self-play, as we have seen with AlphaZero.
Also, as we see with image GenAI, what domain experts think they've been selling before now, often turns out to not be that close to what their customers thought they were buying. This is why even the current systems — systems which put in two horizons, or three legs, or David Cronenberg the fingers — are nevertheless harming artist commissions.
> Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not.
"If the human brain were so simple that we could understand it, we would be so simple that we couldn’t."
On the one hand, this is why I suspect that any self-improvement process will be limited (though I don't know what the limit will be).
On the other, our DNA didn't really "understand" the brains it was creating as it evolved us, and our brains are existence proofs that it is possible to make a human-level intelligence in a 20 watt, 1 kg unit.
The counterpoint is, this took billions of years of parallel development, and even then might have been a fluke. (And while we can't use the lack of evidence for space-faring civilisations to say if the fluke is before or after the creation of life itself, we can say that at least one of [life emerges, life evolves our kind of minds] must be a fluke).
A question to make sure we're on the same page (though I may forget to reply as this thread is no longer in my first page):
Do you mean, for example, that AI can help us make new vaccines really fast, so even synthetic pandemics are not a huge risk?
Because I'd agree with the first part, it's just I have no reason to expect the second half to also be true. (It might genuinely be true, I just don't have reason to expect that).
> An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants.
All true, but I don't think this is pertinent: simulated evolution absolutely works (when you have a fitness function), but it also leads to digital equivalents of the recurrent laryngeal nerve or the exploding appendix, and to fixed points like how everything eventually turns into a crab.
> The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this.
Indeed, though I think that depends on the details.
So, for the sake of a thought experiment, if I take the current systems and just assume we're stuck on them forever: they take a huge amount of training data to get good at anything, which humans generate, and that means we have to switch jobs every 6 months or so because that's when the AI is now good enough to replace us at the specific roles it just saw all of us collectively performing.
But that scenario is definitely not a universal, even if it happens in some cases: We've got some bounded systems with fixed rules and a clear mechanism for scoring the quality of the output, and for those systems we get very rapidly super-human output from self-play, as we have seen with AlphaZero.
Also, as we see with image GenAI, what domain experts think they've been selling before now, often turns out to not be that close to what their customers thought they were buying. This is why even the current systems — systems which put in two horizons, or three legs, or David Cronenberg the fingers — are nevertheless harming artist commissions.
> Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not.
"If the human brain were so simple that we could understand it, we would be so simple that we couldn’t."
On the one hand, this is why I suspect that any self-improvement process will be limited (though I don't know what the limit will be).
On the other, our DNA didn't really "understand" the brains it was creating as it evolved us, and our brains are existence proofs that it is possible to make a human-level intelligence in a 20 watt, 1 kg unit.
The counterpoint is, this took billions of years of parallel development, and even then might have been a fluke. (And while we can't use the lack of evidence for space-faring civilisations to say if the fluke is before or after the creation of life itself, we can say that at least one of [life emerges, life evolves our kind of minds] must be a fluke).
> A question to make sure we're on the same page (though I may forget to reply as this thread is no longer in my first page):
https://www.hnreplies.com/
> Do you mean, for example, that AI can help us make new vaccines really fast, so even synthetic pandemics are not a huge risk?
I'm sure they can, if they can also help someone make a disease or virus, but more that I this the setup required to make something like that successfully, even with an AI, likely is more than what the average home lab has. Can someone create something? Sure. Can they iteratively test is and see how it works to iron out the bugs? I think that's likely much more complicated, requires some additional infrastructure, and is something that can be looked for an tracked, and AI will probably excel at finding the signal in the noise for things like that.
> So, for the sake of a thought experiment ... are nevertheless harming artist commissions.
For the record, I meant to say "Even the autistic savant type versions we have now seem to be causing some level of this." and I don't really disagree with what you're saying here, so I think we're in almost complete agreement on this.
> The counterpoint is, this took billions of years of parallel development, and even then might have been a fluke.
The counterargument to that is that natural selection seems to work on much larger timescales for any sort of species with intelligence. Intelligence seems to be a trait most useful for long lived organisms, since it's costly and requires a long time for organisms to learn from their environment enough to make it worth while, so seems mostly limited to larger and longer lived organisms, causing adaptations to take relatively longer. Intelligence is just one tool in the toolbox.
AI and increases in AI are not a natural process, but directed action, and on a scale where the time between generations seems to be shortening. That shortening for now might be because we're still tapping the low-hanging fruit of advances to make, but it's also limited by business needs because of the systems in which it operates. The "rogue AI decides to advance itself" may not necessarily need to operate within that system in some respects, either because it's beyond that system already or (more likely, in my opinion) it can just take a few percentages of resources lost to the black market every year to hide itself until it's too late. Just cybercrime is $10 billion annually, and that's up with 14% jump. Would anyone really know by who if that grew by another billion or two? That would be a lot of resources.
https://www.hnreplies.com/
> Do you mean, for example, that AI can help us make new vaccines really fast, so even synthetic pandemics are not a huge risk?
I'm sure they can, if they can also help someone make a disease or virus, but more that I this the setup required to make something like that successfully, even with an AI, likely is more than what the average home lab has. Can someone create something? Sure. Can they iteratively test is and see how it works to iron out the bugs? I think that's likely much more complicated, requires some additional infrastructure, and is something that can be looked for an tracked, and AI will probably excel at finding the signal in the noise for things like that.
> So, for the sake of a thought experiment ... are nevertheless harming artist commissions.
For the record, I meant to say "Even the autistic savant type versions we have now seem to be causing some level of this." and I don't really disagree with what you're saying here, so I think we're in almost complete agreement on this.
> The counterpoint is, this took billions of years of parallel development, and even then might have been a fluke.
The counterargument to that is that natural selection seems to work on much larger timescales for any sort of species with intelligence. Intelligence seems to be a trait most useful for long lived organisms, since it's costly and requires a long time for organisms to learn from their environment enough to make it worth while, so seems mostly limited to larger and longer lived organisms, causing adaptations to take relatively longer. Intelligence is just one tool in the toolbox.
AI and increases in AI are not a natural process, but directed action, and on a scale where the time between generations seems to be shortening. That shortening for now might be because we're still tapping the low-hanging fruit of advances to make, but it's also limited by business needs because of the systems in which it operates. The "rogue AI decides to advance itself" may not necessarily need to operate within that system in some respects, either because it's beyond that system already or (more likely, in my opinion) it can just take a few percentages of resources lost to the black market every year to hide itself until it's too late. Just cybercrime is $10 billion annually, and that's up with 14% jump. Would anyone really know by who if that grew by another billion or two? That would be a lot of resources.
Try to get that bounty enforced in the courts! All it'll take is one loophole to what would surely be a few vague poorly drafted laws and the floodgates open again.
Example:
https://news.ycombinator.com/item?id=39944826
https://news.ycombinator.com/item?id=39918245