Let A be AI notkilleveryoneism people.
Let B be AI capabilities developers/supporters.
Let C be people concerned with regulatory capture and centralization by AI firms.
A and B are disjoint.
A and C have some overlap.
B and C have considerable overlap.
(P1) Current SOTA LLMs are good at understanding implicit context.
(P2) A system must be extremely misaligned in order to cause a catastrophe.
(C) So, it will be easy to sufficiently align future more powerful LLMs. P1: The current trajectory of AI research will lead to superhuman AGI.
P2: Superhuman AGI will be capable of escaping any human efforts to control it.
P3: Superhuman AGI will be misaligned by default, i.e. it will likely adopt values and/or set long-term goals that will lead to extinction-level outcomes, meaning outcomes that are as bad as human extinction.
P4: We do not know how to align superhuman AGI, i.e. reliably imbue it with values or define long-term goals that will ensure it does not ultimately lead to an extinction-level outcome, without some amount of trial & error (how nearly all of scientific research works).
C1: P2 + P3 In the case of superhuman AGI, since it will be able to escape human control and misaligned by default, the only survivable path to alignment cannot involve trial & error because the first failed try will result in an extinction-level outcome.
C2: P4 + C1 This means we will not survive superhuman AGI, because our survival would require alignment, towards which we have no survivable path: the only path we know of involves trial & error, which is not survivable.
C3: P1 + C2 Therefore the current trajectory of AI research which will produce superhuman AGI leads to an outcome where we do not survive. P1: If intelligent system A cannot give a detailed account of how it would be bested by a more intelligent system B, then A will not be bested by B.
P2: Humans (so far) cannot give a detailed account of how a more intelligent AI system would best them.
C: So, humans will not be bested by a more intelligent AI system.
https://wiki.aiimpacts.org/doku.php?id=arguments_for_ai_risk:is_ai_an_existential_threat_to_humanity:will_malign_ai_agents_control_the_future:argument_for_ai_x-risk_from_competent_malign_agents:start
https://arxiv.org/abs/2206.13353
https://aiadventures.net/summaries/agi-ruin-list-of-lethalities.html P1: Humans are close to developing powerful non-LLM AI systems.
P2: Humans are not close to developing techniques for safely using powerful AI systems.
P3: If governments ban AI development, then the speed of AI capabilities development will be significantly reduced.
P4: It is a waste of scarce expertise and political capital to focus on making an LLM carve out in AI regulation legislation.
C: If it is extremely unlikely that LLMs will become powerful in the near future, then I am made much better off if governments ban all AI capabilities research (including LLMs). (1) Many smart people are working on capabilities.
(2) Many investment dollars will flow into AI development in the near future.
(3) Many impressive AI systems have recently been developed: Meta's CICERO, OpenAI's GPT4, DeepMind's AlphaGo.
(4) Hardware will continue to improve.
(5) LLM performance significantly improved as data volume and training time increased.
(6) Humans have built other complex artefacts without good theories of the artefact, including: operating systems, airplanes, beer. Premise 1: Human brains are generally intelligent.
Premise 2: If humans brains are generally intelligent, then software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.
Conclusion: Software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.
> Premise 3 is where the problem is, of course.
I don't believe premise 3 is a problem exactly, but I do believe that it is a non-trivial challenge to determine whether or not it is true.
> We have no idea how to build AGI. We know LLMs won't be it.
> Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence.
> Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from.
> This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered".
It looks to me that you are making the following argument:
I believe that argument is about an important point (chance of AI catastrophe) and that it is a pretty good argument. But the original premise 3 says, "If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.", so AFAICT your argument doesn't substantively address it. (ie, your argument's conclusion doesn't tell me anything about whether or not premise 3 is true)
I apologize if I have misunderstood your point.
> Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI.
We may be using the word "alignment" slightly differently. By "alignment" I just meant getting the algorithmic system to have precisely the goal that its human programmers want it to have. I would call, for example, RLHF a "tool" for trying to achieve alignment.
How do you want to use the terms "alignment" and "alignment tool" going forward in the discussion?
> Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is.
I agree that there are other plausible sources of catastrophe for humans, to name a few others: asteroids, supervolcanoes and population collapse.
I understand you to be making a new point now, but I just want to state that I do not believe the existence of other plausible existential threats to be a rebuttal of premise 3.
> LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop.
I agree that LLMs may plausibly cause significant harm in the short term via disinformation and unemployment.
And again, I understand you to be making a new point, but I just want to state that I do not believe the plausibility of such LLM harms is a rebuttal against premise 3.
> Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.
I'm not sure who you mean by "we" here, so I'm not sure if your claim about them is true or not.