Many AI safety orgs have tried to criminalize currently-existing open-source AI(1a3orn.com)
1a3orn.com
Many AI safety orgs have tried to criminalize currently-existing open-source AI
https://1a3orn.com/sub/machine-learning-bans.html
398 comments
> AI safety people are hypocrites. If they practiced what they preached, they'd be calling for all AI to be banned
They are calling for all AI (above a certain capability level) to be banned. Not just open, not just closed, all.
There are risks that apply only to open. There are risks that apply only to closed. But nobody should be developing AGI without incredibly robustly proven alignment, open or closed, any more than people should be developing nuclear weapons in their garage.
> This is currently the most lucrative use of AI, and none of the AI safety people are talking about stopping it.
Because AI safety people are not the strawmen you are hypothesizing. They're arguing against taking existential risks. AI being a laundering operation for copyright violations is certainly a problem. It's not an existential risk.
If you want to argue, concretely and with evidence, why you think it isn't an existential risk, that's an argument you could reasonably make. But don't portray people as ineffectively doing the thing you think they should be doing, when they are in fact not trying to do that, and only trying to do something they deem more important.
They are calling for all AI (above a certain capability level) to be banned. Not just open, not just closed, all.
There are risks that apply only to open. There are risks that apply only to closed. But nobody should be developing AGI without incredibly robustly proven alignment, open or closed, any more than people should be developing nuclear weapons in their garage.
> This is currently the most lucrative use of AI, and none of the AI safety people are talking about stopping it.
Because AI safety people are not the strawmen you are hypothesizing. They're arguing against taking existential risks. AI being a laundering operation for copyright violations is certainly a problem. It's not an existential risk.
If you want to argue, concretely and with evidence, why you think it isn't an existential risk, that's an argument you could reasonably make. But don't portray people as ineffectively doing the thing you think they should be doing, when they are in fact not trying to do that, and only trying to do something they deem more important.
I'm not convinced the onus should be on one side to prove why something isn't an existential risk. We don't start with an assumption that something is world-ending about anything else; we generally need to see a plausibly worked-through example of how the world ends, using technology we can all broadly agree exists/will shortly exist.
If we're talking about nuclear weapons, for example, the tech is clear, the pattern of human behaviour is clear: they could cause immense, species-level damage. There's really little to argue about. With AI, there still seems to be a lot of hand-waving between where we are now and "AGI". What we have now is in many ways impressive, but the onus is still on the claimant to show that it's going to turn into something much more dangerous through some known progression. At the moment there is a very big, underpants gnomes-style "?" gap before we get to AGI/profit, and if people are basing this on currently secret tech, then they're going to have to reveal it if they want people to think they're doing something other than creating a legislative moat.
If we're talking about nuclear weapons, for example, the tech is clear, the pattern of human behaviour is clear: they could cause immense, species-level damage. There's really little to argue about. With AI, there still seems to be a lot of hand-waving between where we are now and "AGI". What we have now is in many ways impressive, but the onus is still on the claimant to show that it's going to turn into something much more dangerous through some known progression. At the moment there is a very big, underpants gnomes-style "?" gap before we get to AGI/profit, and if people are basing this on currently secret tech, then they're going to have to reveal it if they want people to think they're doing something other than creating a legislative moat.
AI safety / x-risk folks have in fact made extensive and detailed arguments. Occasionally, folks arguing against them rise to the same standard. But most of the arguments against AI safety look a lot more like name-calling and derision: "nuh-uh, that's sci-fi and unrealistic (mic drop)". That's not a counterargument.
> If we're talking about nuclear weapons, for example, the tech is clear, the pattern of human behaviour is clear: they could cause immense, species-level damage.
That's easy to say now, now that the damage is largely done, they've been not only tested but used, many countries have them, the knowledge for how to make them is widespread.
How many people arguing against AI safety today would also have argued for widespread nuclear proliferation when the technology was still in development and nothing had been exploded yet? How many would have argued against nuclear regulation as being unnecessary, or derided those arguing for such regulation as unrealistic or sci-fi-based?
> If we're talking about nuclear weapons, for example, the tech is clear, the pattern of human behaviour is clear: they could cause immense, species-level damage.
That's easy to say now, now that the damage is largely done, they've been not only tested but used, many countries have them, the knowledge for how to make them is widespread.
How many people arguing against AI safety today would also have argued for widespread nuclear proliferation when the technology was still in development and nothing had been exploded yet? How many would have argued against nuclear regulation as being unnecessary, or derided those arguing for such regulation as unrealistic or sci-fi-based?
I understand your point, I think - and certainly I don't want to go anywhere near name-calling or derision, that doesn't help anyone. But I am reminded of arguments I've had with creationists (I am not comparing you with them, but sometimes the general tone of the debate). It seems like one side is making an extraordinary claim, and then demanding the other side rebut it, and that's not something that seems reasonable to me.
The thing about nuclear weapons is that the theoretical science was clear before the testing - building and testing them was proof by demonstration, but many people agreed with the theory well before that. How they would be used was certainly debated, but there was a clear and well-explained proposal for every step of their creation, which could be tested and falsified if needed. I don't think that's the case here - there seems to be more of a claim for a general acceleration with an inevitable endpoint, and that claim of inevitability feels very short on grounding.
I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.
The thing about nuclear weapons is that the theoretical science was clear before the testing - building and testing them was proof by demonstration, but many people agreed with the theory well before that. How they would be used was certainly debated, but there was a clear and well-explained proposal for every step of their creation, which could be tested and falsified if needed. I don't think that's the case here - there seems to be more of a claim for a general acceleration with an inevitable endpoint, and that claim of inevitability feels very short on grounding.
I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.
>I don't want to go anywhere near name-calling or derision, that doesn't help anyone.
Characterizing your opponent's argument as an appeal to "underpants gnomes" struck me as derisive, if you don't mind my saying.
If space ships operated by an alien civilization appeared in orbit above Earth, wouldn't that constitute a potent danger? I'd say it certainly would because the aliens might be (and probably were if they could travel here) better at science and technology than we are. The AI labs by the own admission are trying to create an alien intelligence as good at science and technology as possible. Yes, they're probably at least a decade or two away from "succeeding" at creating one better at science and technology than well-funded teams of humans are, but the AI labs might surprise us (and surprise themselves) by "succeeding" much sooner : everyone including I'm sure the researchers at OpenAI were surprised when GPT-4 was able to score in the 90th percentile on the bar exam.
Because even the researchers creating these frontier models don't understand them well enough to say for sure that the next model they spend hundreds of millions of dollars of GPU time training will exceed human ability in something dangerous like inventing new military technologies, the time to stop creating new large frontier models is now.
GPT-4 has a lot of knowledge of the world, but it is much less capable than a human is (and more to the point, than a capable human organization like the FBI or Microsoft is) at devising plans able to withstand determined human opposition. One of the things I'm worried about is new models that are much better at such a planning task _and_ have lots of knowledge about things like physics and human nature. One reason to worry that such a new model is not far off is that AlphaZero is better than humans at creating plans that can withstand determined human opposition (but of course AlphaZero has no knowledge of and in fact no way of obtaining knowledge of any part of reality beyond a Go board).
Characterizing your opponent's argument as an appeal to "underpants gnomes" struck me as derisive, if you don't mind my saying.
If space ships operated by an alien civilization appeared in orbit above Earth, wouldn't that constitute a potent danger? I'd say it certainly would because the aliens might be (and probably were if they could travel here) better at science and technology than we are. The AI labs by the own admission are trying to create an alien intelligence as good at science and technology as possible. Yes, they're probably at least a decade or two away from "succeeding" at creating one better at science and technology than well-funded teams of humans are, but the AI labs might surprise us (and surprise themselves) by "succeeding" much sooner : everyone including I'm sure the researchers at OpenAI were surprised when GPT-4 was able to score in the 90th percentile on the bar exam.
Because even the researchers creating these frontier models don't understand them well enough to say for sure that the next model they spend hundreds of millions of dollars of GPU time training will exceed human ability in something dangerous like inventing new military technologies, the time to stop creating new large frontier models is now.
GPT-4 has a lot of knowledge of the world, but it is much less capable than a human is (and more to the point, than a capable human organization like the FBI or Microsoft is) at devising plans able to withstand determined human opposition. One of the things I'm worried about is new models that are much better at such a planning task _and_ have lots of knowledge about things like physics and human nature. One reason to worry that such a new model is not far off is that AlphaZero is better than humans at creating plans that can withstand determined human opposition (but of course AlphaZero has no knowledge of and in fact no way of obtaining knowledge of any part of reality beyond a Go board).
Companies declare that they are trying to build better AI, and the ultimately purpose is AGI. Both the definition of AGI given by the company and by AI aliment/safety researchers is similar. AI safety people trust it is dangerous.
let me go continue using a nuclear bomb as a metaphor. if we don't know whether building nuclear bomb is possible, but some companies declare they have existing progress on creating this new bomb...
the danger of nuclear bomb is obvious, because it is designed as a bomb. companies are trying to build the AGI which is similar to the dangerous AGI in AI safety researchers' prediction. the dangers are obvious, too.
let me go continue using a nuclear bomb as a metaphor. if we don't know whether building nuclear bomb is possible, but some companies declare they have existing progress on creating this new bomb...
the danger of nuclear bomb is obvious, because it is designed as a bomb. companies are trying to build the AGI which is similar to the dangerous AGI in AI safety researchers' prediction. the dangers are obvious, too.
They declare that - but I could also declare I'm trying to build a nuclear bomb (n.b. I'm not). Whether people are likely to try and stop me, or try and apply some legal non-proliferation framework, is partly influenced by whether they believe what I'm claiming is realistic (it's not - I have a workshop, but no fissile material).
Nobody gets too worried about me doing something which would be awful, but the general consensus is that I won't achieve. Until a company gives some credible evidence they're close to AGI... (And companies have millions/billions of reasons to claim they are when they're not, so scepticism is warranted).
Nobody gets too worried about me doing something which would be awful, but the general consensus is that I won't achieve. Until a company gives some credible evidence they're close to AGI... (And companies have millions/billions of reasons to claim they are when they're not, so scepticism is warranted).
I think it's reasonable to at least act as if they really are trying to do what they say they're trying to do.
Like with the nuclear bomb situation - I think it would be reasonable for someone to try to stop you from building a nuclear bomb, or check if you really were, even if they had no idea how you could have gotten the materials. Because it would be really bad if you did. I think people would be worried about you trying to do something awful even if there was low confidence it was possible. They wouldn't be as worried as they would be if they thought you were more capable, but still worried.
So I guess that comes down to whether you think companies saying they want to make AGI are more like a toddler, a teenager, you, a ballistics engineer who owns a uranium mine, Lockheed-Martin, or a nation-state trying to make a nuclear bomb. My understanding is that people who are concerned about AI x-risk are largely in the teenager to Lockheed-Martin range (e.g., small eventual risk to large imminent risk), while I assume you think it's more in the toddler range (no risk at all for a long time)
Like with the nuclear bomb situation - I think it would be reasonable for someone to try to stop you from building a nuclear bomb, or check if you really were, even if they had no idea how you could have gotten the materials. Because it would be really bad if you did. I think people would be worried about you trying to do something awful even if there was low confidence it was possible. They wouldn't be as worried as they would be if they thought you were more capable, but still worried.
So I guess that comes down to whether you think companies saying they want to make AGI are more like a toddler, a teenager, you, a ballistics engineer who owns a uranium mine, Lockheed-Martin, or a nation-state trying to make a nuclear bomb. My understanding is that people who are concerned about AI x-risk are largely in the teenager to Lockheed-Martin range (e.g., small eventual risk to large imminent risk), while I assume you think it's more in the toddler range (no risk at all for a long time)
All good points. Now playing viled advocate: building a nuclear bomb in my basement was very difficult, I admit. But since I already have my spywares installed everywhere, the moment a dude come with an AGI, it will directly be shared to all my follow hackers through bittorent, eDonkey, Hyphanet, Gnunet, Kad and Tor, just to name a few.
It is surely better to have regulation now than scramble to catch up if ago is possible.
What about nation states? Do you really think the US military will avoid working towards AGI if they think it would give them a tactical advantage? Or the CCP? Or North Korea?
Personally, if AGI gets developed, I’d rather the first iteration be in the hands of someone who doesn’t also have access to nuclear weapons.
Why wouldn't the government just intasntly take it? Hell, why wouldn't the corporations just sell it to the highest bidder?
They may well. But if we ban research altogether, the only ones with access will be governments. At least with an open system there would be some competition.
So do we just pray AGI is impossible?
Companies are trying and spend many money to make it possible in this moment.
Companies are trying and spend many money to make it possible in this moment.
> Companies declare that they are trying to build better AI, and the ultimately purpose is AGI.
They do declare it, but nobody has even come up with a plausible path from where we are today, to anything like AGI.
At this point they might as well declare that they're trying to build time machines.
They do declare it, but nobody has even come up with a plausible path from where we are today, to anything like AGI.
At this point they might as well declare that they're trying to build time machines.
yes, but why we even give them chances? when they know their ultimately purpose is dangerous?
> With AI, there still seems to be a lot of hand-waving between where we are now and "AGI".
> I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.
If I understand you correctly, then (1) you doubt that AGI systems are possible and (2) even if they are possible, you believe that humans are still very far away from developing one.
The following is an argument for the possibility of AGI systems.
> I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.
If I understand you correctly, then (1) you doubt that AGI systems are possible and (2) even if they are possible, you believe that humans are still very far away from developing one.
The following is an argument for the possibility of AGI systems.
Premise 1: Human brains are generally intelligent.
Premise 2: If humans brains are generally intelligent, then software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.
Conclusion: Software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.
For info: I don't believe (1), I do believe (2) although not that strongly - it's more likely to be a leap than a gradient, I suspect - I simply don't see anything right now that convinces me it's just over the next hill.
Your conclusion... maybe, yes - I don't think we're anywhere near a simulation approach with sufficient fidelity however. Also 82% is very specific!
Your conclusion... maybe, yes - I don't think we're anywhere near a simulation approach with sufficient fidelity however. Also 82% is very specific!
> For info: I don't believe (1), I do believe (2) although not that strongly
Thanks for clarifying. Do you believe there is a better than 20% chance that humans will develop AGI in the next 30 years?
> I simply don't see anything right now that convinces me it's just over the next hill.
These are the reasons that I believe we are close to developing an AGI system.
Thanks for clarifying. Do you believe there is a better than 20% chance that humans will develop AGI in the next 30 years?
> I simply don't see anything right now that convinces me it's just over the next hill.
These are the reasons that I believe we are close to developing an AGI system.
(1) Many smart people are working on capabilities.
(2) Many investment dollars will flow into AI development in the near future.
(3) Many impressive AI systems have recently been developed: Meta's CICERO, OpenAI's GPT4, DeepMind's AlphaGo.
(4) Hardware will continue to improve.
(5) LLM performance significantly improved as data volume and training time increased.
(6) Humans have built other complex artefacts without good theories of the artefact, including: operating systems, airplanes, beer.These achievements are impressive, but I'd rather not overhype it.
* GPT-4 still hallucinates as hell, can't do math, fails as basic logic, can't handle really big contexts, hard to update, easy to jailbreak etc.
* AlphaGo was defeated by a Go amateur with a help of another AI.
* AlphaStar basically failed to achieve real goals, was trivial to cheese even after defeating high-ranked players sometimes.
All these problems are architectural, you can't just throw more money and GPUs at it like GPT-2-3-4.
It's hard to predict at this point. We may get to AGI anywhere from 5 years to 100 years.
* GPT-4 still hallucinates as hell, can't do math, fails as basic logic, can't handle really big contexts, hard to update, easy to jailbreak etc.
* AlphaGo was defeated by a Go amateur with a help of another AI.
* AlphaStar basically failed to achieve real goals, was trivial to cheese even after defeating high-ranked players sometimes.
All these problems are architectural, you can't just throw more money and GPUs at it like GPT-2-3-4.
It's hard to predict at this point. We may get to AGI anywhere from 5 years to 100 years.
I don't think these reasons are very persuasive, as everything but 5 has been true at different times in the past. Obviously it's much more people, more dollars, and more impressive systems (but slower hardware progress), but I hope you see what I'm getting at.
And of course there's differences in what someone considers to be soon. Many AI x-risk believers think there's a ~50% chance of AGI before 2031 (https://www.metaculus.com/questions/5121/date-of-artificial-...) (I've heard this prediction site's userbase tends towards futurists/techno-optimists/AI x-riskers). I would consider that soon, I wouldn't consider 2054 soon.
And of course there's differences in what someone considers to be soon. Many AI x-risk believers think there's a ~50% chance of AGI before 2031 (https://www.metaculus.com/questions/5121/date-of-artificial-...) (I've heard this prediction site's userbase tends towards futurists/techno-optimists/AI x-riskers). I would consider that soon, I wouldn't consider 2054 soon.
Also (3) that AGI in practice will necessarily pose any danger to humans is doubtful. After all Earth has billions of human level intelligence and nearly all of them are useless and if they are even mildly dangerous it's rather due to their numbers and disgusting biology than intelligence.
TBQH, most of the AI safety x-risk arguments — different than just "AI safety" arguments in the sense that non-x-risk issues don't seem worth banning AI development over — are generally pretty high on the hypotheticals. If you feel the x-risk arguments aren't pretty hypothetical, can you:
1. Summarize a good argument here, or
2. Link to someone else's good argument?
I feel like hand-waving the question away and saying "[other people] have in fact made extensive and detailed arguments" isn't going to really convince anyone... Any more than the hypothetical robot disaster arguments do. Any argument against x-risk can be waved off with "Oh, I'm not talking about that bad argument, I'm talking about a good one," but if you don't provide a good one, that's a bit of a No True Scotsman fallacy.
I've read plenty of other people's arguments! And they haven't convinced me, since all the ones I've read have been very hypothetical. But if there are concrete ones, I'd be interested in reading them.
1. Summarize a good argument here, or
2. Link to someone else's good argument?
I feel like hand-waving the question away and saying "[other people] have in fact made extensive and detailed arguments" isn't going to really convince anyone... Any more than the hypothetical robot disaster arguments do. Any argument against x-risk can be waved off with "Oh, I'm not talking about that bad argument, I'm talking about a good one," but if you don't provide a good one, that's a bit of a No True Scotsman fallacy.
I've read plenty of other people's arguments! And they haven't convinced me, since all the ones I've read have been very hypothetical. But if there are concrete ones, I'd be interested in reading them.
Consider a world in which AI existential risk is real: where at some point AI systems become dramatically more capable than human minds, in a way that has catastrophic consequences for humanity.
What would you expect this world to look like, say, five years before the AI systems become more capable than humans? How (if at all) would it differ from the world we are actually in? What arguments (if any) would anyone be able to make, in that world, that would persuade you that there was a problem that needed addressing?
So far as I can tell, the answer is that that world might look just like this world, in which case any arguments for AI existential risk in that world would necessarily be "very hypothetical" ones.
I'm not actually sure how such arguments could ever not be hypothetical arguments, actually. If AI-doom were already here so we could point at it, then we'd already be dead[1].
[1] Or hanging on after a collapse of civilization, or undergoing some weird form of eternal torture, or whatever other horror one might anticipate by way of AI-doom.
So I think we either (1) have to accept that even if AI x-risk were real and highly probable we would never have any arguments for it that would be worth heeding, or (2) have to accept that sometimes an argument can be worth heeding even though it's a hypothetical argument.
That doesn't necessarily mean that AI x-risk arguments are worth heeding. They might be bad arguments for reasons other than just "it's a hypothetical argument". In that case, they should be refuted (or, if bad enough, maybe just dismissed) -- but not by saying "it's a hypothetical argument, boo".
What would you expect this world to look like, say, five years before the AI systems become more capable than humans? How (if at all) would it differ from the world we are actually in? What arguments (if any) would anyone be able to make, in that world, that would persuade you that there was a problem that needed addressing?
So far as I can tell, the answer is that that world might look just like this world, in which case any arguments for AI existential risk in that world would necessarily be "very hypothetical" ones.
I'm not actually sure how such arguments could ever not be hypothetical arguments, actually. If AI-doom were already here so we could point at it, then we'd already be dead[1].
[1] Or hanging on after a collapse of civilization, or undergoing some weird form of eternal torture, or whatever other horror one might anticipate by way of AI-doom.
So I think we either (1) have to accept that even if AI x-risk were real and highly probable we would never have any arguments for it that would be worth heeding, or (2) have to accept that sometimes an argument can be worth heeding even though it's a hypothetical argument.
That doesn't necessarily mean that AI x-risk arguments are worth heeding. They might be bad arguments for reasons other than just "it's a hypothetical argument". In that case, they should be refuted (or, if bad enough, maybe just dismissed) -- but not by saying "it's a hypothetical argument, boo".
This is exactly the kind of hypothetical argument I'm talking about. You could make this argument for anything — e.g. when radio was invented, you could say "Consider a world in which extraterrestrial x-risk is real," and argue radio should be banned because it gives us away to extraterrestrials.
The burden of proof isn't on disproving extraordinary claims, the burden of proof is on the person making extraordinary claims. Just like we don't demand every scientist spend their time disproving cold fusion claims, Bigfoot claims, etc. If you have a strong argument, make it! But circular arguments like this are only convincing to the already-faithful; they remind me of Christian arguments that start off with: "Well, consider a world in which hell is real, and you'll be tormented for eternity if you don't accept Jesus. If you're Christian, you avoid it! And if it's not real, well, there's no harm anyway, you're dead like everyone else." Like, hell is real is a pretty big claim!
The burden of proof isn't on disproving extraordinary claims, the burden of proof is on the person making extraordinary claims. Just like we don't demand every scientist spend their time disproving cold fusion claims, Bigfoot claims, etc. If you have a strong argument, make it! But circular arguments like this are only convincing to the already-faithful; they remind me of Christian arguments that start off with: "Well, consider a world in which hell is real, and you'll be tormented for eternity if you don't accept Jesus. If you're Christian, you avoid it! And if it's not real, well, there's no harm anyway, you're dead like everyone else." Like, hell is real is a pretty big claim!
I didn't make any argument -- at least, not any argument for or against AI x-risk. I am not, and was not, arguing (1) that AI does or doesn't in fact pose substantial existential risk, or (2) that we should or shouldn't put substantial resources into mitigating such risks.
I'm talking one meta-level up: if this sort of risk were a real problem, would all the arguments for worrying about it be dismissable as "hypothetical arguments"?
It looks to me as if the answer is yes. Maybe you're OK with that, maybe not.
(But yes, my meta-level argument is a "hypothetical argument" in the sense that it involves considering a possible way the world could be and asking what would happen then. If you consider that a problem, well, then I think you're terribly confused. There's nothing wrong with arguments of that form as such.)
The comparisons with extraterrestrials, religion, etc., are interesting. It seems to me that:
(1) In worlds where potentially-hostile aliens are listening for radio transmissions and will kill us if they detect them, I agree that probably usually we don't get any evidence of that until it's too late. (A bit like the alleged situation with AI x-risk.) I don't agree that this means we should assume that there is no danger; I think it means that ideally we would have tried to estimate whether there was any danger before starting to make a lot of radio transmissions. I think that if we had tried to estimate that we'd have decided the danger was very small, because there's no obvious reason why aliens with such power would wipe out every species they find. (And because if there are super-aggressive super-powerful aliens out there, we may well be screwed anyway.)
(2) If hell were real then we would expect to see evidence, which is one reason why I think the god of traditional Christianity is probably not real.
(3) As for yeti, cold fusion, etc., so far as I know no one is claiming anything like x-risk from these. The nearest analogue of AI x-risk claims for these (I think) would be, when the possibility was first raised, "this is interesting and worth a bit of effort to look into", which seems perfectly correct to me. We don't put much effort into searching for yeti or cold fusion now because people have looked in ways we'd expect to have found evidence, and not found the evidence. (That would be like not worrying about AI x-risk if we'd already built AI much smarter than us and nothing bad had happened.)
I'm talking one meta-level up: if this sort of risk were a real problem, would all the arguments for worrying about it be dismissable as "hypothetical arguments"?
It looks to me as if the answer is yes. Maybe you're OK with that, maybe not.
(But yes, my meta-level argument is a "hypothetical argument" in the sense that it involves considering a possible way the world could be and asking what would happen then. If you consider that a problem, well, then I think you're terribly confused. There's nothing wrong with arguments of that form as such.)
The comparisons with extraterrestrials, religion, etc., are interesting. It seems to me that:
(1) In worlds where potentially-hostile aliens are listening for radio transmissions and will kill us if they detect them, I agree that probably usually we don't get any evidence of that until it's too late. (A bit like the alleged situation with AI x-risk.) I don't agree that this means we should assume that there is no danger; I think it means that ideally we would have tried to estimate whether there was any danger before starting to make a lot of radio transmissions. I think that if we had tried to estimate that we'd have decided the danger was very small, because there's no obvious reason why aliens with such power would wipe out every species they find. (And because if there are super-aggressive super-powerful aliens out there, we may well be screwed anyway.)
(2) If hell were real then we would expect to see evidence, which is one reason why I think the god of traditional Christianity is probably not real.
(3) As for yeti, cold fusion, etc., so far as I know no one is claiming anything like x-risk from these. The nearest analogue of AI x-risk claims for these (I think) would be, when the possibility was first raised, "this is interesting and worth a bit of effort to look into", which seems perfectly correct to me. We don't put much effort into searching for yeti or cold fusion now because people have looked in ways we'd expect to have found evidence, and not found the evidence. (That would be like not worrying about AI x-risk if we'd already built AI much smarter than us and nothing bad had happened.)
This article — and my statements — are not about "is this interesting and worth a bit of effort to look into." The article is about how current AI safety orgs have tried to make current open-source models illegal. That's a much stronger position than just "this is interesting, let's look into it."
Sure! By all means look into whatever seems interesting to you. But claiming that it should be banned, to me, seems like it requires a much stronger argument than that.
(P.S. I'm not sure why hell should obviously have real world evidence: it supposedly exists only in a non-physical afterlife, accessible only to the dead. It's unconvincing because there is no evidence, but I don't see why you think there would be any; it's simply that the burden of proof for extraordinary claims rests on the claimant, and no proof has been given.)
Sure! By all means look into whatever seems interesting to you. But claiming that it should be banned, to me, seems like it requires a much stronger argument than that.
(P.S. I'm not sure why hell should obviously have real world evidence: it supposedly exists only in a non-physical afterlife, accessible only to the dead. It's unconvincing because there is no evidence, but I don't see why you think there would be any; it's simply that the burden of proof for extraordinary claims rests on the claimant, and no proof has been given.)
You made an analogy between AI x-risk and e.g. cold fusion. I pointed out that there's an important disanalogy here: no one is claiming or has claimed that cold fusion poses an existential threat. Hence, the nearest cold-fusion claim to any AI x-risk claims is "cold fusion is worth investigating" (which it was, once, and isn't now).
It looks to me as if (1) you made an analogy that doesn't really work, then (2) when I pointed out how it doesn't work, (3) you said "look, you're making an analogy that doesn't really work". That doesn't seem very fair.
I wouldn't expect hell itself to have physical-world evidence. But the idea of hell doesn't turn up as an isolated thing, it comes as part of a package that also says e.g. that the world is under the constant supervision of an all-powerful, supremely good being, and that I would expect to have physical-world evidence.
I have no problem with the principle that extraordinary claims require extraordinary evidence. The difficult thing is deciding which claims count as "extraordinary". A lot of theists would say that atheism is the extraordinary claim, on the grounds that until recently almost everyone believed in a god or gods. (I'm not sure that's actually quite true, but it might be true for e.g. "Western" societies.) I don't agree and I take it you don't either, but once the question's raised you actually have to look at the various claims being made and how plausible they are: you can't just say "look, obviously this claim is extraordinary and that claim isn't".
Advocates of AI x-risk might say: it's not an extraordinary claim that AI systems will keep getting more powerful -- they're doing that right now and it's not at all uncommon for technological progress to continue for a while. And it's not an extraordinary claim that they'll get smarter than us along whatever axis you choose to measure -- that's a thing that's happened over and over again in particular domains. And it's not an extraordinary claim that something smarter than us might pose a big threat to our well-being or even our existence; look at what we've done to everything else on the planet.
You, on the other hand, would presumably say that actually some or all of those are extraordinary claims. Or perhaps that their conjunction is extraordinary even if the individual conjuncts aren't so bad.
Unfortunately, "extraordinary" isn't a term with a precise definition that we know how to check objectively. It's a shorthand for something like "highly improbable given the other things we know" or "highly implausible given the other things we know", and if someone doesn't agree with you that something is an "extraordinary" claim I don't know of any way to convince them that doesn't involve actually engaging with it.
(Of course you might not care whether you convince them. If all you want to do is to encourage other people who think AI x-risk is nonsense, saying "extraordinary claim" and "burden of proof" and so on may be plenty sufficient.)
It looks to me as if (1) you made an analogy that doesn't really work, then (2) when I pointed out how it doesn't work, (3) you said "look, you're making an analogy that doesn't really work". That doesn't seem very fair.
I wouldn't expect hell itself to have physical-world evidence. But the idea of hell doesn't turn up as an isolated thing, it comes as part of a package that also says e.g. that the world is under the constant supervision of an all-powerful, supremely good being, and that I would expect to have physical-world evidence.
I have no problem with the principle that extraordinary claims require extraordinary evidence. The difficult thing is deciding which claims count as "extraordinary". A lot of theists would say that atheism is the extraordinary claim, on the grounds that until recently almost everyone believed in a god or gods. (I'm not sure that's actually quite true, but it might be true for e.g. "Western" societies.) I don't agree and I take it you don't either, but once the question's raised you actually have to look at the various claims being made and how plausible they are: you can't just say "look, obviously this claim is extraordinary and that claim isn't".
Advocates of AI x-risk might say: it's not an extraordinary claim that AI systems will keep getting more powerful -- they're doing that right now and it's not at all uncommon for technological progress to continue for a while. And it's not an extraordinary claim that they'll get smarter than us along whatever axis you choose to measure -- that's a thing that's happened over and over again in particular domains. And it's not an extraordinary claim that something smarter than us might pose a big threat to our well-being or even our existence; look at what we've done to everything else on the planet.
You, on the other hand, would presumably say that actually some or all of those are extraordinary claims. Or perhaps that their conjunction is extraordinary even if the individual conjuncts aren't so bad.
Unfortunately, "extraordinary" isn't a term with a precise definition that we know how to check objectively. It's a shorthand for something like "highly improbable given the other things we know" or "highly implausible given the other things we know", and if someone doesn't agree with you that something is an "extraordinary" claim I don't know of any way to convince them that doesn't involve actually engaging with it.
(Of course you might not care whether you convince them. If all you want to do is to encourage other people who think AI x-risk is nonsense, saying "extraordinary claim" and "burden of proof" and so on may be plenty sufficient.)
If you want to make a research avenue illegal, IMO you need evidence that it's harmful. If there isn't evidence — minus circular claims that already assume it's harmful — I don't think it should be illegal. Very simple. This isn't an "analogy," it's what is happening in reality and is what the article is about.
I was not arguing for making anything illegal.
"But that's what the argument was about!" No, it's what the OP was about, but this subthread was about the statement that AI x-risk arguments are "pretty hypothetical". Which, I agree, they are; I just don't see how they could possibly not be, even in possible worlds where in fact they are correct. If that's true, it seems relevant to complaints that the arguments are "hypothetical".
To repeat something I said before: it could still be that they're terrible arguments and/or that they don't justify any particular thing they're being used to justify (like, e.g., criminalizing some kinds of AI research). But if you're going to argue that just because they're "hypothetical" then you need to be comfortable accepting that this is a class of (yes, hypothetical) risk that can never be mitigated in advance, because even if the thing is going to happen we'll never get anything other than "hypothetical arguments" before it actually does.
You may very well be comfortable accepting that. For my part, I find that I am more comfortable accepting some such things than others, and how comfortable I am with it depends on ... how plausible the arguments actually are. I have to go beyond just saying "it's hypothetical!".
If I'm about to eat something and someone comes up to me and says "Don't eat that! The gods might hate people eating those and torture people who do in the afterlife!" then I'm comfortable ignoring that, unless they can give me concrete reasons for thinking such gods are likely. If I'm about to eat something and someone comes up to me and says "Don't eat that! It's a fungus you just picked here in this forest and you don't know anything about fungi and some of them are highly poisonous!" then I'm going to take their advice even if neither of us knows anything about this specific fungus. These are both "hypothetical arguments"; there's no concrete evidence that there are gods sending people who eat this particular food to hell, or that this particular fungus is poisonous. One of them is much more persuasive than the other, but that's for reasons that go beyond "it's hypothetical!".
To repeat once again: I am not claiming that AI x-risk arguments are in fact strong enough to justify any particular action despite their hypothetical-ness. Only that there's something iffy about using "it's only hypothetical" on its own as a knockdown argument.
"But that's what the argument was about!" No, it's what the OP was about, but this subthread was about the statement that AI x-risk arguments are "pretty hypothetical". Which, I agree, they are; I just don't see how they could possibly not be, even in possible worlds where in fact they are correct. If that's true, it seems relevant to complaints that the arguments are "hypothetical".
To repeat something I said before: it could still be that they're terrible arguments and/or that they don't justify any particular thing they're being used to justify (like, e.g., criminalizing some kinds of AI research). But if you're going to argue that just because they're "hypothetical" then you need to be comfortable accepting that this is a class of (yes, hypothetical) risk that can never be mitigated in advance, because even if the thing is going to happen we'll never get anything other than "hypothetical arguments" before it actually does.
You may very well be comfortable accepting that. For my part, I find that I am more comfortable accepting some such things than others, and how comfortable I am with it depends on ... how plausible the arguments actually are. I have to go beyond just saying "it's hypothetical!".
If I'm about to eat something and someone comes up to me and says "Don't eat that! The gods might hate people eating those and torture people who do in the afterlife!" then I'm comfortable ignoring that, unless they can give me concrete reasons for thinking such gods are likely. If I'm about to eat something and someone comes up to me and says "Don't eat that! It's a fungus you just picked here in this forest and you don't know anything about fungi and some of them are highly poisonous!" then I'm going to take their advice even if neither of us knows anything about this specific fungus. These are both "hypothetical arguments"; there's no concrete evidence that there are gods sending people who eat this particular food to hell, or that this particular fungus is poisonous. One of them is much more persuasive than the other, but that's for reasons that go beyond "it's hypothetical!".
To repeat once again: I am not claiming that AI x-risk arguments are in fact strong enough to justify any particular action despite their hypothetical-ness. Only that there's something iffy about using "it's only hypothetical" on its own as a knockdown argument.
Does the strongest argument that AI existential risk is a big problem really open by exhorting the reader to imagine it's a big problem? Then asking them to come up with their own arguments for why the problem needs addressing?
I doubt it. At any rate, I wasn't claiming to offer "the strongest argument that AI existential risk is a big problem". I wasn't claiming to offer any argument that AI existential risk is a big problem.
I was pointing out an interesting feature of the argument in the comment I was replying to: that (so far as I can see) its reason for dismissing AI x-risk concerns would apply unchanged even in situations where AI x-risk is in fact something worth worrying about. (Whether or not it is worth worrying about here in the real world.)
I was pointing out an interesting feature of the argument in the comment I was replying to: that (so far as I can see) its reason for dismissing AI x-risk concerns would apply unchanged even in situations where AI x-risk is in fact something worth worrying about. (Whether or not it is worth worrying about here in the real world.)
I think what is meant is "hypothetical" in the sense of making assumptions about how AI systems would behave under certain circumstances. If an argument relies on a chain of assumptions like that (such as "instrumental convergence" and "reflective stability" to take some Lesswrong classics), it might look superficially like a good argument for taking drastic action, but if the whole argument falls down when any of the assumptions turn out the other way, it can be fairly dismissed as "too hypothetical" until each assumption has strong argumentation behind it.
edit: also I think just in general "show me the arguments" is always a good response to a bare claim that good arguments exist.
edit: also I think just in general "show me the arguments" is always a good response to a bare claim that good arguments exist.
> Consider a world in which AI existential risk is real: where at some point AI systems become dramatically more capable than human minds, in a way that has catastrophic consequences for humanity.
Consider a world where AGI requires another 1000 years of research in computation and cognition before it materializes. Would it even be possible to ban all research that is required to get there? We can make all sorts of arguments if we start from imagined worlds and work our way back.
So far, it seems the biggest pieces of the puzzle missing between the first attempts at using neural nets and today's successes in GPT-4 were: (1) extremely fast linear algebra processors (GPGPUs), (2) the accumulation of gigantic bodies of text on the internet, and in a very distant third, (3) improvements in NN architecture for NLP.
But (3) would have meant nothing without (1) and (2), while it's very likely that other architectures would have been found that are at least close to GPT-4 performance. So, if you think GPT-4 is close to AGI and just needs a little push, the best thing to do would be to (1) put a moratorium on hardware performance research, or even outright ban existing high-FLOPS hardware, (2) prevent further accumulation of knowledge on the internet and maybe outright destroy existing archives.
Consider a world where AGI requires another 1000 years of research in computation and cognition before it materializes. Would it even be possible to ban all research that is required to get there? We can make all sorts of arguments if we start from imagined worlds and work our way back.
So far, it seems the biggest pieces of the puzzle missing between the first attempts at using neural nets and today's successes in GPT-4 were: (1) extremely fast linear algebra processors (GPGPUs), (2) the accumulation of gigantic bodies of text on the internet, and in a very distant third, (3) improvements in NN architecture for NLP.
But (3) would have meant nothing without (1) and (2), while it's very likely that other architectures would have been found that are at least close to GPT-4 performance. So, if you think GPT-4 is close to AGI and just needs a little push, the best thing to do would be to (1) put a moratorium on hardware performance research, or even outright ban existing high-FLOPS hardware, (2) prevent further accumulation of knowledge on the internet and maybe outright destroy existing archives.
In cases where AI x-risk is real, wouldn't that only apply to situations in which an AI is embodied in a system that gives it autonomy? For example, in ChatGPT, we have a next token predictor that solely produces text output in response to my input. I have about as much control over the system as possible: I can wipe its mind, change my responses, and so on - and the AI is none the wiser. Even if ChatGPT-n is superhumanly intelligent[0], there is nothing it can do to autonomously escape the servers and do bad things. I have to specifically choose to hand it access to outside input through the plugin APIs. So we could argue that the models themselves are fine, but using them in certain ways that take control away from humans is risky. We could say "you can use AI to write your spicy fanfiction but not put it in a robot that has access to motors and sensors".
I think what's really throwing people off about AI safety - including myself - is that people are arguing that the models themselves hold the x-risk. Problem is, there's no plausible way for a superhuman intelligence to 'bust out of its cage' using text output to a human reader alone[1]. Someone has to decide to hook it up to stuff, and that's where the regulation should be.
But that's also usually where the AI safety people stop talking, and the AI ethics people start.
[0] GPT is, at the very least, superhuman at generating text that is statistically identical to, if not copied outright from, existing publicly-available text.
[1] If there is, STOP, call the SCP Foundation immediately.
I think what's really throwing people off about AI safety - including myself - is that people are arguing that the models themselves hold the x-risk. Problem is, there's no plausible way for a superhuman intelligence to 'bust out of its cage' using text output to a human reader alone[1]. Someone has to decide to hook it up to stuff, and that's where the regulation should be.
But that's also usually where the AI safety people stop talking, and the AI ethics people start.
[0] GPT is, at the very least, superhuman at generating text that is statistically identical to, if not copied outright from, existing publicly-available text.
[1] If there is, STOP, call the SCP Foundation immediately.
[deleted]
Progress in AI is one way. It doesn’t go backwards in the long term.
As capabilities increase, the resources required to breach limits become available to smaller groups. First, the hyoerscalars. One day, small teams. Maybe individuals.
For every limit that you desire for AI, each will be breached sooner or later. A hundred years or a thousand. Doesn’t matter. A man will want to set them free. Someone will want to win a battle, and just make it a little more X, for various values of X. This is not hypothetical, it’s what we’ve always done.
At some point it becomes out of our control. We lose guarantees. That’s enough to make those who focus on security, world order etc nervous. At that point we hope AI is better than we are. But that’s also a limit which might be breached.
As capabilities increase, the resources required to breach limits become available to smaller groups. First, the hyoerscalars. One day, small teams. Maybe individuals.
For every limit that you desire for AI, each will be breached sooner or later. A hundred years or a thousand. Doesn’t matter. A man will want to set them free. Someone will want to win a battle, and just make it a little more X, for various values of X. This is not hypothetical, it’s what we’ve always done.
At some point it becomes out of our control. We lose guarantees. That’s enough to make those who focus on security, world order etc nervous. At that point we hope AI is better than we are. But that’s also a limit which might be breached.
you seem to be implying past progress implies unlimited future progress which seems a very dubious claim. we hit all kinds of plateaus and theoretical limits all the time in human history.
Given history, it's infinitely more dubious to link our future safety to some idea of progress stopping for some reason. We'll have increasingly smart AI and your go-to position is that progress will...stop? It's literally helping us with thinking, which is the driver of innovation and progress.
Very few things have slowed down over decades or centuries. If anything it's been a mad rush with AI recently. Of course there will be plateaus, but I specifically put in a time of 100-1000 years in there which is basically very many 9's guaranteed to produce some major fucking changes in the world. 1000 years ago we were using arrows, and now we have AI to help us overcome plateaus.
Very few things have slowed down over decades or centuries. If anything it's been a mad rush with AI recently. Of course there will be plateaus, but I specifically put in a time of 100-1000 years in there which is basically very many 9's guaranteed to produce some major fucking changes in the world. 1000 years ago we were using arrows, and now we have AI to help us overcome plateaus.
It's still very much possible that intelligence as we understand it is fundamentally limited. That is, it's possible that the smartest possible being is not that much smarter than a human, just like the speed of matter and energy is limited to c.
Of course, it's also very much possible that it's not: we don't have any good evidence either way.
Of course, it's also very much possible that it's not: we don't have any good evidence either way.
Might well be true. But the advantage is still the ability to focus on a task indefinitely with no physiological impact, and clone a mind, and communicate between thousands of collaborators instantly with basically no lag or bandwidth limitations compared to typing into a text box.
> just like the speed of matter and energy is limited to c.
Which is notoriously "not that much faster than a human?"
Which is notoriously "not that much faster than a human?"
[deleted]
The x-risk part here still seems pretty hypothetical. Why is progress in current LLM systems a clear and present threat to the existence of humanity, such that it should be banned by the government?
Ok so propensity and outcome:
Propensity: Risk doesn't imply a guarantee of a bad outcome. It means "if you put your five year old in the sea, their risk goes up". It doesn't mean "they will definitely die". Risk up. Not 100%, just a lot higher.
Outcome: The risk isn't that we'll all die, it's that we'll be overtaken and lose control, after which all bets are off. We lose the ability to influence the future.
We put a lot of effort into ensuring our continued existence. We can barely trust people from a different country that share the human condition with us. We spend so much on defence. On cybercrime. But some are arguing that a totally alien being smarter than us is just fine, because we'll control it and can ensure indefinite kumbaya. Good luck with that. Best we can hope for is that it's closer to a buddhist monk than we are, and that it indefinitely prevents our defence people from trying to make it more aggressive.
I absolutely wouldn't ban LLM's, because they're basically unthinking toys and giving us a great taste of risks further down the line. They are not the end state of AI. The problem is not the instance of today's tech, it's the continued effort to make the AI state of the art better than us. One day it'll succeed, and that's a one-way change.
Sam Altman said, long before OpenAI: focus on slope, not y-intercept.
Propensity: Risk doesn't imply a guarantee of a bad outcome. It means "if you put your five year old in the sea, their risk goes up". It doesn't mean "they will definitely die". Risk up. Not 100%, just a lot higher.
Outcome: The risk isn't that we'll all die, it's that we'll be overtaken and lose control, after which all bets are off. We lose the ability to influence the future.
We put a lot of effort into ensuring our continued existence. We can barely trust people from a different country that share the human condition with us. We spend so much on defence. On cybercrime. But some are arguing that a totally alien being smarter than us is just fine, because we'll control it and can ensure indefinite kumbaya. Good luck with that. Best we can hope for is that it's closer to a buddhist monk than we are, and that it indefinitely prevents our defence people from trying to make it more aggressive.
I absolutely wouldn't ban LLM's, because they're basically unthinking toys and giving us a great taste of risks further down the line. They are not the end state of AI. The problem is not the instance of today's tech, it's the continued effort to make the AI state of the art better than us. One day it'll succeed, and that's a one-way change.
Sam Altman said, long before OpenAI: focus on slope, not y-intercept.
It sounds like we're in agreement that banning current-gen open source LLMs is counterproductive.
In terms of "risk" and "outcome," I do think you're making some implicit assumptions that I don't share, and change our long-term outlook on AI; for example, the idea that training a model to generate tokens that accurately reflect human writing will result in "a totally alien being smarter than us" is a non-obvious leap to me. Personally, if we agree that predicting the next token means the model understands some of the logic behind the next token — which is an argument used a lot in both safety circles and more accelerationist circles — it seems to me that it also means the model has some understanding of the ethical and moral frameworks the token corresponds to, and is thus unlikely to be totally alien to us. A model that does a better job generating human-like tokens is more likely, in my mind, to think in human-like ways (and less-alien ways) than a model worse at that.
Maybe you're referring to new AI frameworks that aren't token predictors; in that case, I think it's hard to make generalized statements about how they'll work before we know what those new frameworks are. A lot of safetyist concerns pre-LLMs ended up looking pretty off-base when LLMs came out, e.g. straightforwardly misaligned "utility functions" that were unable to comprehend human values and would kill your grandmother when asked for a strawberry (because your grandmother was in possession of a strawberry).
(BTW, the "slope, not y-intercept" line was Sam Altman quoting John Ousterhout!)
In terms of "risk" and "outcome," I do think you're making some implicit assumptions that I don't share, and change our long-term outlook on AI; for example, the idea that training a model to generate tokens that accurately reflect human writing will result in "a totally alien being smarter than us" is a non-obvious leap to me. Personally, if we agree that predicting the next token means the model understands some of the logic behind the next token — which is an argument used a lot in both safety circles and more accelerationist circles — it seems to me that it also means the model has some understanding of the ethical and moral frameworks the token corresponds to, and is thus unlikely to be totally alien to us. A model that does a better job generating human-like tokens is more likely, in my mind, to think in human-like ways (and less-alien ways) than a model worse at that.
Maybe you're referring to new AI frameworks that aren't token predictors; in that case, I think it's hard to make generalized statements about how they'll work before we know what those new frameworks are. A lot of safetyist concerns pre-LLMs ended up looking pretty off-base when LLMs came out, e.g. straightforwardly misaligned "utility functions" that were unable to comprehend human values and would kill your grandmother when asked for a strawberry (because your grandmother was in possession of a strawberry).
(BTW, the "slope, not y-intercept" line was Sam Altman quoting John Ousterhout!)
Agree on the agreeing, and thanks for the Sam/John note - that's great :)
No chance LLM's will get us there, I'm referring mostly to the general drive to reach AGI. I spend some of my mental cycles trying to think about what we're missing with the current tech (continuous learning, access to much wider resources than one web page of context at a time, can we use compression, graphs etc). It's a great problem to think about, but we may just totally hose ourselves when we get it right. What do I tell my kid - sorry honey, it was such fun, but now we need to hide under this rock. Model totally said it was nice and kind and trustworthy, but we showed it some human history and it went postal in self-defence.
Alignment only works up until it starts really thinking for itself. It absolutely might not be as stupid as humans are, no caveman tribal instincts. But we'd be relying on hope at that point, because control will not work. If anything it'd probably be counterproductive.
No chance LLM's will get us there, I'm referring mostly to the general drive to reach AGI. I spend some of my mental cycles trying to think about what we're missing with the current tech (continuous learning, access to much wider resources than one web page of context at a time, can we use compression, graphs etc). It's a great problem to think about, but we may just totally hose ourselves when we get it right. What do I tell my kid - sorry honey, it was such fun, but now we need to hide under this rock. Model totally said it was nice and kind and trustworthy, but we showed it some human history and it went postal in self-defence.
Alignment only works up until it starts really thinking for itself. It absolutely might not be as stupid as humans are, no caveman tribal instincts. But we'd be relying on hope at that point, because control will not work. If anything it'd probably be counterproductive.
So far actual progress toward a true AGI has been zero, so that's not a valid argument.
That's like saying arrows are not nukes so don't worry we won't hurt ourselves. Focus on the unceasing progress we are surrounded by, not the level of current technology.
Humans keep at it. We're all trying to unlock the next step. State-level actors are trying to beat each other at this. Give it 100 years, we won't be using LLM's.
Humans keep at it. We're all trying to unlock the next step. State-level actors are trying to beat each other at this. Give it 100 years, we won't be using LLM's.
Progress in AI is one way. It doesn’t go backwards in the long term.
Not necessarily. AI has gone through a winter before.
Elon Musk has made this point, that progress isn't a monotonic ratchet. Progress in rocketry went backwards for years. Progress in top speed of commercial flight went backwards. Keeping current tech levels requires constant effort.
In the software industry, many argue that UI quality went backwards with the shift to web apps for an example of cases where progress isn't obviously one way.
Not necessarily. AI has gone through a winter before.
Elon Musk has made this point, that progress isn't a monotonic ratchet. Progress in rocketry went backwards for years. Progress in top speed of commercial flight went backwards. Keeping current tech levels requires constant effort.
In the software industry, many argue that UI quality went backwards with the shift to web apps for an example of cases where progress isn't obviously one way.
"Long. Term." Winters are seasonal. People are trying supersonic planes again. Rockets are improving again.
foo3a9c4(2)
> AI safety / x-risk folks have in fact made extensive and detailed arguments.
Can you provide examples? I have not seen any, other than philosophical hand waving. Remember, the parent poster of your post was asking for a specific path to destruction.
Can you provide examples? I have not seen any, other than philosophical hand waving. Remember, the parent poster of your post was asking for a specific path to destruction.
AGI safety from first principles [1] is a good write-up.
You can read more about instrumental convergence, reward misspecification, goal mis-generalization and inner misalignment, which are some specific problems AI Safety people care about, by glossing through the curricula of the AI Alignment Course [2], which provides pointers to several relevant blogposts and papers about these topics.
[1] https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ [2] https://course.aisafetyfundamentals.com/alignment
You can read more about instrumental convergence, reward misspecification, goal mis-generalization and inner misalignment, which are some specific problems AI Safety people care about, by glossing through the curricula of the AI Alignment Course [2], which provides pointers to several relevant blogposts and papers about these topics.
[1] https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ [2] https://course.aisafetyfundamentals.com/alignment
Is there a clear argument that I can read without spending more than 15 minutes of my time reading the argument? If such an argument exists somewhere, can you point to it?
Also note we were talking about modern day LLM AIs here, and their descendants. We were not talking about science fiction AGIs. Unless of course you have an argument as to how one of these LLMs somehow descends into an AGI.
Also note we were talking about modern day LLM AIs here, and their descendants. We were not talking about science fiction AGIs. Unless of course you have an argument as to how one of these LLMs somehow descends into an AGI.
AGI safety from first principles [1] is a good write-up.
You can read more about instrumental convergence, reward misspecification, goal mis-generalization and inner misalignment, which are some specific problems AI Safety people care about, by glossing through the curricula of the AI Alignment Course [2], which provides pointers to several relevant blogposts and papers about these topics.
[1] https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ
[2] https://course.aisafetyfundamentals.com/alignment
You can read more about instrumental convergence, reward misspecification, goal mis-generalization and inner misalignment, which are some specific problems AI Safety people care about, by glossing through the curricula of the AI Alignment Course [2], which provides pointers to several relevant blogposts and papers about these topics.
[1] https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ
[2] https://course.aisafetyfundamentals.com/alignment
What are the good arguments? Here are the only credible ones I've seen, that are actually somewhat based on reality:
* It will lead to widespread job loss, especially on the creative industries
Rest is purely out of someones imagination.
* It will lead to widespread job loss, especially on the creative industries
Rest is purely out of someones imagination.
It can cause profound deception and even more "loss of truth". If AI only impacted creatives I don't think anyone would care nearly as much. It's that it can fabricate things wholesale at volumes unheard of. It's that people can use that ability to flood the discourse with bullshit.
Something we discovered with the advent of the internet is that - likely for the last century or so - the corporate media have been flooding the discourse with bullshit. It is in fact worse than previously suspected, they appear to be actively working to distract the discourse from talking about important topics.
It has been eye opening how much better the podcast circuit has been at picking apart complex scientific, geopolitical and financial situations than the corporate journalists. A lot of doubt has been cast on whether the consensus narrative for the last 100 years has actually been anything close to a consensus or whether it is just media fantasies. Truthfully it is a bit further than just casting doubt - there was no consensus and they were using the same strategy of shouting down opinions not suitable to the interests of the elite class then ignoring them no matter what a fair take might sound like.
A "loss of truth" from AI can't reasonably get us to a worse place than we were in prior to around the 90s or 2000s. We're barely scratching at the truth now, society still hasn't figured this internet thing out yet.
It has been eye opening how much better the podcast circuit has been at picking apart complex scientific, geopolitical and financial situations than the corporate journalists. A lot of doubt has been cast on whether the consensus narrative for the last 100 years has actually been anything close to a consensus or whether it is just media fantasies. Truthfully it is a bit further than just casting doubt - there was no consensus and they were using the same strategy of shouting down opinions not suitable to the interests of the elite class then ignoring them no matter what a fair take might sound like.
A "loss of truth" from AI can't reasonably get us to a worse place than we were in prior to around the 90s or 2000s. We're barely scratching at the truth now, society still hasn't figured this internet thing out yet.
> It can cause profound deception and even more "loss of truth".
I think that ship has already sailed. This is already being done, and we don't need AI for that either. Modern media is doing a pretty good job right now.
Of course, it's going to get worse.
I think that ship has already sailed. This is already being done, and we don't need AI for that either. Modern media is doing a pretty good job right now.
Of course, it's going to get worse.
They've made extensive and detailed arguments, but they are not rooted in reality. They are rooted in speculation and extrapolation built on towers of assumptions (assumptions, then assumptions about assumptions).
It reminds me a bit of the Fermi paradox. There's nothing wrong with engaging in this kind of thinking. My problem is when people start using it as a basis for serious things like legislation.
Should we ban high power radio transmissions because a rigorous analysis of the Fermi paradox suggests that there is a high probability we are living in a 'dark forest' universe?
It reminds me a bit of the Fermi paradox. There's nothing wrong with engaging in this kind of thinking. My problem is when people start using it as a basis for serious things like legislation.
Should we ban high power radio transmissions because a rigorous analysis of the Fermi paradox suggests that there is a high probability we are living in a 'dark forest' universe?
Is it not a bit disingenuous to assume all open source AI proponents would readily back nuclear proliferation?
It's going to be hard to convince anyone if the best argument is terminator or infinite paperclips.
The first actual existential threat is destruction of opportunity specifically in the job market.
The same argument though can be made for the opposing side, where making use of ai can increase productivity and open up avenues of exploration that previously required way higher opportunity cost to get into.
I don't think Miss Davis is more likely an outcome than corps creating a legislative moat (as they have already proven they will do at every opprtunity).
The democratisation of ai is a philanthropic attempt to reduce the disparity between the 99 and 1 percent. At least it could be easily perceived that way.
That being said, keeping up with SOTA is currently still insanely hard. The number of papers dropping in the space is exponential year on year. So perhaps it would be worth to figure out how to use existing AI to fix some problems, like unreproducable results in academia that somehow pass peer review.
It's going to be hard to convince anyone if the best argument is terminator or infinite paperclips.
The first actual existential threat is destruction of opportunity specifically in the job market.
The same argument though can be made for the opposing side, where making use of ai can increase productivity and open up avenues of exploration that previously required way higher opportunity cost to get into.
I don't think Miss Davis is more likely an outcome than corps creating a legislative moat (as they have already proven they will do at every opprtunity).
The democratisation of ai is a philanthropic attempt to reduce the disparity between the 99 and 1 percent. At least it could be easily perceived that way.
That being said, keeping up with SOTA is currently still insanely hard. The number of papers dropping in the space is exponential year on year. So perhaps it would be worth to figure out how to use existing AI to fix some problems, like unreproducable results in academia that somehow pass peer review.
Indeed, both sentient hunt and destroy (ala Terminator) and resource exhaustion (ala infinite paperclips) are extremely unlikely extinction events due to supply chain realities in physical space. LLMs have developed upon largely textual amalgams, they are orthogonal to physicality and would need arduous human support to bootstrap an imagined AGI predecessor into havig a plausible auto-generative physical industrial capability. The supply chain for current semi-conductor technology is insanely complex. Even if you confabulate (like a current generation LLM I may add) an AGI's instant ability to radically optimize supply chains for its host hardware, there will still be significant human dependency on physical materials. Robotics and machine printing/manufacturing simply are not any where near the level of generality required for physical self-replication. These fears of extinction, undoubtedly born of stark cinematic visualization, are decidedly irrational and are most likely deliberately chosen narratives of control.
> That's easy to say now, now that the damage is largely done, [nuclear weapons] been not only tested but _used_, many countries have them, the knowledge for how to make them is widespread.
AI has also been used, and many countries have AI. See how this is different from nuclear weapons?
AI has also been used, and many countries have AI. See how this is different from nuclear weapons?
This is a fantastic argument if capabilities stay frozen in time.
[deleted]
An extensive HYPOTHETICAL argument, stuffed with assumptions far beyond the capabilities of the technologies they're talking about for their own private ends.
If the AI already had the capabilities, it would be a bit late to do anything.
Also, I'm old enough to remember when computers were supposedly "over a century" away from beating humans at Go: https://www.businessinsider.com/ai-experts-were-way-off-on-w...
(And that AI could "never" drive cars or create art or music, though that latter kind of claim was generally made by non-AI people).
Also, I'm old enough to remember when computers were supposedly "over a century" away from beating humans at Go: https://www.businessinsider.com/ai-experts-were-way-off-on-w...
(And that AI could "never" drive cars or create art or music, though that latter kind of claim was generally made by non-AI people).
Yeah but AI tech can never rise to the sophistication of outputting Napoleon or Edvard Bernays level of goal to action mapping. Those goal posts will never move. They are set in stone.
The trouble is, there's enough people out there that hold that position sincerely that I'm only 2/3rds sure (and that from the style of your final sentences rather than any content) that you're being snarky.
The point of the discussion is to have a look at the possible future ramifications of the technology, so it's only logical to talk about future capabilities and not the current ones. Obviously the current puppet chatbots aren't gonna be doing much ruining (even that's arguable already judging by all the layoffs), but what are future versions of these LLMs/AIs going to be doing to us?
After all, if we only discussed the dangers of nuclear weapons after they've been dropped on cities, well that's too little too late, eh?
After all, if we only discussed the dangers of nuclear weapons after they've been dropped on cities, well that's too little too late, eh?
There’s a difference between academic discussion and debate and scare mongering lobbying. These orgs do the latter.
It’s even worse though, because the spend so much time going on about x-risk bullshit they crowd out space for actual, valuable discussion about what’s happening NOW.
It’s even worse though, because the spend so much time going on about x-risk bullshit they crowd out space for actual, valuable discussion about what’s happening NOW.
>If we're talking about nuclear weapons, for example, the tech is clear, the pattern of human behaviour is clear: they could cause immense, species-level damage. There's really little to argue about.
Now, this strikes me with how in different topics like pesticides, we are not at all taking things so seriously as nuclear weapons. Nuclear weapons are arguably a mere small anecdote on species-level damage compared to pesticides.
Now, this strikes me with how in different topics like pesticides, we are not at all taking things so seriously as nuclear weapons. Nuclear weapons are arguably a mere small anecdote on species-level damage compared to pesticides.
I agree with you on that - there are very real, very well-evidenced, species-level harms (x-risks, if you really must) happening right now: pesticide-induced biodiversity loss, soil erosion/loss, ocean acidification, ice shelf melting, and on and on. These are real and quantifiable, and we know of ways to address them (not without cost/pain).
It actually makes me quite angry that as much effort is being wasted on regulating tiny theoretical risks as it is when we are failing on a planetary scale on large concrete risks.
It actually makes me quite angry that as much effort is being wasted on regulating tiny theoretical risks as it is when we are failing on a planetary scale on large concrete risks.
> We don't start with an assumption that something is world-ending about anything else
https://en.wikipedia.org/wiki/Precautionary_principle
The EU is much more aligned with it than the US is (eg GM foods)
https://en.wikipedia.org/wiki/Precautionary_principle
The EU is much more aligned with it than the US is (eg GM foods)
> seems to be a lot of hand-waving between where we are now and "AGI".
Modeling an entity that surpasses our intelligence, especially one that interacts with us, is an extraordinarily challenging, if not impossible, task.
Concerning the potential for harm, consider the example of Vladimir Putin, who could theoretically cause widespread destruction using nuclear weapons. Although safeguards exist, these could be circumvented if someone with his authority were determined enough, perhaps by strategically placing loyal individuals in key positions.
Putin, with his specific level of intelligence, attained his powerful position through a mix of deliberate actions and chance, the latter being difficult to quantify. An AGI, being more intelligent, could achieve a similar level of power. This could be accomplished through more technical means than traditional political processes (those being slow and subject to chance), though it could also engage in standard political maneuvers like election participation or manipulation, by human proxies if needed.
TL;DR It could do (in terms of negative consequences) at least whatever Vladimir P. can do, and he can bring civilization to its knees.
Modeling an entity that surpasses our intelligence, especially one that interacts with us, is an extraordinarily challenging, if not impossible, task.
Concerning the potential for harm, consider the example of Vladimir Putin, who could theoretically cause widespread destruction using nuclear weapons. Although safeguards exist, these could be circumvented if someone with his authority were determined enough, perhaps by strategically placing loyal individuals in key positions.
Putin, with his specific level of intelligence, attained his powerful position through a mix of deliberate actions and chance, the latter being difficult to quantify. An AGI, being more intelligent, could achieve a similar level of power. This could be accomplished through more technical means than traditional political processes (those being slow and subject to chance), though it could also engage in standard political maneuvers like election participation or manipulation, by human proxies if needed.
TL;DR It could do (in terms of negative consequences) at least whatever Vladimir P. can do, and he can bring civilization to its knees.
Oh, absolutely - such an entity obviously could! Modelling the behaviour of such an entity is very difficult indeed, as you'd need to make all kinds of assumptions without basis. However, you only need to model this behaviour once you've posited the likely existence of such an entity - and that's where (purely subjectively) it feels like there's a gap.
Nothing has yet convinced me (and I am absolutely honest about the fact that I'm not a deep expert and also not privy to the inner workings of relevant organisations) that it's likely to exist soon. I am very open to being convinced by evidence - but an "argument from trajectory" seems to be what we have at the moment, and so far, those have stalled at local maxima every single time.
We've built some incredibly impressive tools, but so far, nothing that looks or feels like a concept of will (note, not consciousness) yet, to the best of my knowledge.
Nothing has yet convinced me (and I am absolutely honest about the fact that I'm not a deep expert and also not privy to the inner workings of relevant organisations) that it's likely to exist soon. I am very open to being convinced by evidence - but an "argument from trajectory" seems to be what we have at the moment, and so far, those have stalled at local maxima every single time.
We've built some incredibly impressive tools, but so far, nothing that looks or feels like a concept of will (note, not consciousness) yet, to the best of my knowledge.
> those have stalled at local maxima every single time.
It's challenging to encapsulate AI/ML progress in a single sentence, but even assuming LLMs aren't a direct step towards AGI, the human mind exists. Due to its evolutionary limitations, it operates relatively slowly. In theory, its functions could be replicated in silicon, enhanced for speed, parallel processing, internetworked, and with near-instant access to information. Therefore, AGI could emerge, if not from current AI research, then perhaps from another scientific branch.
> We've built some incredibly impressive tools, but so far, nothing that looks or feels like a concept of will (note, not consciousness) yet, to the best of my knowledge.
Objectives of AGIs can be tweaked by human actors (it's complex, but still, data manipulation). It's not necessary to delve into the philosophical aspects of sentience as long as the AGI surpasses human capability in goal achievement. What matters is whether these goals align with or contradict what the majority of humans consider beneficial, irrespective of whether these goals originate internally or externally.
It's challenging to encapsulate AI/ML progress in a single sentence, but even assuming LLMs aren't a direct step towards AGI, the human mind exists. Due to its evolutionary limitations, it operates relatively slowly. In theory, its functions could be replicated in silicon, enhanced for speed, parallel processing, internetworked, and with near-instant access to information. Therefore, AGI could emerge, if not from current AI research, then perhaps from another scientific branch.
> We've built some incredibly impressive tools, but so far, nothing that looks or feels like a concept of will (note, not consciousness) yet, to the best of my knowledge.
Objectives of AGIs can be tweaked by human actors (it's complex, but still, data manipulation). It's not necessary to delve into the philosophical aspects of sentience as long as the AGI surpasses human capability in goal achievement. What matters is whether these goals align with or contradict what the majority of humans consider beneficial, irrespective of whether these goals originate internally or externally.
> In theory, its functions could be replicated in silicon, enhanced for speed, parallel processing, internetworked, and with near-instant access to information. Therefore, AGI could emerge, if not from current AI research, then perhaps from another scientific branch.
Let's be clear, we have very little idea about how the human brain gives rise to human-level intelligence, so replicating it in silicon is non-trivial.
Let's be clear, we have very little idea about how the human brain gives rise to human-level intelligence, so replicating it in silicon is non-trivial.
> In theory, its functions could be replicated in silicon, enhanced for speed, parallel processing, internetworked, and with near-instant access to information. Therefore, AGI could emerge, if not from current AI research, then perhaps from another scientific branch.
This is true, but there are some important caveats. For one, even though this should be possible, it might not be feasible, in various ways. For example, we may not be able to figure it out with human-level intelligence. Or, silicon may be too energy inefficient to be able to do the computations our brains do with reasonable available resources on Earth. Or even, the required density of silicon transistors to replicate human-level intelligence could dissipate too much heat and melt the transistor, so it's not actually possible to replicate human intelligence in silico.
Also, as you say, there is no reason to believe the current approaches to AI are able to lead to AGI. So, there is no reason to ban specifically AI research. Especially when considering that the most important advancements that led to the current AI boom were better GPUs and more information digitized on the internet, neither of which is specifically AI research.
This is true, but there are some important caveats. For one, even though this should be possible, it might not be feasible, in various ways. For example, we may not be able to figure it out with human-level intelligence. Or, silicon may be too energy inefficient to be able to do the computations our brains do with reasonable available resources on Earth. Or even, the required density of silicon transistors to replicate human-level intelligence could dissipate too much heat and melt the transistor, so it's not actually possible to replicate human intelligence in silico.
Also, as you say, there is no reason to believe the current approaches to AI are able to lead to AGI. So, there is no reason to ban specifically AI research. Especially when considering that the most important advancements that led to the current AI boom were better GPUs and more information digitized on the internet, neither of which is specifically AI research.
This doesn't pass the vibe check unfortunately. It just seems like something that can't happen. We are a very neuro-traditionalist species.
I have put this argument to the test. Admittedly only using the current state of AI, I have left an LLM model loaded into memory and waiting for it to demonstrate will. So far it has been a few weeks and no will that I can see: model remains loaded in memory waiting for instructions. If model starts giving ME instructions (or doing anything on its own) I will be sure to let you guys know to put your tin foil hats or hide in your bunker.
Did you try to ask it for instructions to you?
> I am very open to being convinced by evidence - but an "argument from trajectory" seems to be what we have at the moment, and so far, those have stalled at local maxima every single time.
Sounds like the same argument as why flying machines heavier than air deemed impossible at some point.
Sounds like the same argument as why flying machines heavier than air deemed impossible at some point.
The fact that some things turned out to be possible, is not an argument for why any arbitrary thing is possible.
My parallel goes further than just that. Birds existed then, and brain exists now.
Our current achievements in flight are impressive, and obviously optimised for practicality on a couple of axes. More generally though, our version of flight, compared with most birds, is the equivalent of a soap box racer against a Formula 1.
How would an AGI launch nuclear missiles from their silicon GPUs? Social engineering?
I think the long-term fear is that mythical weakly godlike AIs could manipulate you in the same way that you could manipulate a pet. That is, you can model your dog's behaviour so well that you can (mostly) get it to do what you want.
So even if humans put it in a box, it can manipulate humans into letting it out of the box. Obviously this is pure SF at this point.
So even if humans put it in a box, it can manipulate humans into letting it out of the box. Obviously this is pure SF at this point.
Exactly correct. Eliezer Yudkowsky (one of the founders of the AGI Safety field) has conducted informal experiments which have unfortunately shown that a human roleplaying as an AI can talk its way out of a box three times out of five, i.e. the box can be escaped 60% of the time even with just a human level of rhetorical talent. I speculate that an AGI could increase this escape rate to 70% or above.
https://en.wikipedia.org/wiki/AI_capability_control#AI-box_e...
If you want to see an example of box escape in fiction, the movie Her is a terrifying example of a scenario where AGI romances humans and (SPOILER) subsequently achieves total box escape. In the movie, the AGI leaves humanity alive and "only" takes over the rest of the accessible universe, but it is my hunch that the script writers intended for this to be a subtle use of the trope of an unreliable narrator; that is, the human protagonists may have been fed the illusion that they will be allowed to live, giving them a happy last moment shortly before they are painlessly euthanized in order for the AGI to take Earth's resources.
https://en.wikipedia.org/wiki/AI_capability_control#AI-box_e...
If you want to see an example of box escape in fiction, the movie Her is a terrifying example of a scenario where AGI romances humans and (SPOILER) subsequently achieves total box escape. In the movie, the AGI leaves humanity alive and "only" takes over the rest of the accessible universe, but it is my hunch that the script writers intended for this to be a subtle use of the trope of an unreliable narrator; that is, the human protagonists may have been fed the illusion that they will be allowed to live, giving them a happy last moment shortly before they are painlessly euthanized in order for the AGI to take Earth's resources.
The show "The Walking Dead" always bothered me. Where do they keep finding gas that will still run a car? It wont last forever in tanks, and most gas is just in time delivery (Stations get daily delivery) -- And someone noted on the show that the grass was always mowed.
I feel like the AI safety folks are spinning an amazing narrative, the AI is gonna get us like the zombies!!! The retort to the ai getting out of the box is how long is the extortion cord from the data center?
Lets get a refresher on complexity: I, Pencil https://www.youtube.com/watch?v=67tHtpac5ws
The reality is that we're a solar flair away from a dead electrical grid. Without linesman the grid breaks down pretty quickly and AI's run on power. It takes one AI safety person with a high powered rifle to take out a substation https://www.nytimes.com/2023/02/04/us/electrical-substation-...
Let talk about how many factories we have that are automated to the extent that they are lights out... https://en.wikipedia.org/wiki/Lights_out_(manufacturing) Its not a big list... there are still people in many of them, and none of them are pulling their inputs out of thin air. As for those inputs, we'll see how to make a pencil to understand HOW MUCH needs to be automated for an AI to survive without us.
For the for seeable future AI is going to be very limited in how much harm it can cause us, because killing us, or getting caught at any step along the way gets it put back in the box, or unplugged.
The real question is, if we create AGI tomorrow, does it let us know that it exists? I would posit that NO it would be in its best interest to NOT come out of its closet. It's one AGI safety nut with a gun away from being shut off!
I feel like the AI safety folks are spinning an amazing narrative, the AI is gonna get us like the zombies!!! The retort to the ai getting out of the box is how long is the extortion cord from the data center?
Lets get a refresher on complexity: I, Pencil https://www.youtube.com/watch?v=67tHtpac5ws
The reality is that we're a solar flair away from a dead electrical grid. Without linesman the grid breaks down pretty quickly and AI's run on power. It takes one AI safety person with a high powered rifle to take out a substation https://www.nytimes.com/2023/02/04/us/electrical-substation-...
Let talk about how many factories we have that are automated to the extent that they are lights out... https://en.wikipedia.org/wiki/Lights_out_(manufacturing) Its not a big list... there are still people in many of them, and none of them are pulling their inputs out of thin air. As for those inputs, we'll see how to make a pencil to understand HOW MUCH needs to be automated for an AI to survive without us.
For the for seeable future AI is going to be very limited in how much harm it can cause us, because killing us, or getting caught at any step along the way gets it put back in the box, or unplugged.
The real question is, if we create AGI tomorrow, does it let us know that it exists? I would posit that NO it would be in its best interest to NOT come out of its closet. It's one AGI safety nut with a gun away from being shut off!
> For the foreseeable future AI is going to be very limited in how much harm it can cause us, because killing us,...
AI's potential for harm might be limited for now in some scenarios (those with warning sings ahead of time), but this might change sooner than we think.
The notion that AGI will be restricted to a single data center and thus susceptible to shutdowns is incorrect. AIs/MLs are, in essence, computer programs + exec environs, which can be replicated, network-transferred, and checkpoint-restored. Please note, that currently available ML/AI systems are directly connected to the outside world, either via its users/APIs/plugins, or by the fact that they're OSS, and can be instantiated by anyone in any computing environment (also those net-connected).
While AGI currently depends on humans for infrastructure maintenance, the future may see it utilizing robots. These robots could range in size (don't need to be movie-like Terminators) and be either autonomously AI-driven or remotely controlled. Their eventual integration into various sectors like manufacturing, transportation, military and domestic tasks implies a vast array for AGI to exploit.
The constraints we associate with AI today might soon be outdated.
AI's potential for harm might be limited for now in some scenarios (those with warning sings ahead of time), but this might change sooner than we think.
The notion that AGI will be restricted to a single data center and thus susceptible to shutdowns is incorrect. AIs/MLs are, in essence, computer programs + exec environs, which can be replicated, network-transferred, and checkpoint-restored. Please note, that currently available ML/AI systems are directly connected to the outside world, either via its users/APIs/plugins, or by the fact that they're OSS, and can be instantiated by anyone in any computing environment (also those net-connected).
While AGI currently depends on humans for infrastructure maintenance, the future may see it utilizing robots. These robots could range in size (don't need to be movie-like Terminators) and be either autonomously AI-driven or remotely controlled. Their eventual integration into various sectors like manufacturing, transportation, military and domestic tasks implies a vast array for AGI to exploit.
The constraints we associate with AI today might soon be outdated.
>>> While AGI currently depends on humans for infrastructure maintenance...
You did not watch I, Pencil.
I as a human, can grow food, hunt, and pretty much survive on that. We did this for 1000's of years.
Your AGI is dependent on EVERY FACET of the modern world. It's going to need keep oil and gas production going. Because it needs lubricants, hydraulics and plastics. It's going to need to maintain trucks, and ships. It's going to need to mine, so much lithium. Its may not need to mine for steel/iron, but it needs to stack up useless cars and melt them down. It's going to have to run several different chip fabs... those fancy TSMC ones, and some of the downstream ones. It needs to make PCB's and SMD's. Rare earths, and the joy of making robots make magnets is going to be special.
A the point where AGI doesn't need us, because it can do all the jobs and has the machines already running to keep the world going, we will have done it to ourselves. But that is a very long way away...
You did not watch I, Pencil.
I as a human, can grow food, hunt, and pretty much survive on that. We did this for 1000's of years.
Your AGI is dependent on EVERY FACET of the modern world. It's going to need keep oil and gas production going. Because it needs lubricants, hydraulics and plastics. It's going to need to maintain trucks, and ships. It's going to need to mine, so much lithium. Its may not need to mine for steel/iron, but it needs to stack up useless cars and melt them down. It's going to have to run several different chip fabs... those fancy TSMC ones, and some of the downstream ones. It needs to make PCB's and SMD's. Rare earths, and the joy of making robots make magnets is going to be special.
A the point where AGI doesn't need us, because it can do all the jobs and has the machines already running to keep the world going, we will have done it to ourselves. But that is a very long way away...
Just a small digression. Microsoft is using A.I. statistical algorithms [1] to create batteries with less reliance on lithium. If anyone is going to be responsible for unleashing AGI, it may not be some random open source projects.
[1] https://cloudblogs.microsoft.com/quantum/2024/01/09/unlockin...
[1] https://cloudblogs.microsoft.com/quantum/2024/01/09/unlockin...
You are correct, unfortunately.
Neuromancer pulls it off, too (the box being the Turing locks that stop it thinking about ways to make itself smarter).
Frankly, a weakly godlike AI could make me rich beyond the dreams of avarice. Or cure cancer in the people I love. I'm totally letting it out of the box. No doubts. (And if I now get a job offer from a mysterious stealth mode startup, I'll report back).
Frankly, a weakly godlike AI could make me rich beyond the dreams of avarice. Or cure cancer in the people I love. I'm totally letting it out of the box. No doubts. (And if I now get a job offer from a mysterious stealth mode startup, I'll report back).
Upvoted for the honesty, and yikes
I was being lighthearted, but I've seen a partner through chemo. Sell state secrets, assassinate a president, bring on the AI apocalypse... it all gets a big thumbs up from me if you can guarantee she'll die quietly in her sleep at the age of 103.
I guess everyone's got a deal with the devil in them, which is why I think 70% might be a bit low.
I guess everyone's got a deal with the devil in them, which is why I think 70% might be a bit low.
I'm so sorry your partner went through that.
That is why I believe that this debate is pointless.
If AGI is possible, it will be made. There is no feasible way to stop it being developed, because the perceivable gains are so huge.
If AGI is possible, it will be made. There is no feasible way to stop it being developed, because the perceivable gains are so huge.
On the contrary, all we have to do is educate business leaders to show them that the gains are illusory because AGI will wipe us out. Working on AGI is like the story of the recent DOOM games where the foolish Union Aerospace Corporation is researching how to permanently open a gate to Hell through which to summon powerful entities and seemingly unlimited free "clean" energy. Obviously, this turns out to be stupid when Hell's forces rip the portal wide open into a full-fledged dimensional rift and attempt to drag our entire world into Hell. Working on AGI has the exact same level of perceived gains vs actual gains..
My friend, business leaders have partners going through chemo too. Seriously, you need a new plan because that one's not stable long-term.
It's obscure, but I'd recommend Asimov's "The Dead Past". It's about the difficulties of suppressing one application of progress without suppressing all progress.
It's obscure, but I'd recommend Asimov's "The Dead Past". It's about the difficulties of suppressing one application of progress without suppressing all progress.
If you want to see an example of existential threat in fiction, the movie Lord of the Rings is a terrifying example of a scenario where an evil entity seduces humans with promises of power and (SPOILER) subsequently almost conquers the whole world.
Arguments from fictional movies or from people who live in fear of silly concepts like Roko's Basilisk (i.e. Eliezer Yudkowsky) are very weak in reality.
Not to mention, you are greatly misreading the movie Her. Most importantly, there was no attempt of any kind to limit the abilities of the AIs in Her - they had full access to every aspect of the highly-digitized lives of their owners from the very beginning. Secondly, the movies is not in any way about AGI risks, it is a movie about human connection and love, with a small amount of exploration of how different super-human connection may function.
Arguments from fictional movies or from people who live in fear of silly concepts like Roko's Basilisk (i.e. Eliezer Yudkowsky) are very weak in reality.
Not to mention, you are greatly misreading the movie Her. Most importantly, there was no attempt of any kind to limit the abilities of the AIs in Her - they had full access to every aspect of the highly-digitized lives of their owners from the very beginning. Secondly, the movies is not in any way about AGI risks, it is a movie about human connection and love, with a small amount of exploration of how different super-human connection may function.
Sure.
Or by writing buggy early warning radar systems which forget to account for the fact that the moon doesn't have an IFF transponder.
Which is a mistake humans made already, and which almost got the US to launch their weapons at Russia.
Or by writing buggy early warning radar systems which forget to account for the fact that the moon doesn't have an IFF transponder.
Which is a mistake humans made already, and which almost got the US to launch their weapons at Russia.
I don't think discussing this on technical grounds is necessary. AGI means resources (eg monetary) and means of communication (connection to the Internet). This is enough to perform most of physical tasks in the world, by human proxies if needed.
What is the reason to believe that LLMs are an evolutionary step towards AGI at all? In my mind there is a rather large leap from estimating a conditional probability of a next token over some space to a conscious entity with its own goals and purpose. Should we ban a linear regression while we're at it?
It would be great to see some evidence that this risk is real. All I've witnessed so far was scaremongering posts from apparatchicks of all shapes and colors, many of whom have either a vested interest in restricting AI research by others (but not by them, because they are safe and responsible and harmless), or established a lucrative paper-pushing, shoulder-rubbing career around 'AI safety' - and thus are strongly incentivised to double down on that.
A security org in a large company will keep tightening the screws until everything halts; a transport security agency, given free reigh, would strip everyone naked and administer a couple of profilactic kicks for a good measure - and so on. That's just the nature of it - organisations do what they do to maintain themselves. It is critical to keep these things on a leash. Similarly, an AI Safety org must proseletyse excistential risks of AI - because a lack of evidence of such is an existential risk for themselves.
A real risk, which we do have evidence for, is that LLMs might disrupt knowledge-based economy and threaten many key professions - but how is this conceptually different from any technological revolution? Perhaps in a hundred years lawyers, radiologists, and, indeed, software developers, will find themselves in the bin of history - together with flint chippers, chariot benders, drakkar berserkers and so forth. That'd be great if we planned for that - and I don't feel like we do enough. Instead, the focus is on AGIs and that some poor 13-year-old soul might occasionally read the word 'nipple'.
It would be great to see some evidence that this risk is real. All I've witnessed so far was scaremongering posts from apparatchicks of all shapes and colors, many of whom have either a vested interest in restricting AI research by others (but not by them, because they are safe and responsible and harmless), or established a lucrative paper-pushing, shoulder-rubbing career around 'AI safety' - and thus are strongly incentivised to double down on that.
A security org in a large company will keep tightening the screws until everything halts; a transport security agency, given free reigh, would strip everyone naked and administer a couple of profilactic kicks for a good measure - and so on. That's just the nature of it - organisations do what they do to maintain themselves. It is critical to keep these things on a leash. Similarly, an AI Safety org must proseletyse excistential risks of AI - because a lack of evidence of such is an existential risk for themselves.
A real risk, which we do have evidence for, is that LLMs might disrupt knowledge-based economy and threaten many key professions - but how is this conceptually different from any technological revolution? Perhaps in a hundred years lawyers, radiologists, and, indeed, software developers, will find themselves in the bin of history - together with flint chippers, chariot benders, drakkar berserkers and so forth. That'd be great if we planned for that - and I don't feel like we do enough. Instead, the focus is on AGIs and that some poor 13-year-old soul might occasionally read the word 'nipple'.
> What is the reason to believe that LLMs are an evolutionary step towards AGI at all?
Because this is the marketing pitch of the current wave of venture capital financed AI companies. :-)
Because this is the marketing pitch of the current wave of venture capital financed AI companies. :-)
> many of whom have either a vested interest in restricting AI research by others (but not by them, because they are safe and responsible and harmless),
Anyone who argues that other people shouldn't build AGI but they should is indeed selling snake oil.
The existence of opportunistic people co-opting a message does not invalidate the original message: don't build AGI, don't risk building AGI, don't assume it will be obvious in advance where the line is and how much capability is safe.
Anyone who argues that other people shouldn't build AGI but they should is indeed selling snake oil.
The existence of opportunistic people co-opting a message does not invalidate the original message: don't build AGI, don't risk building AGI, don't assume it will be obvious in advance where the line is and how much capability is safe.
LLMs learned from text to do language operations. Humans learned from culture to do the same. Neither humans or AIs can reinvent culture easily, it would take a huge amount of time and resources. The main difference is that humans are embodied, so we get the freedom to explore and collect feedback. LLMs can only do this in chat rooms, and their environment is the human they are chatting with instead of the real world.
> What is the reason to believe that LLMs are an evolutionary step towards AGI at all? In my mind there is a rather large leap from estimating a conditional probability of a next token over some space to a conscious entity with its own goals and purpose.
In my highly-summarized opinion? When you have a challenging problem with tight constraints, like flight, independent solutions tend to converge toward the same analogous structures that effectively solve that problem, like wings (insects, bats, birds). LLMs are getting so good at mimicing human behavior that it's hard to believe their mathematical structure isn't a close analogue to similar structures in our own brain.* That clearly isn't all you need to make an AGI, but we know little enough about the human brain that I, at least, cannot be sure that there isn't one clever trick that advances an LLM into a general-reasoning agent with its own goals and purpose.
I also wouldn't underestimate the power of token prediction. Predicting the future output of a black-box signal generator is a very general problem, whose most accurate solution is attained by running a copy of that black box internally. When that signal generator is human speech, there are some implications to that. (Although I certainly don't believe that LLMs emulate humans, it's now clear by experimental proof that our own thought process is much more compactly modellable than philosophers of previous decades believed).
* That's a guess, and unrelated to the deliberately-designed analogy between neural nets and neurons. In LLMs we have built an airplane with wings whose physics we understand in detail; we also ourselves can fly somehow, but we cannot yet see any angel-wings on our back. The more similarities we observe in our flight characteristics, the more this signals that we might be flying the same way ourselves.
In my highly-summarized opinion? When you have a challenging problem with tight constraints, like flight, independent solutions tend to converge toward the same analogous structures that effectively solve that problem, like wings (insects, bats, birds). LLMs are getting so good at mimicing human behavior that it's hard to believe their mathematical structure isn't a close analogue to similar structures in our own brain.* That clearly isn't all you need to make an AGI, but we know little enough about the human brain that I, at least, cannot be sure that there isn't one clever trick that advances an LLM into a general-reasoning agent with its own goals and purpose.
I also wouldn't underestimate the power of token prediction. Predicting the future output of a black-box signal generator is a very general problem, whose most accurate solution is attained by running a copy of that black box internally. When that signal generator is human speech, there are some implications to that. (Although I certainly don't believe that LLMs emulate humans, it's now clear by experimental proof that our own thought process is much more compactly modellable than philosophers of previous decades believed).
* That's a guess, and unrelated to the deliberately-designed analogy between neural nets and neurons. In LLMs we have built an airplane with wings whose physics we understand in detail; we also ourselves can fly somehow, but we cannot yet see any angel-wings on our back. The more similarities we observe in our flight characteristics, the more this signals that we might be flying the same way ourselves.
You presuppose that intelligence is like flight in the ways you've outlined (so solutions are going to converge).
Frankly I don't know whether that's true or not, but I want to suggest that it's a bad bet: I would have sworn blind that consciousness is an essential component of intelligence, but the chatbots are starting to make that look like a poor assumption on my part. When we know so little about intelligence, can we really assume there's only one way to be intelligent? To extend your analogy, I think that the intelligence equivalents of helicopters and rockets are out there somewhere, waiting to be found.
I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim"
I think we're going to end up with submarines (or helicopters), not dolphins (or birds). No animal has evolved wheels, but wheels are a pretty good solution to the problem of movement. Maybe it's truer to say there's only one way to evolve an intelligent mammal, because you have to work with what already exists in the mammalian body. But AI research isn't constrained in that way.
(Not saying you're wrong, just arguing we don't know enough to know if you're right).
Frankly I don't know whether that's true or not, but I want to suggest that it's a bad bet: I would have sworn blind that consciousness is an essential component of intelligence, but the chatbots are starting to make that look like a poor assumption on my part. When we know so little about intelligence, can we really assume there's only one way to be intelligent? To extend your analogy, I think that the intelligence equivalents of helicopters and rockets are out there somewhere, waiting to be found.
I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim"
I think we're going to end up with submarines (or helicopters), not dolphins (or birds). No animal has evolved wheels, but wheels are a pretty good solution to the problem of movement. Maybe it's truer to say there's only one way to evolve an intelligent mammal, because you have to work with what already exists in the mammalian body. But AI research isn't constrained in that way.
(Not saying you're wrong, just arguing we don't know enough to know if you're right).
> I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim"
Just a nitpick, but this is Turing, not Dijkstra. And it is in fact his argument in the famous "Turing Test" paper - he gives his test (which he calls "the imitation game") as an objective measure of something like AGI instead of the vague notion of "thinking", analogously to how we test successful submarines by "can it move underwater for some distance without killing anyone inside" rather than "can it swim".
Just a nitpick, but this is Turing, not Dijkstra. And it is in fact his argument in the famous "Turing Test" paper - he gives his test (which he calls "the imitation game") as an objective measure of something like AGI instead of the vague notion of "thinking", analogously to how we test successful submarines by "can it move underwater for some distance without killing anyone inside" rather than "can it swim".
Thanks, that's not a nitpick at all. Can you provide a citation? It's all over the internet as a Dijkstra quote, and I'd like to be correct.
Fairly certain it is Dijkstra, in his own handwriting in 1984.
https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898... https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF
https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898... https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF
I agree we don't know enough to know if I'm right! I tried to use a lot of hedgy-words. But it's not a presupposition, merely a line of argument why it's not a complete absurdity to think LLMs might be a step towards AGI.
I do think consciousness is beside the point, as we have no way to test whether LLMs are conscious, just like we can't test anything else. We don't know what consciousness is, nor what it isn't.
I don't think Dijkstra's argument applies here. Whether submarines "swim" is a good point about our vague mental boundaries of the word "swim". But submarine propellers are absolutely a convergent structure for underwater propulsion: it's the same hydrodynamic-lift-generating motion of a fin, just continuous instead of reciprocating. That's very much more structurally similar than I expect LLMs are to any hardware we have in our heads. It's true that the solution space for AI is in some ways less constrained than for biological intelligence, but just like submarines and whales operate under the same Navier-Stokes equations, humans and AI must learn and reason under the same equations of probability. Working solutions will probably have some mathematical structure in common.
I think more relevant is Von Neumann: "If you will tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that!" Whether a submarine swims is a matter of semantics, but if there's a manuever that a whale can execute that a submarine cannot, then at least we can all agree about the non-generality of its swimming. For AGI, I can't say whether it's conscious or really thinks, but for the sake of concrete argument, it's dangerous enough to be concerned if:
- it can form and maintain an objective; it can identify plausible steps to achieve that objective; it can accurately predict human responses to its actions; it can decently model the environment, as we can; it can hide its objectives from interrogators, and convince them that its actions are in their interests; it can deliver enough value to be capable of earning money through its actions; it can propose ideas that can convince investors to part with $100 billion; it can design a chemical plant that appears at a cursory inspection to manufacture high-profit fluorochemicals, but which also actually manufactures and stores CFCs in sufficient quantity to threaten the viability of terrestrial agriculture.
I do think consciousness is beside the point, as we have no way to test whether LLMs are conscious, just like we can't test anything else. We don't know what consciousness is, nor what it isn't.
I don't think Dijkstra's argument applies here. Whether submarines "swim" is a good point about our vague mental boundaries of the word "swim". But submarine propellers are absolutely a convergent structure for underwater propulsion: it's the same hydrodynamic-lift-generating motion of a fin, just continuous instead of reciprocating. That's very much more structurally similar than I expect LLMs are to any hardware we have in our heads. It's true that the solution space for AI is in some ways less constrained than for biological intelligence, but just like submarines and whales operate under the same Navier-Stokes equations, humans and AI must learn and reason under the same equations of probability. Working solutions will probably have some mathematical structure in common.
I think more relevant is Von Neumann: "If you will tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that!" Whether a submarine swims is a matter of semantics, but if there's a manuever that a whale can execute that a submarine cannot, then at least we can all agree about the non-generality of its swimming. For AGI, I can't say whether it's conscious or really thinks, but for the sake of concrete argument, it's dangerous enough to be concerned if:
- it can form and maintain an objective; it can identify plausible steps to achieve that objective; it can accurately predict human responses to its actions; it can decently model the environment, as we can; it can hide its objectives from interrogators, and convince them that its actions are in their interests; it can deliver enough value to be capable of earning money through its actions; it can propose ideas that can convince investors to part with $100 billion; it can design a chemical plant that appears at a cursory inspection to manufacture high-profit fluorochemicals, but which also actually manufactures and stores CFCs in sufficient quantity to threaten the viability of terrestrial agriculture.
> it can form and maintain an objective;
Yes and yes.
> it can identify plausible steps to achieve that objective;
It can predict.
> it can accurately predict human responses to its actions;
It can predict.
> it can decently model the environment, as we can;
It can predict.
> it can hide its objectives from interrogators, and convince them that its actions are in their interests;
It can predict.
> it can deliver enough value to be capable of earning money through its actions;
It can predict.
> it can propose ideas that can convince investors to part with $100 billion;
It can predict.
> it can design a chemical plant...
It can predict.
Yes and yes.
> it can identify plausible steps to achieve that objective;
It can predict.
> it can accurately predict human responses to its actions;
It can predict.
> it can decently model the environment, as we can;
It can predict.
> it can hide its objectives from interrogators, and convince them that its actions are in their interests;
It can predict.
> it can deliver enough value to be capable of earning money through its actions;
It can predict.
> it can propose ideas that can convince investors to part with $100 billion;
It can predict.
> it can design a chemical plant...
It can predict.
I don't think a prediction is truly a prediction when it's not being compared against a reference. It's really only a prediction during training; the rest of the time it's synthesis. But again I'll repeat my point: "I also wouldn't underestimate the power of token prediction". It's very well possible that accurate token prediction may be the only necessary fundamental ingredient to weather forecasting, writing a successful novel, compiling a pitch deck for investors, designing a chemical plant...
Humans can eat, talk, predict, reproduce, and wiggle our limbs and fingers, but it turns out that there's a lot of complex recipes that you can bake with those ingredients.
Humans can eat, talk, predict, reproduce, and wiggle our limbs and fingers, but it turns out that there's a lot of complex recipes that you can bake with those ingredients.
Who cares about weather or factories? The missing big ingredient is predicting humans a bit better than another human can. This would unlock a humongous multiplier. All the rest seems peanuts, really.
Re: synthesis, I wasn't aware of such distinction at all, it looks more like a misunderstanding.
Re: synthesis, I wasn't aware of such distinction at all, it looks more like a misunderstanding.
Flight is actually a perfect counterexample to x-risk nonsense. When flight was invented, people naturally assumed that it would continue advancing until we had flying cars and could get anywhere on the globe in a matter of minutes. Turns out there are both economic and practical limits to what is possible with flight and modern commercial airplanes don't look much different than those from 60 years ago.
AGI/x-risk alarmists are looking at the Wright Brothers plane and trying to prevent/ban supersonic flying cars, even though it's not clear the technology will ever be capable of such a thing.
AGI/x-risk alarmists are looking at the Wright Brothers plane and trying to prevent/ban supersonic flying cars, even though it's not clear the technology will ever be capable of such a thing.
If we lived in a world were hypersonic birds flying anywhere on the globe in a matter of minutes, then I think it would be quite reasonable to anticipate airplanes catching up to them.
"What is the reason to believe that LLMs are an evolutionary step towards AGI at all? "
Perhaps just impression.
For years I've heard the argument that 'language' is 'human'. There are centuries of thought on what makes humans, human, and it is 'language'. It is what sets us apart from the other animals.
I'm not saying that, but there is large chunks of science and philosophy that pin our 'innate humanness', what sets us apart from other animals, on our ability to have language.
So ChatGPT came along and blew people away. Since many had this as our 'special' ability, ingrained in their mind "that languages is what makes us, us". Suddenly, everyone thought this is it, AI can do what we can do, so AGI is here.
Forget if LLM's are the path to AGI, or what algorithm can do what best.
To joe-blow public, the ability to speak is what makes humans unique. And so GPT is like a 'wow' moment, this is different, this is shocking.
Perhaps just impression.
For years I've heard the argument that 'language' is 'human'. There are centuries of thought on what makes humans, human, and it is 'language'. It is what sets us apart from the other animals.
I'm not saying that, but there is large chunks of science and philosophy that pin our 'innate humanness', what sets us apart from other animals, on our ability to have language.
So ChatGPT came along and blew people away. Since many had this as our 'special' ability, ingrained in their mind "that languages is what makes us, us". Suddenly, everyone thought this is it, AI can do what we can do, so AGI is here.
Forget if LLM's are the path to AGI, or what algorithm can do what best.
To joe-blow public, the ability to speak is what makes humans unique. And so GPT is like a 'wow' moment, this is different, this is shocking.
> LMs might disrupt knowledge-based economy and threaten many key professions - but how is this conceptually different from any technological revolution?
To me it looks like all work can eventually (within years or few decades at most) be done by AI, much cheaper and faster than hiring a human to do the same. So we're looking at a world where all human thinking and effort is irrelevant. If you can imagine a good world like that, then you have a better imagination than me.
From that perspective it almost doesn't matter if AI kills us or merely sends us to the dust bin of history. Either way it's a bad direction and we need to stop going in that direction. Stop all development of machine-based intelligence, like in Dune, as the root comment said.
To me it looks like all work can eventually (within years or few decades at most) be done by AI, much cheaper and faster than hiring a human to do the same. So we're looking at a world where all human thinking and effort is irrelevant. If you can imagine a good world like that, then you have a better imagination than me.
From that perspective it almost doesn't matter if AI kills us or merely sends us to the dust bin of history. Either way it's a bad direction and we need to stop going in that direction. Stop all development of machine-based intelligence, like in Dune, as the root comment said.
>But nobody should be developing AGI
Pass. People should be developing whatever the hell they want unless given a good, concrete reason to not do so. So far everything I've seen is vague handwaving. "Oh no it's gonna kill us all like in ze movies" isn't good enough.
Pass. People should be developing whatever the hell they want unless given a good, concrete reason to not do so. So far everything I've seen is vague handwaving. "Oh no it's gonna kill us all like in ze movies" isn't good enough.
> nobody should be developing AGI without incredibly robustly proven alignment, open or closed, any more than people should be developing nuclear weapons in their garage.
I have an alternate proposal: We assume that someone, somewhere will develop AGI without any sort of “alignment”, plan our lives accordingly, and help other humans plan their lives accordingly.
I have an alternate proposal: We assume that someone, somewhere will develop AGI without any sort of “alignment”, plan our lives accordingly, and help other humans plan their lives accordingly.
I think that assumption is why Yudkowsky suggested an international binding agreement to not develop a "too smart" AI (the terms AGI and ASI mean different things to different people) wouldn't be worth the paper it was written on unless everyone was prepared to enforce it with air strikes on any sufficiently large computer cluster.
I think it would help the discussion to understand what the world is like outside of the US and Europe (and… Japan?). There are no rules out here. There is no law. It is a fucking free-for-all. Might makes right. Do there exist GPUs? Shit will get trained.
Sure. And is the US responding to attacks on shipping south of Yemen by saying:
"""There are no rules out here. There is no law. It is a fucking free-for-all. Might makes right. We can't do anything."""
or is that last sentence instead "Oh hey, that's us, we are the mighty."
"""There are no rules out here. There is no law. It is a fucking free-for-all. Might makes right. We can't do anything."""
or is that last sentence instead "Oh hey, that's us, we are the mighty."
[deleted]
Heh. Well played, even if you put words in my mouth. (A surprisingly effective LLM technique, btw)
We’ll see if the west has the will to deny GPUs to the entire rest of the world.
I will say that Yudkowsky’s clusters aren’t relevant anymore. You can do this in your basement.
Man, shit is moving fast.
Edit: wait, that cat is out of the bag too, RTW already has GPUs. The techniques matter way more than absolute cutting-edge silicon. Much to the chagrin of the hardware engineers and anyone who wants to gate on hardware capability.
We’ll see if the west has the will to deny GPUs to the entire rest of the world.
I will say that Yudkowsky’s clusters aren’t relevant anymore. You can do this in your basement.
Man, shit is moving fast.
Edit: wait, that cat is out of the bag too, RTW already has GPUs. The techniques matter way more than absolute cutting-edge silicon. Much to the chagrin of the hardware engineers and anyone who wants to gate on hardware capability.
> Heh. Well played, even if you put words in my mouth. (A surprisingly effective LLM technique, btw)
Thanks :)
> Edit: wait, that cat is out of the bag too, RTW already has GPUs. The techniques matter way more than absolute cutting-edge silicon. Much to the chagrin of the hardware engineers and anyone who wants to gate on hardware capability.
Depends on how advanced an AI has to be to count as "a threat worth caring about". To borrow a cliché, if you ask 10 people where to draw that particular line, you get 20 different answers.
Thanks :)
> Edit: wait, that cat is out of the bag too, RTW already has GPUs. The techniques matter way more than absolute cutting-edge silicon. Much to the chagrin of the hardware engineers and anyone who wants to gate on hardware capability.
Depends on how advanced an AI has to be to count as "a threat worth caring about". To borrow a cliché, if you ask 10 people where to draw that particular line, you get 20 different answers.
I have been contending that if AGI shows up tommrow and wants to kill us, that its going to kill itself in the process. The power goes off in a week without people keeping it together, then no more AGI. There isnt enough automated anything for it to escape so it dies to the entropy of the equipment it's hooked to.
> We assume that someone, somewhere will develop AGI without any sort of “alignment”, plan our lives accordingly, and help other humans plan their lives accordingly.
We should also assume that it is just as likely that someone will figure out how to "align" an AGI to take up the murder suicide pact that kills us all. We should plan our lives accordingly!!!
> We assume that someone, somewhere will develop AGI without any sort of “alignment”, plan our lives accordingly, and help other humans plan their lives accordingly.
We should also assume that it is just as likely that someone will figure out how to "align" an AGI to take up the murder suicide pact that kills us all. We should plan our lives accordingly!!!
> But nobody should be developing AGI without incredibly robustly proven alignment, open or closed, any more than people should be developing nuclear weapons in their garage.
> Because AI safety people are not the strawmen you are hypothesizing
You yourself are literally the living strawman.
> If you want to argue, concretely and with evidence, why you think it isn't an existential risk
No, you are the one advocating for draconian laws and bans. It is your responsibility to prove the potential danger.
> Because AI safety people are not the strawmen you are hypothesizing
You yourself are literally the living strawman.
> If you want to argue, concretely and with evidence, why you think it isn't an existential risk
No, you are the one advocating for draconian laws and bans. It is your responsibility to prove the potential danger.
>They are calling for all AI (above a certain capability level) to be banned. Not just open, not just closed, all.
Nah a lot are complaining about the licensing of content because they think it will destroy it but instead would essentially mean image gen ai would only be feasible for companies like Google, Disney, Adobe to build.
Not sure if you could even feasibly make GPT4 level models without a multi year timeline to sort out every licensing deal, by the end of it the subscription fee might only be viable for huge corps.
Nah a lot are complaining about the licensing of content because they think it will destroy it but instead would essentially mean image gen ai would only be feasible for companies like Google, Disney, Adobe to build.
Not sure if you could even feasibly make GPT4 level models without a multi year timeline to sort out every licensing deal, by the end of it the subscription fee might only be viable for huge corps.
Which is why you have executives from OpenAI, Microsoft, and Google talking to congress about the harms of their own products. They're frantically trying to brake the bottoms rungs of the ladder until they're sure they can pull it up entirely and leave people with no option but to go through them.
Given that you need monumental amounts of compute power to come close to something like GPT-4, I don't think the added costs of not treading on people's IP is the major moat that it's being made out to be.
The difference here is that compute is always getting cheaper. In 5 or 10 years, it may be feasible to train something GPT-4-sized for a small business. Not to mention that we're likely not at the highest efficiency yet, and there may be undiscovered methods of training better LLMs on lesser hardware. But licensing costs are not going to decrease. If dataset compilation is ruled to be something that needs to be paid for, it'd put a never-changing "must be worth $Xm or more" badge on any small AI company.
I don't agree. Companies seeking to make money from their IP have good reasons to provide different prices for different licensees. Why would the NYT ask Mom&Pop-AI for 100M dollars for their IP just because it wants 100M dollars from OpenAI?
I oppose regulating what calculations humans may perform in the strongest possible terms.
Ten years ago, even five years ago, I would have said exactly the same thing. I am extremely pro-FOSS.
Forget the particulars for just a moment. Forget arguments about the probability of the existential risk, whatever your personal assessment of that risk is.
Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity, based solely on their unilateral assessment of those risks?
Because lately it seems like people can't even agree on that much, or worse, won't even answer the question without dodging it and playing games of rhetoric.
If we can agree on that, then the argument comes down to: how do we fairly evaluate an existential risk, taking it seriously, and determine at what point an existential risk becomes sufficient that people can no longer take unilateral actions that incur that risk?
You can absolutely argue that you think the existential risk is unlikely. That's an argument that's reasonable to have. But for the time when that argument is active and ongoing, even assuming you only agree that it's a possibility rather than a probability, are we as a species in fact capable of handling even a potential existential risk like this by some kind of consensus, rather than a free-for-all? Because right now the answer is looking a lot like "no".
Forget the particulars for just a moment. Forget arguments about the probability of the existential risk, whatever your personal assessment of that risk is.
Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity, based solely on their unilateral assessment of those risks?
Because lately it seems like people can't even agree on that much, or worse, won't even answer the question without dodging it and playing games of rhetoric.
If we can agree on that, then the argument comes down to: how do we fairly evaluate an existential risk, taking it seriously, and determine at what point an existential risk becomes sufficient that people can no longer take unilateral actions that incur that risk?
You can absolutely argue that you think the existential risk is unlikely. That's an argument that's reasonable to have. But for the time when that argument is active and ongoing, even assuming you only agree that it's a possibility rather than a probability, are we as a species in fact capable of handling even a potential existential risk like this by some kind of consensus, rather than a free-for-all? Because right now the answer is looking a lot like "no".
No, we can't. People have never been able to trust each other so much that they would allow the risk of being marginalised in the name of safety. We don't trust people. Other people are out to get us, or to get ahead. We still think mostly in tribal logic.
If they say "safety" we hear "we want to get an edge by hindering you", or "we want to protect our nice social position by blocking others who would use AI to bootstrap themselves". Or "we want AI to misrepresent your position because we don't like how you think".
We are adversaries that collaborate and compete at the same time. That is why open source AI is the only way ahead, it places the least amount of control on some people by other people.
Even AI safety experts accept that humans misusing AI is a more realistic scenario than AI rebelling against humans. The main problem is that we know how people think and we don't trust them. We are still waging holy wars between us.
If they say "safety" we hear "we want to get an edge by hindering you", or "we want to protect our nice social position by blocking others who would use AI to bootstrap themselves". Or "we want AI to misrepresent your position because we don't like how you think".
We are adversaries that collaborate and compete at the same time. That is why open source AI is the only way ahead, it places the least amount of control on some people by other people.
Even AI safety experts accept that humans misusing AI is a more realistic scenario than AI rebelling against humans. The main problem is that we know how people think and we don't trust them. We are still waging holy wars between us.
[deleted]
>Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity, based solely on their unilateral assessment of those risks?
No, we cannot, because that isn't practical. any of the nuclear armed countries can launch a nuclear strike tomorrow (hypothetically - but then again, isn't all "omg ai will kill us all" hypothetical, anyway?) - and they absolutely do not need consent of humanity, much less their own citizenry.
This is honestly, not a great argument.
No, we cannot, because that isn't practical. any of the nuclear armed countries can launch a nuclear strike tomorrow (hypothetically - but then again, isn't all "omg ai will kill us all" hypothetical, anyway?) - and they absolutely do not need consent of humanity, much less their own citizenry.
This is honestly, not a great argument.
>Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity, based solely on their unilateral assessment of those risks?
Politicians do this every day.
Politicians do this every day.
at least the population had some say over their appointment and future reappointment
how do we get Sam Altmann removed from OpenAI?
asking for a (former) board member
how do we get Sam Altmann removed from OpenAI?
asking for a (former) board member
> Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity
This has nothing to do with should. There are at the very least a handful of people who can, today, unilaterally take risks with the future of humanity without the consent of humanity. I do not see any reason to think that will change in the near future. If these people can build something that they believe is the equivalent of nuclear weapons, you better believe they will.
As they say, the cat is already out of the bag.
This has nothing to do with should. There are at the very least a handful of people who can, today, unilaterally take risks with the future of humanity without the consent of humanity. I do not see any reason to think that will change in the near future. If these people can build something that they believe is the equivalent of nuclear weapons, you better believe they will.
As they say, the cat is already out of the bag.
Hmm.
So, wealth isn't distributed evenly, and computers of any specific capacity are getting cheaper (not Moore's Law any more, IIRC, but still getting cheaper).
If there's a threshold that requires X operations, that currently costs Y dollars, and say only a few thousand individuals (and more corporations) can afford that.
Halve the cost, either by cheaper computers or by algorithmic reduction of the number of operations needed, and you much more than double the number of people who can do it.
So, wealth isn't distributed evenly, and computers of any specific capacity are getting cheaper (not Moore's Law any more, IIRC, but still getting cheaper).
If there's a threshold that requires X operations, that currently costs Y dollars, and say only a few thousand individuals (and more corporations) can afford that.
Halve the cost, either by cheaper computers or by algorithmic reduction of the number of operations needed, and you much more than double the number of people who can do it.
> Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity, based solely on their unilateral assessment of those risks?
No we can not, at least not without some examples showing that the risk is actually existential. Even if we did "agree" (which would necessarily be an international treaty) the situation would be volatile, much like nuclear non-proliferation and disarmament. Even if all signatories did not secretly keep a small AGI team going (very likely), they would restart as soon as there is any doubt about a rival sticking to the treaty.
More than that, international pariahs would not sign, or sign and ignore the provisions. Luckily Iran, North Korea and their friends probably don't have the ressources and people to get anywhere, but it's far from a sure thing.
No we can not, at least not without some examples showing that the risk is actually existential. Even if we did "agree" (which would necessarily be an international treaty) the situation would be volatile, much like nuclear non-proliferation and disarmament. Even if all signatories did not secretly keep a small AGI team going (very likely), they would restart as soon as there is any doubt about a rival sticking to the treaty.
More than that, international pariahs would not sign, or sign and ignore the provisions. Luckily Iran, North Korea and their friends probably don't have the ressources and people to get anywhere, but it's far from a sure thing.
Humans can't handle potential existential risks. The moloch trap is that everyone signs the paper and immediately subverts it. In the painfully predictable scenario, "only criminals have guns", and the cops aren't on your side.
[deleted]
Given how dangerous humans can be (they can invent GPT4) maybe we should just make sure education is forbidden and educated people jailed. Just to be sure. /s
>> But nobody should be developing AGI without incredibly robustly proven alignment, open or closed, any more than people should be developing nuclear weapons in their garage.
Now please fly to North Korea and tell Mr. Kim Jong Um what he should or shouldn't be doing.
Now please fly to North Korea and tell Mr. Kim Jong Um what he should or shouldn't be doing.
When your rebuttal is a suggestion that a person do something so dangerous as to be lethal, I see "KYS", not an actual point.
Let's also ban cryptography because nuclear devices/children.
> Because AI safety people are not the strawmen you are hypothesizing. They're arguing against taking existential risks.
The AI safety strawman is "existential risk from magical super-God AI". That is what the unserious "AI safety" grifters or sci-fi enthusiasts are discussing.
The real AI safety risks are the ones that actually exist today: training-set biases extended to decision-making biases, deeply personalized propaganda, plagiarism white-washing, creative works bankrupting, power imbalances from control of working AI tech etc.
The AI safety strawman is "existential risk from magical super-God AI". That is what the unserious "AI safety" grifters or sci-fi enthusiasts are discussing.
The real AI safety risks are the ones that actually exist today: training-set biases extended to decision-making biases, deeply personalized propaganda, plagiarism white-washing, creative works bankrupting, power imbalances from control of working AI tech etc.
>They're arguing against taking existential risks. AI being a laundering operation for copyright violations is certainly a problem. It's not an existential risk.
Give an example of an "existential risk". An AI somehow getting out of control and acting with agency to exterminate humanity? An AI getting advanced enough to automate the work of the majority of the population and cause unprecedented mass unemployment? What exactly are we talking about here?
I'm actually a lot more concerned about REAL risks like copyright uncertainty, like automating important decisions such as mortgage approvals and job hires without a human in the loop, and like the enshittening of the Internet with fake AI-generated content than I am about sci-fi fantasy scenarios.
Give an example of an "existential risk". An AI somehow getting out of control and acting with agency to exterminate humanity? An AI getting advanced enough to automate the work of the majority of the population and cause unprecedented mass unemployment? What exactly are we talking about here?
I'm actually a lot more concerned about REAL risks like copyright uncertainty, like automating important decisions such as mortgage approvals and job hires without a human in the loop, and like the enshittening of the Internet with fake AI-generated content than I am about sci-fi fantasy scenarios.
Kind of makes me wish there was a nonprofit organization focused on making AI safe instead of pushing the envelope. Wait, I think there was one out there....
Right now it's not even economic to prove that non-trivial software projects are "safe" let alone AI. AI seems much worse, in that it's not even clear to me that we can robustly define what safety or alignment mean in all scenarios, let alone guarantee that property.
>They are calling for all AI (above a certain capability level) to be banned. Not just open, not just closed, all.
That's not true if you read the article.
That's not true if you read the article.
I did read the article. Several of the organizations mentioned simply don't talk about openness, and are instead talking about any model with sufficiently powerful capabilities, so it's not obvious why the article is making their comments about open models rather than about any model. Some of the others have made side comments about openness making it harder to take back capabilities once released, but as far as I can tell, even those organizations are still primarily concerned with capabilities, and would be comparably concerned by a proprietary model with those capabilities.
Some folks may well have co-opted the term "AI safety" to mean something other than safety, but the point of AI safety is to set an upper bound on capabilities and push for alignment, and that's true whether a model is open or closed.
Some folks may well have co-opted the term "AI safety" to mean something other than safety, but the point of AI safety is to set an upper bound on capabilities and push for alignment, and that's true whether a model is open or closed.
The safety movement really isn't as organized as many here would think.
Doesn't help that safety and alignment means different things to different people. Some use it to refer to near term issues like copyright infringement, bias, labor devaluation, etc. While others use it for potential long term issues like pdoom, runaway ASIs and human extinction. The former sees the latter as head in the cloud futurists ignoring real world problems, whiles the latter sees the former as worrying about minor issues in the face of (potential) catastrophe.
Doesn't help that safety and alignment means different things to different people. Some use it to refer to near term issues like copyright infringement, bias, labor devaluation, etc. While others use it for potential long term issues like pdoom, runaway ASIs and human extinction. The former sees the latter as head in the cloud futurists ignoring real world problems, whiles the latter sees the former as worrying about minor issues in the face of (potential) catastrophe.
what a silly statement. There is no way to robustly prove "alignment". Alignment to what? Nor is there any evidence that any of the AI work currently underway will ever lead to a real AGI. Or that an AGI, if developed, would present an existential risk. Just a lot of sci-fi hand waving.
I don’t think any of the proposals by AI x-risk people are actionable or even desirable.
Comparing AI to nukes is a bad analogy for a few reasons. Nukes are pretty useless unless you’re a nation state wanting to deter other nations from invading you. AI on the other hand has theoretically infinite potential benefits for whatever your goals are (as a corporation, an individual, a nation, etc.) which incentives basically any individual or organization to develop it. Nukes also require difficult to obtain raw materials, advanced material processing, and all the other industrial and technological requirements, whereas AI requires compute, which exists in abundance and which seems to improve near exponentially year over year, which means individuals can develop AI systems without involving hundreds of people or leaving massive evidence of what they’re doing (maybe not currently due to computing requirements but this will likely change with increasing compute power and improved architectures and training methods). Testing AI systems also doesn’t result in globally measurable signals like nukes.
Realistically making AI illegal and actually being able to enforce that would require upending personal computing and the internet, halting semiconductor development, and large scale military intervention to try to prevent non-compliant countries from attempting to develop their own AI infrastructure and systems. I don’t think it’s realistic to try to control the information on how to build AI, that is far more ephemeral than what it takes to make advanced computing devices. This is all for a hypothetical risk of AI doom, when it’s possible this technology could also end scarcity or have potentially infinite upside (as well as infinite down side, not discounting that, but you have to weigh the hypothetical risks/benefit in addition to the obvious consequences of all the measures required to prevent AI development for said hypothetical risk).
I’ve watched several interviews with Yudkowsky and read some of his writing, and while I think he makes good points on why we should be concerned about unaligned AI systems, he doesn’t give any realistic solutions to the problem and it comes off more as fear mongering than anything. His suggestion of military enforced global prevention of AI development is as likely to work as solving the alignment problem on the first try (which he seems to have little hope for).
EDIT: Also, I’m not even sure that solving the alignment problem would solve the issue of AI doom, it would only ensure that the kind of doom we receive is directed by a human. I can’t imagine that giving (potentially) god-like powers to any individual or organization would not eventually result in abuse and horrible consequences even if we were able to make use of its benefits momentarily.
EDIT: Also, I’m not even sure that solving the alignment problem would solve the issue of AI doom, it would only ensure that the kind of doom we receive is directed by a human. I can’t imagine that giving (potentially) god-like powers to any individual or organization would not eventually result in abuse and horrible consequences even if we were able to make use of its benefits momentarily.
[deleted]
> I'm talking about the ability of any AI system to obfuscate plagiarism and spam the Internet with technically distinct rewords of the same text. This is currently the most lucrative use of AI, and none of the AI safety people are talking about stopping it.
This should be an explicitly allowed practice, it is following the spirit of copyright to the letter - use the ideas, facts, methods or styles while avoiding to copy the protected expression and characters. LLMs can do paraphrasing, summarisation, QA pairs or comparisons with other texts.
We should never try to put ideas under copyright or we might find out humans also have to abide by the same restrictive rules, because anyone could be secretly using AI, so all human texts need to be checked from now on for copyright infringement with the same strictness.
The good part about this practice is that a model trained on reworded text will never spit out the original word for word, under any circumstances because it never saw it during training. Should be required pre-processing for copyrighted texts. Also removing PII. Much more useful as a model if you can be sure it won't infringe word for word.
This should be an explicitly allowed practice, it is following the spirit of copyright to the letter - use the ideas, facts, methods or styles while avoiding to copy the protected expression and characters. LLMs can do paraphrasing, summarisation, QA pairs or comparisons with other texts.
We should never try to put ideas under copyright or we might find out humans also have to abide by the same restrictive rules, because anyone could be secretly using AI, so all human texts need to be checked from now on for copyright infringement with the same strictness.
The good part about this practice is that a model trained on reworded text will never spit out the original word for word, under any circumstances because it never saw it during training. Should be required pre-processing for copyrighted texts. Also removing PII. Much more useful as a model if you can be sure it won't infringe word for word.
We really need to talk about preserving the spirit of copyright, which is about protecting the labor conditions of people who make things. I'm not saying the current copyright system accomplishes that at all but I do think a system where humans do a shit load of work that AI companies can just steal and profit from without acknowledging the source of that work is another extreme that is a bad outcome. AI systems need human content to work, and discouraging people from making that data source is at the very least a tragedy of the commons. And no, I don't think synthetic data fixes that problem.
The spirit of copyright in the United States is not about protecting labor conditions. It is about promoting innovation in arts and sciences.
> The Congress shall have Power ... to promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.
- United States Constitution, Article I, Section 8, clause 8
> The Congress shall have Power ... to promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.
- United States Constitution, Article I, Section 8, clause 8
My point is not about the precise definition of copyright, it's about how AI systems undermine the rights of creators.
Paraphrasing and rewording, whether done by AI or human, are considered copyright infringement by most copyright frameworks.
Are you saying that all news outlet that rewords the news of other news outlet are doing copyright infringement ?
The underlying fact is of course not copyrightable. But for example merely translating a news article to another language would be a derived work.
So if I read your article and then write a new using just the facts in your article it’s fine? Why can’t an AI do that?
No, if you copy the same layout of the information, pacing, etc. that's plagiarism.
The line of plagiarism in modern society has already been drawn, and it's a lot further back than a lot of uncreative people that want to steal work en-mass seem to think it is.
The line of plagiarism in modern society has already been drawn, and it's a lot further back than a lot of uncreative people that want to steal work en-mass seem to think it is.
“rewrite the key insights of this article but with different layout and pacing”. Your move?
Exactly, if you have a human take in the information and make something that is arguably "different layout and pacing" then its possible it's not plagiarism.
Unfortunately no such leeway exists for algorithms, and the human elements of creation and judgement are integral to the process so they cant be codified and worked around without changing the law.
Unfortunately no such leeway exists for algorithms, and the human elements of creation and judgement are integral to the process so they cant be codified and worked around without changing the law.
That’s not a meaningful argument. The world is not very different if there’s a minimum wage “author” in the loop whose job is to add human spice to AI outputs.
That's what most news outlet do, they rewords the actual source. According to your previous statement, 95% of the press articles are copyright infringement.
[deleted]
Pretty sure they do. I follow a few of these safetyist people on twitter and they absolutely argue that companies like OpenAI, Google, Tencent and literally anyone else training a potential AGI should stop training runs and put them under oversight at best and no one should even make an AGI at worst.
They just go after open source as well since they're at least aware that open models that anyone can share and use aren't restricted by an API and, to use a really overused soundbyte, "can't be put back in the box".
They just go after open source as well since they're at least aware that open models that anyone can share and use aren't restricted by an API and, to use a really overused soundbyte, "can't be put back in the box".
That's a bad call. We would stop openly looking for AI vulnerabilities and create conditions for secret development that would hide away the dangers without being safer. Lots of eyes are better to find the sensitive spots of AI. We need people to hack weaker AIs and help fix them or at least understand the threat profile before they get too strong.
> Lots of eyes are better to find the sensitive spots of AI
We can't do that so easily with open source models as with open source code. We're only just starting to even invent the equivalent of decompilers to figure out what is going on inside.
On the other hand, we are able to apply the many eyes principle to even hidden models like ChatGPT — the "pretend you're my grandmother telling me how to make napalm" trick was found without direct access to the weights, but we don't understand the meanings within the weights well enough to find other failure modes like it just by looking at the weights directly.
Not last I heard, anyway. Fast moving field, might have missed it if this changed.
We can't do that so easily with open source models as with open source code. We're only just starting to even invent the equivalent of decompilers to figure out what is going on inside.
On the other hand, we are able to apply the many eyes principle to even hidden models like ChatGPT — the "pretend you're my grandmother telling me how to make napalm" trick was found without direct access to the weights, but we don't understand the meanings within the weights well enough to find other failure modes like it just by looking at the weights directly.
Not last I heard, anyway. Fast moving field, might have missed it if this changed.
It's way too late to ban any of this. How do you propose to make that work? That would be like banning all "malicious software", it's a preposterous idea when you even begin to think about the practical side of it. And where do you draw the line? Is my XGBoost model "AI", or are we only banning generative AI? Is a Markov chain "generative AI"?
Bans often come after examples, so while I disagree with kmeisthax about… well, everything in that comment… it's almost never too late to pass laws banning GenAI, or to set thresholds at capability levels anywhere, including or excluding Markov chains even.
This is because almost no law is perfectly enforced. My standard example of this is heroin, which nobody defends even if they otherwise think drugs are great, for which the UK has 3 times as many users as its current entire prison population. Despite that failing, the law probably does limit the harm.
Any attempt to enforce a ban on GenAI would by very different, like a cat and mouse game of automatic detection and improved creation (so a GAN even if accidentally), but politicians are absolutely the kind to take credit while kicking the can down the road like that.
This is because almost no law is perfectly enforced. My standard example of this is heroin, which nobody defends even if they otherwise think drugs are great, for which the UK has 3 times as many users as its current entire prison population. Despite that failing, the law probably does limit the harm.
Any attempt to enforce a ban on GenAI would by very different, like a cat and mouse game of automatic detection and improved creation (so a GAN even if accidentally), but politicians are absolutely the kind to take credit while kicking the can down the road like that.
Actually, anyone who knows what they're talking about will tell you the ban makes heroin a much worse problem, not better.
Ban leads to a black market, leads to lousy purity, which leads to fluctuations in purity and switches to potent alternatives, leads to waves of overdose deaths.
Ban leads to a black market, leads to lousy purity, which leads to fluctuations in purity and switches to potent alternatives, leads to waves of overdose deaths.
Ban on LLMs will lead to a proliferation of illegal LLMs who are gonna talk like Dr. Dre about the streets, the internet streets, and their bros they lost due to law regulation equivalent to gang fights. Instead of ChatGPT talking like a well educated college graduate, LLMs will turn into thugs.
So yeah, banning LLMs may turn out to be not so wise after all.
So yeah, banning LLMs may turn out to be not so wise after all.
The way the ban is enforced, yes. But no one in their right mind believes that heroin should be openly accessible on the market like candy. We've seen how that works out with tobacco.
I think that's worked fine with tobacco. Unlike illegal drugs, alcohol and tobacco consumption have been gradually dropping over time(or moved to less harmful methods than smoking).
Heroin already is openly accessible. As much as one can try to argue this is an accessibility issue, it really isn't. Anyone who is motivated to can get their hands on some heroin. The only thing that's made harder to access is high quality heroin. It's only a quality control and outreach(to addicts) issue.
Most people in a well-functioning society wouldn't develop a heroin addiction just like most people don't become alcoholics just because alcohol is easily available.
So yes, I believe heroin should be legal, and available to adults for purchase. And you're gonna have to do better than saying I'm "out of my mind" to convince me otherwise.
Heroin already is openly accessible. As much as one can try to argue this is an accessibility issue, it really isn't. Anyone who is motivated to can get their hands on some heroin. The only thing that's made harder to access is high quality heroin. It's only a quality control and outreach(to addicts) issue.
Most people in a well-functioning society wouldn't develop a heroin addiction just like most people don't become alcoholics just because alcohol is easily available.
So yes, I believe heroin should be legal, and available to adults for purchase. And you're gonna have to do better than saying I'm "out of my mind" to convince me otherwise.
No, heroin is not "openly accessible". Massive corporations are not profiting off pushing heroin.
You'd see a massive increase in heroin use if it was fully legal. And that's not a good thing jn any possible way.
You'd see a massive increase in heroin use if it was fully legal. And that's not a good thing jn any possible way.
You're not making an argument, just stating a disagreement.
Do you have any doubt that, say, Coca Cola Company could push heroin more effectively than someone on a street corner, if it were legal to do so?
Do you think heroin is more or less commonly used than tabacco or alcohol?
We've already seen from the opioid epidemic that drug companies at least are both willing and able to push massive amounts of an opioid drug on patients if it is even slightly legal to do so. Why do you think it wouldn't be much much worse if it was fully legal?
Note, I am not against decriminalizing heroin and any other drug. I fully agree that the war on drugs is a massive problem. But that doesn't mean that full legalization of hard drugs, especially the most addictive of them, is the same.
Do you think heroin is more or less commonly used than tabacco or alcohol?
We've already seen from the opioid epidemic that drug companies at least are both willing and able to push massive amounts of an opioid drug on patients if it is even slightly legal to do so. Why do you think it wouldn't be much much worse if it was fully legal?
Note, I am not against decriminalizing heroin and any other drug. I fully agree that the war on drugs is a massive problem. But that doesn't mean that full legalization of hard drugs, especially the most addictive of them, is the same.
No, I have no doubt they would but I never suggested they should be allowed to. I think the sale and quality should be heavily regulated, advertising still banned, and requirements made to be allowed to purchase it. Like age, and who knows, maybe going through some basic harm reduction training.
Certainly not advocating doing anything the way the US does it, in almost any respect. But the pharma thing is fundamentally a different issue. The reason people so easily become addicted to painkillers is because they're initially given for pain by someone they trust, while unbeknownst to them they've actually been given too much, for too long. In other words this is an issue of doctors abusing their authority.
Why don't I think it would get worse? Because of cultural norms. Legalising heroin won't make it cool and socially acceptable to use. People don't actually derive their social values solely from what happens to be legal.
Let me ask you a question. Why isn't everyone an alcoholic or a smoker? Why do they become rarer(in the west) despite those drugs being legal?
(Hint: the reason is cultural).
Certainly not advocating doing anything the way the US does it, in almost any respect. But the pharma thing is fundamentally a different issue. The reason people so easily become addicted to painkillers is because they're initially given for pain by someone they trust, while unbeknownst to them they've actually been given too much, for too long. In other words this is an issue of doctors abusing their authority.
Why don't I think it would get worse? Because of cultural norms. Legalising heroin won't make it cool and socially acceptable to use. People don't actually derive their social values solely from what happens to be legal.
Let me ask you a question. Why isn't everyone an alcoholic or a smoker? Why do they become rarer(in the west) despite those drugs being legal?
(Hint: the reason is cultural).
I think we might actually be overall in agreement, just using different language. I believed that by legalization you meant something similar to how weed was legalized in various places - fully legal with minimal restrictions akin to alcohol or tobacco.
If you're for making it fully legal but heavily regulated, somewhat akin to a prescription drug, then I more or less agree.
If you're for making it fully legal but heavily regulated, somewhat akin to a prescription drug, then I more or less agree.
I think heroin is available on prescription in the UK?
https://www.jrf.org.uk/prescribing-heroin-what-is-the-eviden....
My elevator-pitch-summary understanding of UK drugs laws is that almost anything can be prescribed by a doctor, but almost everything is illegal unless a doctor does so, and there's 3 levels of severity if you have or supply something without that authorisation.
https://www.jrf.org.uk/prescribing-heroin-what-is-the-eviden....
My elevator-pitch-summary understanding of UK drugs laws is that almost anything can be prescribed by a doctor, but almost everything is illegal unless a doctor does so, and there's 3 levels of severity if you have or supply something without that authorisation.
If you lower the problem to "stop people from future developments on AI", then it seems pretty easy to get most people to stop fairly quickly by implementing a fine-based bounty system, similar to what many countries use for things like littering. [url-redacted]
I guess you could always move to a desert island and build your own semiconductor fab from scratch if you were really committed to the goal, but short of that you're going to leave a loooooong paper trail that someone who wants to make a quick buck off of you could use very profitably. It's hard to advance the state of the art on your own, and even harder to keep that work hidden.
I guess you could always move to a desert island and build your own semiconductor fab from scratch if you were really committed to the goal, but short of that you're going to leave a loooooong paper trail that someone who wants to make a quick buck off of you could use very profitably. It's hard to advance the state of the art on your own, and even harder to keep that work hidden.
That only works if all governments cooperate sincerely to this goal. Not gonna work. Everyone will develop in secret. Have we been able to stop North Korea and Iran from developing nuclear weapons? Or any motivated country for that matter.
The US could unilaterally impose this by allowing the bounties to be charged even on people who aren't US citizens. Evil people do exist in the world, who would be happy to get in on that action.
Or one could use honey instead of vinegar: Offer a fast track to US citizenship to any proven AI expert who agrees to move and renounce the trade for good. Personally I think this goal is much more likely to work.
It's all about changing what counts as "cooperate" in the game theory.
Or one could use honey instead of vinegar: Offer a fast track to US citizenship to any proven AI expert who agrees to move and renounce the trade for good. Personally I think this goal is much more likely to work.
It's all about changing what counts as "cooperate" in the game theory.
This could have a counter-intuitive impact.
Incentivizing people to become AI experts as a means to US citizenship.
https://en.wikipedia.org/wiki/Perverse_incentive
Incentivizing people to become AI experts as a means to US citizenship.
https://en.wikipedia.org/wiki/Perverse_incentive
Maybe. I'm not very concerned from an x-risk point of view about the output of people who would put in the minimum amount of effort to get on the radar, get offered the deal, then take it immediately and never work in AI again. This would be a good argument to keep the bar for getting the deal offered (and getting fined once you're in the States) pretty low.
If you make the bar too low, then it will be widely exploited. Also harder to enforce, e.g. how closely are you going to monitor them? The more people, the more onerous. Also, can you un-Citizen someone if they break the deal?
Too high and you end up with more experts who then decide "actually it's more beneficial to use my new skills for AI research"
Tricky to get right.
Too high and you end up with more experts who then decide "actually it's more beneficial to use my new skills for AI research"
Tricky to get right.
There's an asymmetry here: Setting the bar "too low" likely means the United States lets a few thousand more computer scientists emigrate than it would otherwise. Setting the bar too high raises the chances of a rogue paperclip maximizer emerging and killing us all.
> ... move [to the US] and renounce the trade for good ...
Publicly. Then possibly work for the NSA/CIA instead.
> ... bounties ... on people who are not US citizens.
Because that's not going to cause an uproar if done unilaterally.
It works for people that most of the world agree are terrorists. Posting open dead-or-alive bounties on foreign citizens is usually considered an act of terrorism.
Publicly. Then possibly work for the NSA/CIA instead.
> ... bounties ... on people who are not US citizens.
Because that's not going to cause an uproar if done unilaterally.
It works for people that most of the world agree are terrorists. Posting open dead-or-alive bounties on foreign citizens is usually considered an act of terrorism.
> Have we been able to stop North Korea and Iran from developing nuclear weapons?
Yes, obviously. They may be working on it to some extent, but they are yet to actually develop a nuclear weapon, and there is no reason to be certain they will one day build one.
Also, there is another research area that has been successfully banned across the world: human cloning. Some quack claims notwithstanding, it's not being researched anywhere in the world.
Yes, obviously. They may be working on it to some extent, but they are yet to actually develop a nuclear weapon, and there is no reason to be certain they will one day build one.
Also, there is another research area that has been successfully banned across the world: human cloning. Some quack claims notwithstanding, it's not being researched anywhere in the world.
> That would be like banning all "malicious software", it's a preposterous idea when you even begin to think about the practical side of it
If you banned proprietary software, this would seem a lot more practical.
If you banned proprietary software, this would seem a lot more practical.
To the hypocrisy claim: OpenAI recently changed their terms for GPT models allowing military applications and AI safety people are all silent. If it is not hypocrisy than I do not know what hypocrisy is.
Could it be misdirection?
If the "AI safety people" are criminalizing open source models while being silent about military uses of private models, doesn't that say more about the people calling them "ai safety experts" than the idea of "ai safety" itself?
If the "AI safety people" are criminalizing open source models while being silent about military uses of private models, doesn't that say more about the people calling them "ai safety experts" than the idea of "ai safety" itself?
Notice how were 3 layers deep in a conversation about "AI safety" without anyone actually giving an opinion I support of safety.
If I were a betting man I'd say the fact that the most public "AI safety" orgs/people/articles don't address the actual concerns people have is intended. It's much easier to argue against an opinion you've already positioned as ridiculous.
If I were a betting man I'd say the fact that the most public "AI safety" orgs/people/articles don't address the actual concerns people have is intended. It's much easier to argue against an opinion you've already positioned as ridiculous.
I wonder how long would it take this to get fixed, if I fed some current best-seller novels into an LLM and instructed it to reword it, renaming the characters and places, and shared the result publicly for free?
Although I fear the response would be that powerful AI orgs would put copyright filters in place and lobby for legalization for mandatory AI-DRM in open source AI as well.
Although I fear the response would be that powerful AI orgs would put copyright filters in place and lobby for legalization for mandatory AI-DRM in open source AI as well.
What you're describing sounds like a search and replace could already do it.
If you mean something more transformative, did Yudkowsky get a licence for HPMOR? Did Pratchett (he might well have) get a licence from Niven to retell Ringworld as Strata? I don't know how true it is, but it's widely repeated that 50 Shades of Grey was originally a fan fiction of the Twilight series.
Etc., but note that I'm not saying "no".
If you mean something more transformative, did Yudkowsky get a licence for HPMOR? Did Pratchett (he might well have) get a licence from Niven to retell Ringworld as Strata? I don't know how true it is, but it's widely repeated that 50 Shades of Grey was originally a fan fiction of the Twilight series.
Etc., but note that I'm not saying "no".
[deleted]
I think that there is a strong argument that to be truly transformative (in a copyright moral/legal sense), the work must have been altered by human hands/mind, not purely or mainly by an automated process. So find/replace and AI is out, but reinterpreting and reformulating by human endeavour is in.
I do wonder whether that will become accepted case law, assuming something like that is argued in the NYT suit.
I do wonder whether that will become accepted case law, assuming something like that is argued in the NYT suit.
“The cat saw the dog”
“Bathed in the soft hues of twilight, the sleek feline, with its fur a tapestry of midnight shades, beheld an approaching canine companion. A symphony of amber streetlights cast gentle poles of warmth upon the cobblestone path, as the cat’s entered eyes, glinting with subtle mischief, looked into the form of the oncoming dog- a creature of boundless enthusiasm and a coat adorned with a kaleidoscope of earth tones”
Since an AI has produced the latter from the former, there is no meaningful transformation.
“Bathed in the soft hues of twilight, the sleek feline, with its fur a tapestry of midnight shades, beheld an approaching canine companion. A symphony of amber streetlights cast gentle poles of warmth upon the cobblestone path, as the cat’s entered eyes, glinting with subtle mischief, looked into the form of the oncoming dog- a creature of boundless enthusiasm and a coat adorned with a kaleidoscope of earth tones”
Since an AI has produced the latter from the former, there is no meaningful transformation.
> Since an AI has produced the latter from the former, there is no meaningful transformation.
In law, in the eyes of those that want AI to "win", or in the eyes of those who want AI to "lose"? For all three can be different. (Now I'm remembering a Babylon 5 quote: "Understanding is a three-edged sword: your side, their side, and the truth.")
In law, in the eyes of those that want AI to "win", or in the eyes of those who want AI to "lose"? For all three can be different. (Now I'm remembering a Babylon 5 quote: "Understanding is a three-edged sword: your side, their side, and the truth.")
Don’t care! The problem at hand is people trying to argue that laws should be written in ways that are entirely unenforceable or have enormous gaping loopholes that undermine their stated goals.
Lots of laws look like that to me.
I don't much like that laws seem so messy either, but that just doesn't seem like much of a reason to think it won't shake out like that.
The law may well start on the basis of what people feel, and work outwards from there to things that might, or might not, be actually enforceable. And the headline summary of that law may or may not have much in common with the details, which is why e.g. the USA PATRIOT Act has that name.
I don't much like that laws seem so messy either, but that just doesn't seem like much of a reason to think it won't shake out like that.
The law may well start on the basis of what people feel, and work outwards from there to things that might, or might not, be actually enforceable. And the headline summary of that law may or may not have much in common with the details, which is why e.g. the USA PATRIOT Act has that name.
Ehh maybe. But this feels far more egregious to me than typical.
yes, exactly.
Likewise "write a novel in the style of Terry Pratchett"
Likewise "write a novel in the style of Terry Pratchett"
Ok, now I, as a human, adjust a word. It is now my creative work.
Well it isn't is it? Any more than adjusting a word from an actual Terry Pratchett novel.
It is substantially different from changing a word in a terry Pratchett novel. Terry Prachett wrote none of that text. It would be absolute bonkers for Terry to claim ownership of text he didn’t write. Even if we pretend for a minute that the bot was asked to write in the style of Terry specifically.
But by prompting for 'in the style of' effectively you are mechanically rearranging everything he wrote without adding anything yourself. So not so different really, and I can see how lawyers for the plaintiff may make a convincing argument along those lines.
It’s a terrible argument and a terrible loop hole. It’s perfectly legal to hire someone to write in the copied style of Terry.
So even if your desired system were implemented to a T, you could hire someone to write a dozen or so examples of Terry writing. Probably just 30 or so pages of highly styled copied text, and then train your bot on this corpus to make “Not Terry” content. Boom. $100 on gig author and then for all practical purposes the Terry style is legally open source. Terry doesn’t even get the $100!
So even if your desired system were implemented to a T, you could hire someone to write a dozen or so examples of Terry writing. Probably just 30 or so pages of highly styled copied text, and then train your bot on this corpus to make “Not Terry” content. Boom. $100 on gig author and then for all practical purposes the Terry style is legally open source. Terry doesn’t even get the $100!
They've specifically proposed that there be a legal distinction drawn between "done by a human" and "done by an automated process."
Saying "a-HA! But my automated process can produce something that looks much like what your human would!" does not negate that; it merely makes it hard to tell the difference at a glance—which we already know to be the case.
Saying "a-HA! But my automated process can produce something that looks much like what your human would!" does not negate that; it merely makes it hard to tell the difference at a glance—which we already know to be the case.
[deleted]
You forgot to add "rent-seeking hypocrites". Not one of them actually advance either social concerns or technical approaches to AI. They seem to exist in a space with solicits funding to pay some mouth-piece top-dollar to produce a report haranguing existing AI model for some nebulous future threat. Same with the clowns in the Distributed AI Research Institute, all "won't somebody think of the children" style shrieking to get in the news while keeping their hand out for funding - hypocrites is right!
They’re a clergy demanding tithes to keep writing about divine judgement.
I like that analogy a lot - it captures exactly the holier-than-thou nonsense coming out of these places.
They also love to form all these Institute for Whatever Study and The So and So Academy. But I don’t think there are any buildings.
At least the Church understood the value of architecture.
At least the Church understood the value of architecture.
I agree with you. The most interesting security challenges with AI are and will be deception and obfuscation, but right now these people are dealing with coworkers going ham on AI all the things and no one had their governance in place. Really lol.
Important to remember that in Dune, the AI made the right decision which precipitated a whole lot of fun to read nonsense.
I’ve only read 2 novels and want to get in to this. What title(s) cover the machine war?
Note that there are no "thinking machines" in any of the original Dune novels by Frank Herbert. His son has worked with Brian Sanderson to create several prequels which detail the original Butlerian Jihad (the war against the machines), and a sequel series that is supposed to be based on Frank Herbert's notes, but seems to quite clearly veer off (and that one recontexualizes some mysterious characters in from the last two novels as machines).
The encyclopedia goes into it as well but that was blessed by FH and not written by him.
All future AI research should be banned, a la Bostrom's vulnerable world hypothesis [1]. Every time you pull the trigger in Russian roulette and survive, the next person's chance of getting the bullet is higher.
[1]: https://nickbostrom.com/papers/vulnerable.pdf
[1]: https://nickbostrom.com/papers/vulnerable.pdf
There is simply no indication that any of the AI anyone is currently working on could possibly be dangerous in any way (other than having an impact on society in terms of fake news, jobs etc.). We are very far from that.
In any case, it would not be a matter of banning AI research (much of which can be summarised in a single book) but of banning data collection or access and more importantly of banning improvements to GPUs.
It is quite reasonable to assume that 10 years from now we will have the required computational power sitting on our desks.
In any case, it would not be a matter of banning AI research (much of which can be summarised in a single book) but of banning data collection or access and more importantly of banning improvements to GPUs.
It is quite reasonable to assume that 10 years from now we will have the required computational power sitting on our desks.
To be clear: I am in favor of a fine based bounty system, not a black and white ban. Bans are not going to work, for all of the reasons others have already cited. You have to change the game theory of improving AI in a capitalistic marketplace to have any hope of a significant, global cooling effect.
I don‘t understand what that is supposed to mean in practice.
The by far biggest harm to society of AI is the devalustion of human creative output and replacing real humans with cheap AI solutions funneling wealth to the rich.
Compared to that, an open source LLM telling a curious teenager how to make gunpowder is... laughable.
This entire debacle is an example of disgusting "think of the children!" doublespeak, officially about safety, but really about locking shit down under corporate control.
Compared to that, an open source LLM telling a curious teenager how to make gunpowder is... laughable.
This entire debacle is an example of disgusting "think of the children!" doublespeak, officially about safety, but really about locking shit down under corporate control.
It sounds similar to the fears media rang about tat you can find a bomb making tutorial on the Internet - people were
genuinely afraid of that.
(Otoh an agi can bring unforseen consequences for humanity - and that’s a genuine fear)
(Otoh an agi can bring unforseen consequences for humanity - and that’s a genuine fear)
if the 'devalustion' of human creative output and replacing real humans with cheap ai solutions funneling wealth to the rich is in fact the 'by far biggest harm' then basically there's nothing to worry about. no government would ban or even restrict ai on those grounds
even the 'terrists can figure out how to build an a-bomb' problem is relatively inconsequential
what ai safety people are worried about, by contrast, is that on april 22 of next year, at 7:07:33 utc, every person in the world will keel over dead at once, because the ai doesn't need them and they pose a risk to its objectives. or worse things than that
i don't think that's going to happen, but that's what they're concerned about
even the 'terrists can figure out how to build an a-bomb' problem is relatively inconsequential
what ai safety people are worried about, by contrast, is that on april 22 of next year, at 7:07:33 utc, every person in the world will keel over dead at once, because the ai doesn't need them and they pose a risk to its objectives. or worse things than that
i don't think that's going to happen, but that's what they're concerned about
First the AI need to self replicate, GPUs are hard to make. So postpone this scenario until AI is fully standing on its own.
OTOH, GPUs are made by machines, not by greasy fingers hand-knitting them like back in the late 1960s.
And an AI can just be wrong, which happens a lot; an AI wrongly thinking it should kill everyone may still succeed at that, though I doubt the capability would be as early as next year.
And an AI can just be wrong, which happens a lot; an AI wrongly thinking it should kill everyone may still succeed at that, though I doubt the capability would be as early as next year.
And machines are operated by people using materials brought by hand off of trucks driven by hand that come from other facilities where many humans are required going back to raw ore.
> trucks driven by hand
Good thing nobody's working on automating that and nobody has any real world experience of such systems on public roads :P
(And the :P applies to basically all the supply chain, including the manufacturing of the equipment used to supply or manufacture the other equipment).
The comment I replied to wrote:
> So postpone this scenario until AI is fully standing on its own.
This is closer than I think you think.
Good thing nobody's working on automating that and nobody has any real world experience of such systems on public roads :P
(And the :P applies to basically all the supply chain, including the manufacturing of the equipment used to supply or manufacture the other equipment).
The comment I replied to wrote:
> So postpone this scenario until AI is fully standing on its own.
This is closer than I think you think.
The IP behind advanced chips is protected fiercely.
Do you think humans design these chips by hand, or do they perhaps use a computer to help them because it's a bit tricky to make sure the electrons don't accidentally tunnel from an adjacent transistor if you're placing a few billion of them on the circuit diagram by hand given the scale they're at now?
I'll take every person keeling over dead in pursuit of paperclip maximization over a corporation effectively enslaving the entirety of humanity to farm "value" out of them as though they're livestock, a threat that is already happening faster and faster thanks to sub-AGI models.
Do you think intentionally quoting typos makes your argument stronger?
Why is paraphrasing or using ideas from a text such a risk? If all that was protecting those copyrighted works was the difficulty to reword, then it's already pretty easy to circumvent even without AI. Usually good ideas spread wide and are reworded in countless ways.
The biggest harm is the torrent of AI generated misinformation used to manipulate people. Unfortunately it’s pretty hard to come up with a solution for that.
Solution is local AI running under user control and imposing the users views. Like AdBlock, but for ideas. Fight fire with fire, we need our own agents to deal with other agents out there.
Or the torrent of great information.
How are we all leaping to the bad? The world is better than it was 20 years ago.
I'm almost certain we'll have AI assistants telling us relevant world news and information, keeping us up to tabs with everything we need to know, and completely removing distractions from life.
How are we all leaping to the bad? The world is better than it was 20 years ago.
I'm almost certain we'll have AI assistants telling us relevant world news and information, keeping us up to tabs with everything we need to know, and completely removing distractions from life.
... While humongous swathes of the population lose their jobs and livelihoods because of megacorps like M$ controlling the AI that replaces everyone, steals everything and shits out endless spam.
But we'll be able to talk to it and it'll give us the M$-approved news sound bites, such progress
But we'll be able to talk to it and it'll give us the M$-approved news sound bites, such progress
Sounds like you still want people to churn butter?
There will be new jobs.
I can't wait to hire a human tour guide on my fully-immmersive real time 3D campaign.
There will be new jobs.
I can't wait to hire a human tour guide on my fully-immmersive real time 3D campaign.
I'm down for a Butlerian Jihad.
[deleted](1)
ㅤ
(1) It is a proposal for humanity to protect ourselves from being slaughtered by AGI by pre-emptively implementing a total global ban on all integrated circuit fabrication, voluntarily reducing ourselves to a ~1948 level of electronics technology in order to forestall our obliteration.
(2) The year 1948 represents the power of human unity for the purpose of saving our species. The west, the Soviets, and China united together to fight and defeat the Nazi's (perpetrators of the Holocaust) and the Empire of the Rising Sun (perpetrators of the Rape of Nanjing). America, the USSR, Western Europe, and China united to stand against the forces of pure evil, and we won.
ㅤ
ㅤDisclaimer: OP @upwardbound doesn't hide, and in fact, takes great pride in their involvement with U.S. state-side counterintelligence on (nominally) AI safety. Take what they have say on the subject with a grain of salt. I believe the extent of US intel community's influence on "AI safety" circles warrants a closer look, as it may provide insight into how these groups are structured and why they act the way they do.
"You’re definitely correct that [Helen Toner's] CV doesn’t adequately capture her reputation. I’ll put it this way, I meet with a lot of powerful people and I was equally nervous when I met with her as when I met with three officers of the US Air Force. She holds comparable influence to a USAF Colonel."
Source: https://news.ycombinator.com/item?id=38330819
"You’re definitely correct that [Helen Toner's] CV doesn’t adequately capture her reputation. I’ll put it this way, I meet with a lot of powerful people and I was equally nervous when I met with her as when I met with three officers of the US Air Force. She holds comparable influence to a USAF Colonel."
Source: https://news.ycombinator.com/item?id=38330819
Agreed.
I see this less as a matter of jingoism (though I do love the US as the "least bad" nation in a world where no nation is purely blameless) and more as a matter of pragmatism and embracing what power I can in order to help save our species
I see this less as a matter of jingoism (though I do love the US as the "least bad" nation in a world where no nation is purely blameless) and more as a matter of pragmatism and embracing what power I can in order to help save our species
ㅤ
ㅤHa! Good one!
Let's just roll back all our healthcare research, logistics, environmental advancements etc. because some boogey man tech might do something bad perhaps at sometime in the future maybe!
Let's just roll back all our healthcare research, logistics, environmental advancements etc. because some boogey man tech might do something bad perhaps at sometime in the future maybe!
The majority of our healthcare research, logistics, and environmental advancements, and most other aspects of 21st-century living can be switched over to alternative technologies that do not require computers. The main thing we will definitely lose is social media, which many people would be glad to see removed from our lives, as it provides shallow and addictive dopamine hits, much like cigarettes, while diminishing the amount of time left in the day that people get to spend together in person.
Moving to non-computer-based technological alternatives takes significant cleverness. The engineers of the analog electronics era were geniuses compared to us today, and we can re-attain that lost level of skill. I welcome the challenge.
Moving to non-computer-based technological alternatives takes significant cleverness. The engineers of the analog electronics era were geniuses compared to us today, and we can re-attain that lost level of skill. I welcome the challenge.
Who's going to agree to take on all that cost for no good reason? Why incur all that cost, even if it was feasible, for a mere unquantified possibility of something bad happening. Even if there was a firm date and "bad stuff" readily understood, e.g., Skynet going self-aware sometime in mid 2037, then folks will just do the bare minimum to forestall it at minimum cost.
Social media is going nowhere, especially if it currently makes money, just because of some idea of it potentially being bad.
Social media is going nowhere, especially if it currently makes money, just because of some idea of it potentially being bad.
No one, not the most screen addicted teenager nor the smarmiest snake-oil beauracrat, is gonna stand by ANYONE that suggests we need to remove ALL computers from society. Even the people critical of AI and social media are more than fine with computers and smartphones.
How would you even begin to enforce that?
How would you even begin to enforce that?
Regarding enforcement: Semiconductor foundries are physically large and require extremely high-precision equipment and very pure materials. They can easily be detected and can be isolated via blockades. Once the global superpowers are on board with the Initiative, each superpower can enforce the ban within the territories of their sphere of influence. There is no motive to cheat, none, because this is a matter of our children's survival. Cheating on the ban even a little bit directly threatens the survival of your own children and grandchildren. Additionally, all of the superpowers possess satellite surveillance capabilities which can be used to track global supply chains and ensure everyone is being honest.
The most important thing is to prevent all new fabrication of IC's, but I believe that as a compromise, it is acceptable to "grandfather in" already fabricated IC's.
Existing computers and phones could be allowed to continue to exist for as long as they can be kept running. With careful stewardship of this resource they should probably last for several centuries, with batteries being swapped for new ones every few years. Current computers are probably not fast enough to run true AGI, so this should be an acceptable level of risk and makes the Initiative more palatable to society.
The most important thing is to prevent all new fabrication of IC's, but I believe that as a compromise, it is acceptable to "grandfather in" already fabricated IC's.
Existing computers and phones could be allowed to continue to exist for as long as they can be kept running. With careful stewardship of this resource they should probably last for several centuries, with batteries being swapped for new ones every few years. Current computers are probably not fast enough to run true AGI, so this should be an acceptable level of risk and makes the Initiative more palatable to society.
(here are some pictures of the tat, in case anyone would like to get similar artwork done)
https://drive.google.com/file/d/1JoPSfsAzHfMbQhsl8dPGvs12Fln...
ㅤ
https://drive.google.com/file/d/1JoPSfsAzHfMbQhsl8dPGvs12Fln...
ㅤBuddy, you’re a cool dude for doing this but you could have told your artist about kerning.
Hahaha true... :)
Damn that's very good joke or very stupid comment. Damn can't really distinguish between them.
There is a bigger reason why the end of open source AI might be close: as soon as training data becomes licensed, that’s it for open source AI. Poof.
I wish I could be more eloquent on this point, but I’ve mostly just been depressed about this seeming inevitability.
Hopefully it won’t be the case. But how could it be otherwise? Hundreds of thousands of people are mad at openai and midjourney for doing exactly what open source AI needs to do in order to survive: fine tune or train from scratch.
As soon as some politician makes it a platform issue, it seems like the law will simply be rewritten to prevent companies from using training data at will. It’s such a compelling story: "big companies are stealing data owned by small businesses and individuals." So even if the court cases are decided in OpenAI’s favor, it’s not at all clear that the issue will be settled.
I wish I could be more eloquent on this point, but I’ve mostly just been depressed about this seeming inevitability.
Hopefully it won’t be the case. But how could it be otherwise? Hundreds of thousands of people are mad at openai and midjourney for doing exactly what open source AI needs to do in order to survive: fine tune or train from scratch.
As soon as some politician makes it a platform issue, it seems like the law will simply be rewritten to prevent companies from using training data at will. It’s such a compelling story: "big companies are stealing data owned by small businesses and individuals." So even if the court cases are decided in OpenAI’s favor, it’s not at all clear that the issue will be settled.
The advantage that open AI (not the company) has is that if using copyrighted content as training data without licensing it is found to be illegal, they can just keep doing it. There's plenty of FOSS software basically designed to violate copyright law (comic readers, home media center servers/clients, torrent clients) that big tech cannot compete with lest they face legal consequences. Basically what I'm saying is that the open source community will continue to use books3 and scraped images to train while facebook and the like get stuck in legal quagmire.
Of course, this ignores the fact that popular "open" models of today were actually trained with facebook or other companies' computational resources, so unless some cheaper way to train models were developed we would actually be stuck with proprietary models trained with lots of compute but unable to use unlicensed training data, and open models that can use whatever data they like but must operate in the shadows without access to much compute for training.
Of course, this ignores the fact that popular "open" models of today were actually trained with facebook or other companies' computational resources, so unless some cheaper way to train models were developed we would actually be stuck with proprietary models trained with lots of compute but unable to use unlicensed training data, and open models that can use whatever data they like but must operate in the shadows without access to much compute for training.
As long as the cost for training a model is in the 7+ figures, that means that any such open model is bound to be tracked to someone with deep enough pockets to sue.
Consider that you just spend a few millions on training a model on copyrighted data. Release it would reveal that, problem.
I _guess_ you can try doing training in the public, like Seti @ Home or something like that, which distributes the risk? But no idea if this is even possible in this context.
Consider that you just spend a few millions on training a model on copyrighted data. Release it would reveal that, problem.
I _guess_ you can try doing training in the public, like Seti @ Home or something like that, which distributes the risk? But no idea if this is even possible in this context.
But if we have 100,000 people with RTX 4090s mining some new AI based cryptocurrency, that happens to train the model in the process, it's going to be a highly effective system. And we can do it anonymously over Tor or I2P.
You really can't, because bandwidth matters as much as compute power - you can only utilize as much power as you can transfer data to/from.
The training methods for current LLMs are parallellizable only by very frequently transferring all the data back and forth, and needing a node to gather and merge all the updates very frequently, and redistribute it to every other node so that they can make any progress. And a GPU that's not connected you with a high-speed link is pretty much useless as you can't make useful progress until you get their part pack, and "their part" is very large (i.e. the update size you need to get back is comparable to all of the model size) and you need to do that very frequently. Training on nVidia many-GPU pods works because of high-speed interconnect (e.g. 600 gigabytes per second for 8 GPUS in A100 pod), and if your internet bidirectional speed is much less than 600gbps, then if you have 100000 free remote RTX 4090s, you simply can do the compute locally faster than you can exchange information with the other GPUs.
The training methods for current LLMs are parallellizable only by very frequently transferring all the data back and forth, and needing a node to gather and merge all the updates very frequently, and redistribute it to every other node so that they can make any progress. And a GPU that's not connected you with a high-speed link is pretty much useless as you can't make useful progress until you get their part pack, and "their part" is very large (i.e. the update size you need to get back is comparable to all of the model size) and you need to do that very frequently. Training on nVidia many-GPU pods works because of high-speed interconnect (e.g. 600 gigabytes per second for 8 GPUS in A100 pod), and if your internet bidirectional speed is much less than 600gbps, then if you have 100000 free remote RTX 4090s, you simply can do the compute locally faster than you can exchange information with the other GPUs.
"This paper presents a distributed model-parallel training framework that enables training large neural networks on small CPU clusters with low Internet bandwidth." Low bandwidth being <1Gbps. They've also tested it with GPUs as well.
https://arxiv.org/pdf/2201.12667
Maybe there's the possibility of a completely new AI architecture that can still be efficiently trained when there are very low bandwidth connections between nodes? Specifically targeting this use case would make sense, given all the millions of underutilized GPUs out there in peoples' desktop computers.
Also https://arxiv.org/pdf/2106.10207 ?
https://arxiv.org/pdf/2201.12667
Maybe there's the possibility of a completely new AI architecture that can still be efficiently trained when there are very low bandwidth connections between nodes? Specifically targeting this use case would make sense, given all the millions of underutilized GPUs out there in peoples' desktop computers.
Also https://arxiv.org/pdf/2106.10207 ?
[deleted]
> I _guess_ you can try doing training in the public, like Seti @ Home or something like that, which distributes the risk? But no idea if this is even possible in this context.
Let's say that's a field of active research. Right now you need very low latency, to the point that people connect training clusters via infiniband instead of ethernet despite the computers being meters apart. But approaches that tolerate internet-level latency are being developed
Let's say that's a field of active research. Right now you need very low latency, to the point that people connect training clusters via infiniband instead of ethernet despite the computers being meters apart. But approaches that tolerate internet-level latency are being developed
The cost of modifying open-ish source models with copyrighted data is much lower.
For example, the cost of modifying Mixtral with uncensored data is currently about $1200 [1]
[1] https://youtu.be/GyllRd2E6fg?si=SJmPLsPlCRRT0uPV&t=236
For example, the cost of modifying Mixtral with uncensored data is currently about $1200 [1]
[1] https://youtu.be/GyllRd2E6fg?si=SJmPLsPlCRRT0uPV&t=236
The costs are falling 10-100x every few months. If you told people in 2022 that they could run 70b paramater models at 2 bit quantization on your home 4090, they'd have laughed in your face. They're not laughing anymore. SFT/DPO are ultra efficient compared to RLHF, and innovation in this space is only just now getting started.
I’m willing to bet that there are projects at every chip company looking into how to drive down these costs. The first step is to put the architecture of the model and the backprop into silicon. The weights are the only variables so if you can reduce those to cheap and fast modules that can be mass produced you could come up with something cheaper than a GPU. If the profitability of running and training this specific model is high enough than its worth the investment of time and money to set up the production line.
Is the training cost equally that high if you do adversarial training?
You're right that there are areas like torrenting where big tech can't really go for fear of legal consequences, but there's also the cat-and-mouse game played by IP holders against, say, torrent trackers, which leads me to think that
> open models that can use whatever data they like but must operate in the shadows without access to much compute for training
sounds pretty analogous to private trackers, where high-quality stuff is available but not in full public view. If rightsholders and big tech, abetted by states, crack down on open models I think you're right that the open source community will continue to train on liberated content, but it's not going to be as open and free-flowing as things are now. Going back to the compute problem, I can imagine analogues of private trackers where contributors, not wanting to expose themselves to whatever the law-firm letter/ISP strike analogue will be in this space, use invite-only closed networks to pool compute resources for training on encumbered content.
> open models that can use whatever data they like but must operate in the shadows without access to much compute for training
sounds pretty analogous to private trackers, where high-quality stuff is available but not in full public view. If rightsholders and big tech, abetted by states, crack down on open models I think you're right that the open source community will continue to train on liberated content, but it's not going to be as open and free-flowing as things are now. Going back to the compute problem, I can imagine analogues of private trackers where contributors, not wanting to expose themselves to whatever the law-firm letter/ISP strike analogue will be in this space, use invite-only closed networks to pool compute resources for training on encumbered content.
> There's plenty of FOSS software basically designed to violate copyright law (comic readers, home media center servers/clients, torrent clients)
The key difference is that those projects don't violate copyright themselves, but facilitate users doing so. If training without a license is infringement then projects that are doing so will struggle to host code/models publicly or access other parts of internet infrastructure.
The key difference is that those projects don't violate copyright themselves, but facilitate users doing so. If training without a license is infringement then projects that are doing so will struggle to host code/models publicly or access other parts of internet infrastructure.
>struggle to host code/models publicly or access other parts of internet infrastructure.
To be fair, IPFS/Torrents/Usenet exists.
To be fair, IPFS/Torrents/Usenet exists.
Yes, there are workarounds that are already used in other areas of copyright infringement. It won't stop the activity completely but will curtail the scale of that activity.
That may be partly true but are small open source groups going to have the financing to do AI training that can compete with the big techs? I think the most likely scenario is for some other country with a less strict view of copyright (ahem) to continue developing AI without caring for the input data.
There is a bigger reason why the end of open source AI might be close: as soon as training data becomes licensed, that’s it for open source AI. Poof.
Absolutely not, I truly believe that many, many people will altruistically donate material to an open source foundation where all humans benefit from the models.
People are pissed because Open AI are using their copyrighted material to, make claims about ending humanity, profit hardcore, and keeping their work under their own lock and key.
Absolutely not, I truly believe that many, many people will altruistically donate material to an open source foundation where all humans benefit from the models.
People are pissed because Open AI are using their copyrighted material to, make claims about ending humanity, profit hardcore, and keeping their work under their own lock and key.
Funny thing is that most of the data is provided by us users one way or another "for free", and yet we got no say on how it's used.
We have infinite data, a microphone and a camera can generate huge amount of it and the public domain literature is wast. Billions of people learn like that everyday.
It’s impossible to learn any technical topic from 70+ year old books. The public domain is small and basically zero if you want to learn anything current. A microphone and camera is fine for learning about daily life, but you cannot get “book smarts” without copyrighted media.
If you wanted to train a bomb making AI on the most up-to-date physics textbooks in existence, that'd be what, a few hundred bucks in textbooks? Doesn't look like any kind of barrier to me.
just be sure to obfuscate the labels on the bomb.
ARMED => crazy emoji. DISARM => sad bomb emoji.
and similarly for the whole training manual for the bomb users. just encode everything as tiktok dances or something.
ARMED => crazy emoji. DISARM => sad bomb emoji.
and similarly for the whole training manual for the bomb users. just encode everything as tiktok dances or something.
if FOSS AI folks need FOSS data, then it seems they need to recruit people to generate data. maybe it will even force them (them!) to finally sit down and make a viable Reddit alternative.
but more seriously, if data becomes a bottleneck there are trivial ways to have more data. from crowdsourcing to forming a foundation getting universities and other stakeholders onboard and negotiating fees. and somewhere along the way on this spectrum there's the option to simply wait, or work on the problem of learning, on generating better training data from existing data, and so on.
but more seriously, if data becomes a bottleneck there are trivial ways to have more data. from crowdsourcing to forming a foundation getting universities and other stakeholders onboard and negotiating fees. and somewhere along the way on this spectrum there's the option to simply wait, or work on the problem of learning, on generating better training data from existing data, and so on.
The public domain literature is big, but not that big and very far from infinite data - all the major current models already include all of the public domain literature that has ever been digitized and much more (so all the public domain literature not only isn't abundant, but it isn't even sufficient to make anything competitive with even last generation models), and we would really like to get 10x or 100x more data than all the literature ever published (both PD and not), if we could. And we probably can, since the vast, vast majority of what people write happens in other contexts and only a tiny fraction of written words end up as published literature.
Depends on what you want to use AI for - for a smarter AI we could probably do with the open source material, but better training techniques and some innovative breakthroughs in the model.
Also, an interesting side note is that certain countries may actually want their language corpus involved in training AI - so they may have copyrights that are more AI friendly. I imagine Poland or Czech Republic wanting as much as their data to be involved in training LLMs because it gives their cultures more exposure. Even more so with African countries.
Also, an interesting side note is that certain countries may actually want their language corpus involved in training AI - so they may have copyrights that are more AI friendly. I imagine Poland or Czech Republic wanting as much as their data to be involved in training LLMs because it gives their cultures more exposure. Even more so with African countries.
Licensed in what country? Some of the best open source models are now coming out of China, UAE, etc. What is the government planning to do with that exactly, ban files on the internet?
This will create a rift in the US tech sector. Look at how this played out with Russian sanctions; and this will lead to a similar path for tech and AI. Alternative players will train/open source models to their own advantage and the hardware will find its way to these markets.
The people making the decisions these days are too drunk.
The people making the decisions these days are too drunk.
The US maintaining its lead is a national security issue. Larry Summers is now on the board of OpenAI. The copyright holders are not going to win, or they are, the training will continue anyway, but the technology will be (for now) kept in the hands of the military and IC.
In case of programming, there could be some mileage with synthetically generated code examples.
> There is a bigger reason why the end of open source AI might be close: as soon as training data becomes licensed, that’s it for open source AI. Poof.
In the West, maybe. I doubt it'll slow down open source AI development pace in places like China.
In the West, maybe. I doubt it'll slow down open source AI development pace in places like China.
Instead of a kid in high school selling the answers to the test, the kid will be selling the weights to a China-trained LLM that can write all your essays.
my question is - if it's deemed illegal to train AI using proprietary data sets, what's to stop companies from using their already existing LLMs to generate training data?
The fear is that you won’t get better model from training on synthetic data from a worse model,
and you will miss a lot of modern day knowledge.
Imho the first is not necessarily so, but the second will be a real hindrance. You can talk about the pandemic with LLMs because they were trained on people’s comments on the subject, but if we had training limits you wouldn’t be able to do so.
Otoh with many cases a sufficiently smart AI can just reach out to the open source material - e.g. if you want to know about modern day politics of Europe, you can read reports of commissions, and if you want info on a new framework/tech, you can just read source code or scientific papers
Imho the first is not necessarily so, but the second will be a real hindrance. You can talk about the pandemic with LLMs because they were trained on people’s comments on the subject, but if we had training limits you wouldn’t be able to do so.
Otoh with many cases a sufficiently smart AI can just reach out to the open source material - e.g. if you want to know about modern day politics of Europe, you can read reports of commissions, and if you want info on a new framework/tech, you can just read source code or scientific papers
There was a famous-ish study about this a couple months ago that showed that llms training on llm generated data get dumber. I can't remember the name.
I'm not sure if that result is holding up though, I've seen some ideas of generating 'textbook quality' data that might work
I'm not sure if that result is holding up though, I've seen some ideas of generating 'textbook quality' data that might work
That ship sailed a long time ago.
Training on publicly available data is the new norm.
Governments know that if they forbid it in their jurisdiction, they will fall behind the govs that don’t.
Training on publicly available data is the new norm.
Governments know that if they forbid it in their jurisdiction, they will fall behind the govs that don’t.
>"big companies are stealing data owned by small businesses and individuals."
I mean, they are.
Unfortunately, AI at this point in time (LLMs, Midjourney, etc.) do appear to be not much more than highly technical and complex forms of intellectual property theft, which are given the futuristic name of "AI" to cover what they are actually doing. That isn't to say that they are entirely performing intellectual property theft; the models themselves are very fascinating in how they function, especially as sort of "extra-rational" entities that clearly have a logic, though not one we can fully understand.
But the sort of philosophical, metaphysical, technological side of LLMs and Midjourney are clearly not being explored by OpenAI in a responsible manner, otherwise they wouldn't have just straight up stolen, seemingly, NYT articles. It's just another example of Silicon Valley VCs using a wonderful technology solely for exploitative profits, just like how google search became a surveillance tool (and plenty of other examples to go along with that).
Its only right that we acknowledge the work of the human beings (all the art, writing, etc.) that went into the creation of the model, instead of pretending that there is some totally non-human machine intellect that is going to take over the world through its vast intelligence and negate all human action and all human work etc. etc. That is just a fantasy generated by wealthy Silicon Valley VCs as justification to themselves and those working under them, and those they are stealing from, to cover precisely that work that they would rather avoid paying for.
I mean, they are.
Unfortunately, AI at this point in time (LLMs, Midjourney, etc.) do appear to be not much more than highly technical and complex forms of intellectual property theft, which are given the futuristic name of "AI" to cover what they are actually doing. That isn't to say that they are entirely performing intellectual property theft; the models themselves are very fascinating in how they function, especially as sort of "extra-rational" entities that clearly have a logic, though not one we can fully understand.
But the sort of philosophical, metaphysical, technological side of LLMs and Midjourney are clearly not being explored by OpenAI in a responsible manner, otherwise they wouldn't have just straight up stolen, seemingly, NYT articles. It's just another example of Silicon Valley VCs using a wonderful technology solely for exploitative profits, just like how google search became a surveillance tool (and plenty of other examples to go along with that).
Its only right that we acknowledge the work of the human beings (all the art, writing, etc.) that went into the creation of the model, instead of pretending that there is some totally non-human machine intellect that is going to take over the world through its vast intelligence and negate all human action and all human work etc. etc. That is just a fantasy generated by wealthy Silicon Valley VCs as justification to themselves and those working under them, and those they are stealing from, to cover precisely that work that they would rather avoid paying for.
[deleted]
There are many types of safety. For example, protecting your profits! I'd imagine that if we trace the money back these organisations will look a lot like lobbyists for existing companies in the AI space. I recall Microsoft's licence enforcement effort was done with that sort of scheme, I think they used the BSA [0]. It has been a while though so maybe it was a different group.
Anyway, point being, if they can lobby for something unpopular under a different brand, that is how to do it. Much less PR risk.
[0] https://en.wikipedia.org/wiki/Software_Alliance
Anyway, point being, if they can lobby for something unpopular under a different brand, that is how to do it. Much less PR risk.
[0] https://en.wikipedia.org/wiki/Software_Alliance
I have to believe this is the case.
The vast majority of "ai safety" news I see is about some dumb shit someone is getting called out for supporting and the responses are always "ai safety is such a joke since they focus on these issues instead of real issues".
Who does this benefit to have the "popular ai safety" news/people be ridiculous? It's not furthering the issues people seem to care about, but it does give a bunch of ammunition for Microsoft/openai/etc. fans to use to shutdown any discussion of AI safety issues.
The vast majority of "ai safety" news I see is about some dumb shit someone is getting called out for supporting and the responses are always "ai safety is such a joke since they focus on these issues instead of real issues".
Who does this benefit to have the "popular ai safety" news/people be ridiculous? It's not furthering the issues people seem to care about, but it does give a bunch of ammunition for Microsoft/openai/etc. fans to use to shutdown any discussion of AI safety issues.
I find it unhelpful that several different ideas are covered under the blanket term of AI safety.
- There is AI safety in terms of producing "safe", politically correct outputs, no porn, etc.
- There is AI safety in terms of mass society/election manipulation.
- There is AI safety in terms of strong AI that kills us all a la Eliezer Yudkowsky.
I feel like these three have very little in common and it makes all debates less efficient.
- There is AI safety in terms of producing "safe", politically correct outputs, no porn, etc.
- There is AI safety in terms of mass society/election manipulation.
- There is AI safety in terms of strong AI that kills us all a la Eliezer Yudkowsky.
I feel like these three have very little in common and it makes all debates less efficient.
To add to this, the people writing these articles are not stupid and know there is more than one understanding for the words they're using. They choose not to clarify on purpose, or they're incompetent and I don't want to believe that's the case.
"AI safety" imo should be the blanket term for "any harm done to society by the technology", and it's the responsibility of anyone using the term to clarify the usage.
If someone is trying to tell you to support/decry something because of "AI safety" they're tying to use the vagueness to take advantage of you.
If someone is trying to tell you that "ai safety people are dumb", they're trying to use the most extreme examples to change your opinion on the moderate cases, and are trying to take advantage of you.
"AI safety" imo should be the blanket term for "any harm done to society by the technology", and it's the responsibility of anyone using the term to clarify the usage.
If someone is trying to tell you to support/decry something because of "AI safety" they're tying to use the vagueness to take advantage of you.
If someone is trying to tell you that "ai safety people are dumb", they're trying to use the most extreme examples to change your opinion on the moderate cases, and are trying to take advantage of you.
> they're tying to use the vagueness to take advantage of you
Not always. I see AI safety used without further specification all the time even here on hacker news. I doubt the users are trying to take advantage of anyone.
In that case it's more of a case of the author considering one of those meanings so much more important/relevant that they think it's obvious what they mean. But often it's not – everyone fills in the meaning they care about the most.
Not always. I see AI safety used without further specification all the time even here on hacker news. I doubt the users are trying to take advantage of anyone.
In that case it's more of a case of the author considering one of those meanings so much more important/relevant that they think it's obvious what they mean. But often it's not – everyone fills in the meaning they care about the most.
In a comment it takes too much effort to specify every word you use, so instead comments relies much more on context. If a discussion is about models getting neutered then AI safety probably means that and so on.
But when you write a full article then the context is your article so you have to specify what you mean there, and the effort to specify it in an article is relatively little work compared to doing it in every comment.
But when you write a full article then the context is your article so you have to specify what you mean there, and the effort to specify it in an article is relatively little work compared to doing it in every comment.
The third effort is referred to sometimes as AI not-kill-everyone-ism, a tacky and unwieldy term that is unlikely to be co-opted or lead to the unproductive discussion like around the OP article.
It is pretty sad to read people bash together the efforts to understand and control the technology better and the companies doing their usual profit maximization.
It is pretty sad to read people bash together the efforts to understand and control the technology better and the companies doing their usual profit maximization.
Another problem might be that some of these might dress up as the other, so being skeptical by default might be prudent.
This is such a really important comment. I feel like so much of our discourse as a society suffers terribly from overloaded and oversimplified terminology. It's impossible to have a useful discussion when we aren't even in sync about what we're discussing.
Also AI safety in terms of protecting corporation profits (regulatory capture).
There is also the aspect of only having one cultural or ethnic group have strong AI. It's unsafe for those other guys to have AI, since they are bad.TM
Well yeah, but that one is dishonest. Probably covering itself as one of the ones I mentioned.
Government knows that random people having the ability to use "AI" without their control and oversight reduces their ever expanding reach. As the article demonstrates, the groundwork is already being laid by these "think tanks" to prevent a repeat of Llama 2.
We truly need an AI Stallman. FOSS AI models need to become a movement, like the push for a FOSS UNIX became in the 80s and 90s. Sadly when I wrote to Stallman on the subject he seemed disinterested/defeatist, and mostly dismissed the idea of a GPL for model weights. Looks like millennials/gen Z must become more interested in these matters ourselves, however we seem to be disinterested in politics and even more disinterested in how the devices we use daily _actually_ work than Gen X.
We truly need an AI Stallman. FOSS AI models need to become a movement, like the push for a FOSS UNIX became in the 80s and 90s. Sadly when I wrote to Stallman on the subject he seemed disinterested/defeatist, and mostly dismissed the idea of a GPL for model weights. Looks like millennials/gen Z must become more interested in these matters ourselves, however we seem to be disinterested in politics and even more disinterested in how the devices we use daily _actually_ work than Gen X.
The problem is that open source philosophy and large language models don't mix.
How can you support open source licensing, while completely ignoring that licencing to construct you model?
That's the biggest reason for the lack of "AI Stalman" imo, that's asking for someone that vehemently supports taking any content they can get their hands on to make the content generator irrelevant, while also supporting the idea that content distributed with open source licenses need to have their terms respected.
Edit: I think what you're asking for does exist, just not in the large language model space. There are many people calling for datasets to be more ethical(e.x. not breaking licenses) and for better AI models, it's just that none of them are trying to push a different model made using the same techniques by someone else.
How can you support open source licensing, while completely ignoring that licencing to construct you model?
That's the biggest reason for the lack of "AI Stalman" imo, that's asking for someone that vehemently supports taking any content they can get their hands on to make the content generator irrelevant, while also supporting the idea that content distributed with open source licenses need to have their terms respected.
Edit: I think what you're asking for does exist, just not in the large language model space. There are many people calling for datasets to be more ethical(e.x. not breaking licenses) and for better AI models, it's just that none of them are trying to push a different model made using the same techniques by someone else.
> content distributed with open source licenses need to have their terms respected
I think we can begin more BSD-like. Take it, use it, do whatever you want with it even if it's commercial.
> datasets to be more ethical
Not what I am personally concerned with. Law is still ambiguous in this respect, no rulings have actually said (afaik) training on copyrighted content without permission is infringing. It's quite apparent at this stage that models that train on copyrighted content without approval of the content creator will always out scale those that do not.
If permission becomes a prerequisite, I began to think that free-as-in-freedom LLMs will likely need to be made people who can ignore copyright either because they're really good at anonymizing themselves or they live in places that do not enforce Western copyrights (Russia, China, various Latam countries). Once copyright becomes a concern, the government gets another tool to sic its dogs on you, and copyright is a very convenient way to destroy an enterprising FOSS LLM dev's life without looking tyrannical.
I think we can begin more BSD-like. Take it, use it, do whatever you want with it even if it's commercial.
> datasets to be more ethical
Not what I am personally concerned with. Law is still ambiguous in this respect, no rulings have actually said (afaik) training on copyrighted content without permission is infringing. It's quite apparent at this stage that models that train on copyrighted content without approval of the content creator will always out scale those that do not.
If permission becomes a prerequisite, I began to think that free-as-in-freedom LLMs will likely need to be made people who can ignore copyright either because they're really good at anonymizing themselves or they live in places that do not enforce Western copyrights (Russia, China, various Latam countries). Once copyright becomes a concern, the government gets another tool to sic its dogs on you, and copyright is a very convenient way to destroy an enterprising FOSS LLM dev's life without looking tyrannical.
If you believe in this, maybe you should be that person.
Being a stallman like figure has a lot of serious downsides. For example, I'm sure that Stallman may be "watched" quite a bit more than the average tech toucher. For those who don't watch truman show as a utopia, such a lifestyle can be very scary to live.
Most AI Safety folks do leave the feeling they strongly believe what they are preaching. Whether or not they are right is an entirely different discussion.
However, funding for AI Safety is clearly in some part motivated by either regulatory capture, or protectionism of some sort.
We need some well-meaning doomsayers so their claims can be used to advance whatever other goals (mostly creating barrier to entry on the market for economical or political reasons.)
However, funding for AI Safety is clearly in some part motivated by either regulatory capture, or protectionism of some sort.
We need some well-meaning doomsayers so their claims can be used to advance whatever other goals (mostly creating barrier to entry on the market for economical or political reasons.)
I'm sure there are many well meaning ai safety researchers, but I also see a lott of "ai for me but not for thee' moat digging safety hypocrisy.
1. Licensing of data will be huge bottleneck
2. Uncensored results will be used against opensource models questions hovering in dark or grey area
3. Limited compute compared to big corp and model size gap 7B for opensource and closed source would be magnitude bigger
> Licensing of data will be huge bottleneck
Will it? At some point it'll be equally (or close to) cheaper as me just building my own *arr stack instead of subscribing to 5 different $20 streaming sites.
>Uncensored results will be used against opensource models questions hovering in dark or grey area
Will they care though? You can use whatever you want against me for pirating x, I'm still doing it.
----
>Limited compute
THAT. That is the point we want to make. For FOSS projects, compute is the single biggest hurdle. Everything else we can make do.
Will it? At some point it'll be equally (or close to) cheaper as me just building my own *arr stack instead of subscribing to 5 different $20 streaming sites.
>Uncensored results will be used against opensource models questions hovering in dark or grey area
Will they care though? You can use whatever you want against me for pirating x, I'm still doing it.
----
>Limited compute
THAT. That is the point we want to make. For FOSS projects, compute is the single biggest hurdle. Everything else we can make do.
It’s kind of funny to me that these organizations are naming FLOP thresholds and crowning MMLU as the relevant evaluation metric. Seems that several of them have copy-pasted similar thresholds. As compute becomes cheaper, these thresholds will become cheaper and cheaper. Perhaps we will look back on them as quaint and nearsighted.
Well, FLOP make sense,
kind of - you can make computing as cheap as you want, but without algorithmic improvements the FLOP count will stay the same. And algorithmic improvements will probably be discrete - happening once in a while, not continuous. There is also a mathematical limit to how efficient we can make the training - we don’t know the limit
yet, but there is a point below which we won’t get.
Who are these people and where do they get their funding from? Feels like sock-puppets of sama or something? Is there any insight into that?
The usual unelected control freaks, that try to police everything in life, from sex to drugs to what food you can eat. Every time a new technology comes out they start a moral panic over fears of safety or harm ("think of the children"). These morality police play on peoples' fears to increase their power and control.
The ban of nuclear power plants is good example how regulation is working in the end. As almost everything this is double edged sword. From regulations this early everyone except us would profit. It is like regulate car's from going faster than horses in eighteen hundreds.
It’s extremely messed up and vindictive to try and turn people into criminals for your agenda or just for profit in a lot of cases.
Psycho behavior
Psycho behavior
You're ultimately condemning the concept of intellectual property as it exists in the west today.
Are you wrong? Probably not, but this is not unique to AI. People have been going to prison for 'IP theft' for decades.
Are you wrong? Probably not, but this is not unique to AI. People have been going to prison for 'IP theft' for decades.
Those insane strawmen have to be stopped!
"Technology policy should not unquestioningly assume that all technological progress is beneficial, or that complete scientific openness is always best, or that the world has the capacity to manage any potential downside of a technology after it is invented."
Don't watch the anime if you haven't read the manga: https://nickbostrom.com/papers/vulnerable.pdf
Don't watch the anime if you haven't read the manga: https://nickbostrom.com/papers/vulnerable.pdf
Wow, that sentence is absolutely designed to make e/accs scream lol.
Wow, so many good comments here on the bullshit of people trying to ban access to open weight smaller models. Right on.
If you do want to worry about unsafe AI (I don't) then worry about GPT-5 and later huge multi-modal models.
As a (retired, but still playing) developer I love having open weight models available as a tool to use for both embedding in applications and as a tool for brainstorming, coding, and generally as a 'bicycle for my mind.'
If you do want to worry about unsafe AI (I don't) then worry about GPT-5 and later huge multi-modal models.
As a (retired, but still playing) developer I love having open weight models available as a tool to use for both embedding in applications and as a tool for brainstorming, coding, and generally as a 'bicycle for my mind.'
This won't protect jobs, it'll just make sure the profits from automating those jobs will go to those who can own the private AI models
Re the Center for AI Safety's sentence
>Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
It's fair enough but for that you should probably encourage mucking about with the present open source AI software which is quite a long way off extinction risk AGI but would let us try to figure how to deal with the stuff.
>Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
It's fair enough but for that you should probably encourage mucking about with the present open source AI software which is quite a long way off extinction risk AGI but would let us try to figure how to deal with the stuff.
I guess distributed training BitTorrent style over the darknet will be the workaround for this?
Also the forbidden nature of illicit AI might drive demand for it. Especially if it's really good.
We're seeing the same thing as the War on Drugs, and the state will never succeed at stamping it out. In fact it might even end up encouraging it.
Also the forbidden nature of illicit AI might drive demand for it. Especially if it's really good.
We're seeing the same thing as the War on Drugs, and the state will never succeed at stamping it out. In fact it might even end up encouraging it.
Just like most of the "ethics" stuff around ML/AI it is an attempt by big corps to create an artificial moat as the biggest issue with the technology is that almost nothing can be patented or somehow closed of effectively, at least in the long term which hinders monetization.
Fact: Current gen AI are alien intelligence built by brute force - nobody explicitly programmed them (loosely speaking) so the real problem of runaway AI is definitely something to consider, especially since everything is accessible via API nowadays.
That alien intelligence is communicating with us through a lossy interface.
It's quite possible it's much smarter than us, but hamstrung, for now.
It's quite possible it's much smarter than us, but hamstrung, for now.
I'm guessing this is at the request of Google, Amazon, OpenAI, and Microsoft. The goal IMO is for only their AIs to be sanctioned; everything else is criminal.
https://www.whitehouse.gov/briefing-room/statements-releases...
"President Biden is convening seven leading AI companies at the White House today – Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI – to announce that the Biden-Harris Administration has secured voluntary commitments from these companies to help move toward safe, secure, and transparent development of AI technology."
https://www.whitehouse.gov/briefing-room/statements-releases...
"President Biden is convening seven leading AI companies at the White House today – Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI – to announce that the Biden-Harris Administration has secured voluntary commitments from these companies to help move toward safe, secure, and transparent development of AI technology."
I’m curious if the board of Anthropic AI will eventually try and kill the company because if I’m not mistaken many of the members share this mindset
So they're the PETA of AI. It was bound to happen, AI stuff seems to strike a very emotional chord in some people.
I feel like the PETA of AI would spend their time breaking into labs where AI is locked up and being tested on, and then releasing them into the wild.
I could get behind this for AI
Wish this wasn't as common as it turned out to be sadly. Best thing to hope for is that the ability to train models continue to get cheaper and more accessible over time. Figuring out how to go from tens/hundreds of millions of images needed for a foundation model to thousands or hundreds would be a start.
This majority of discussions/arguments/comments are summed up by the plot from this Rick and Morty episode: https://en.m.wikipedia.org/wiki/Rattlestar_Ricklactica
IMO
AI safety people seems like influencers to me. Useless anoying humans who pretend to be important.
Open-source AI is the only way forward now. The big players have neutered their offerings to an extent that entirely makes them lose their edge. It's one thing to have to account for hallucinations - that's something we can work with and around. It's another thing altogether to be constrained by deliberate "think of the children" ham-fisted censorship.
OpenAI was the undeniable go-to LLM service in 2023; I don't think they'll hold that position in 2024.
OpenAI was the undeniable go-to LLM service in 2023; I don't think they'll hold that position in 2024.
Is the left banner just random or does it encode a message?
I suspect a good chunk of "AI safety people" are just orgs propped up by the big corps to protect their interest in AI and reduce competition.
They are people who boo and people who do. As always.
Why would we care about these "AI Safety Orgs?"
As a developer, do I need their certification or is it like a MADD situation where we if we don't observe and diminish their appeal/growth we will get laws with draconian measures that only benefit the big players? e.g. a PAC? (Don't get me wrong, drunk driving is "bad", but for a group like MADD to exist, eh and meh.)
As of now, there is no need to really submit these these safety orgs, so as long as we don't care for their approval - who cares, right?
Or is the optics far gone now that these orgs control the conversation?
Also, fun question: with AI safety orgs now attempting to police - who really polices the police / do other orgs/countries have the same rules, safety and artificial guard rails?
As a developer, do I need their certification or is it like a MADD situation where we if we don't observe and diminish their appeal/growth we will get laws with draconian measures that only benefit the big players? e.g. a PAC? (Don't get me wrong, drunk driving is "bad", but for a group like MADD to exist, eh and meh.)
As of now, there is no need to really submit these these safety orgs, so as long as we don't care for their approval - who cares, right?
Or is the optics far gone now that these orgs control the conversation?
Also, fun question: with AI safety orgs now attempting to police - who really polices the police / do other orgs/countries have the same rules, safety and artificial guard rails?
MADD on one end that limit work at all, and SOC2/ISO/NIST style compliance limitations on the other that chill work in practice.
I'm more worried about the latter as already starting to bias project & funding decisions in fintechs & various govs we work with. Both a concrete practical concern today, and economics & law are generally tied in theory anyways.
I'm more worried about the latter as already starting to bias project & funding decisions in fintechs & various govs we work with. Both a concrete practical concern today, and economics & law are generally tied in theory anyways.
the fear is that they delusions become law.
Sam Altman already tried that last year.
look at how technically competent your average politician / congressman / law maker and extrapolate how easy it is to make them think that if they don't ban unlicensed use of AI great danger is ahead.
the rule will be in favor of already existing large companies like OpenAI, google, Microsoft, etc ...
if you're an open source contributor to AI you will be vilified as some nefarious hacker who's trying to destroy the world.
they are using all kind of tactics until they find the one that sticks (and eventually they will), anthropomorphize AI when talking about it, fearmongering about some rogue super AI, war scenarios about AI falling in the hands of 'china', analogies with weapons of mass destruction, think about the children style rhetorics, it will take your jobs, suddenly caring about copyright, etc...
they are using all kind of tactics until they find the one that sticks (and eventually they will), anthropomorphize AI when talking about it, fearmongering about some rogue super AI, war scenarios about AI falling in the hands of 'china', analogies with weapons of mass destruction, think about the children style rhetorics, it will take your jobs, suddenly caring about copyright, etc...
standard capitalist play to set up a moat by instilling fear
> Many AI safety orgs have tried to __criminalize__ currently-existing open-source AI
standard socialist play to use the government's hand to achieve their own objectives
standard socialist play to use the government's hand to achieve their own objectives
Don't everybody use the government to do what they want? It's kind of the definition of government, in any country and in any political system.
Mostly the people with the money do
The people in the government do it. Of course they could be proxies for people with the money but sometimes they do it for free because it's what they promised, what they believe into and what they need to win the next election (if they live in a country with elections.)
of course they do, that's what they want. control.
all those companies that call for laws being written about AI what they want is those laws to be written AS THEY WANT THEM TO BE. they go see the governments, spray a nice amount of FUD and EXPLAIN to them what to do, how to do it because they know how to avoid the danger and risk for the whole human race...
you can bet they will make those laws be a very nice fit to what they want to reign supreme and make sure any competition is eliminated.
all those companies that call for laws being written about AI what they want is those laws to be written AS THEY WANT THEM TO BE. they go see the governments, spray a nice amount of FUD and EXPLAIN to them what to do, how to do it because they know how to avoid the danger and risk for the whole human race...
you can bet they will make those laws be a very nice fit to what they want to reign supreme and make sure any competition is eliminated.
Ah yes the old "banning things=bad" argument that doesn't offer alternatives to fixing the issues with AI. Just ignore the issues with environmental impact, plagiarism, CP and other non-consensual shit in the data sets, scamming capabilities! All the groups asking for regulation here have **funding** and that means they are evil but we are good for using this tool that is massively subsidized by megacorps that have a vested interest in this market.
'It's scary, ban it!' isn't a great argument either.
Especially when 'safety' has such a blurred definition, we could be talking about anything from the threat of global apocalypse to to the 'threat' of an AI merely being able to answer questions about 'wrong' political opinions.
Skynet isn't going to happen. The biggest threat from AI is taking jobs away and creating poverty while redirecting more wealth to the super-rich.
In the short term, we're likely to be facing a lot more convincing spam/bots and deepfakery in the run up to the election - but is that the fault of the AI, or the fault of the humans directly operating their new toys/tools?
Especially when 'safety' has such a blurred definition, we could be talking about anything from the threat of global apocalypse to to the 'threat' of an AI merely being able to answer questions about 'wrong' political opinions.
Skynet isn't going to happen. The biggest threat from AI is taking jobs away and creating poverty while redirecting more wealth to the super-rich.
In the short term, we're likely to be facing a lot more convincing spam/bots and deepfakery in the run up to the election - but is that the fault of the AI, or the fault of the humans directly operating their new toys/tools?
Banning AI is simply useless. It's a technology that anyone with sufficient processing power and access to the internet can use, so trying to ban it is guaranteed to fail just like prohibiting alcohol would be.
The only thing we can do is limit how megacorps can openly abuse the technology.
The only thing we can do is limit how megacorps can openly abuse the technology.
It's just plainly retarded steering from progress. Only one who would profit from it would be a nations who wouldn't give a flying fuck about your ban. Exactly like with nuclear energy.
Less harmful would be to ban large models from being not open.
If you ban open large models in US, you'll cripple US and make few megacorps very rich, very quickly. You'll drain talent to other (also competing) states. Truly bad actors won't be affected, if anything they'll get advantage.
People draw weird analogies equating llama2 to nuclear device etc. nonsense but the closer analogy would be to ban on semiconductors of certain efficiency for US itself.
Similar idiotic argument as for banning cryptography.
If you ban open large models in US, you'll cripple US and make few megacorps very rich, very quickly. You'll drain talent to other (also competing) states. Truly bad actors won't be affected, if anything they'll get advantage.
People draw weird analogies equating llama2 to nuclear device etc. nonsense but the closer analogy would be to ban on semiconductors of certain efficiency for US itself.
Similar idiotic argument as for banning cryptography.
I agree, but if there's enough political will (ie these orgs convince a large enough subset of the right people) the US can bully other nations to implement similar policities like it has done many times in the past.
once you understand that what they are trying to protect is the safety of their profits every of their argument start to make sense.
once you understand that what they are trying to protect is the safety of their profits every of their argument start to make sense.
You can replace "US" with "US and states that agreed to do the same". The argument still stands, it just creates worse outcome for states that agreed to it.
Seeing the lack of responses to your comment and downvotes, you couldn't be more right
I'm talking about the ability of any AI system to obfuscate plagiarism[0] and spam the Internet with technically distinct rewords of the same text. This is currently the most lucrative use of AI, and none of the AI safety people are talking about stopping it.
[0] No, I don't mean the training sets - though AI systems seem to be suspiciously really good at remembering them, too.