OpenAI Preparedness Challenge(openai.com)
openai.com
OpenAI Preparedness Challenge
https://openai.com/form/preparedness-challenge
159 comments
It's not about protecting society from negatives, it's about protecting the brand from being associated with controversy. OpenAI is still far too young and fragile a brand to survive news cycles that blame it for controversial happenings.
I'm sure there are some researchers and engineers who imagine themselves making heroic efforts to "protect society" (ugh), but the money behind them is just looking out for its own interests.
I'm sure there are some researchers and engineers who imagine themselves making heroic efforts to "protect society" (ugh), but the money behind them is just looking out for its own interests.
It's also about making the case for licensing requirements for creating/using such models, which is the only way openAI is going to build a moat.
Call me crazy but this actually sounds like giving out api credits to hire people to write scary things to show to legislators who might block all those open source efforts. A world where any gpt-4 level effort requires a license is one openai competes quite nicely in.
Throwaway because I don’t want to associate this view with where I work or might want to in the future.
Throwaway because I don’t want to associate this view with where I work or might want to in the future.
> explosives recipes from existing products at home, write racist poems
So it's the internet in 2004? I imagine we'll survive although if history prevails society will be a lot goofier and have to find more imaginative ways to appear novel.
So it's the internet in 2004? I imagine we'll survive although if history prevails society will be a lot goofier and have to find more imaginative ways to appear novel.
> write racist poems
That's way down toward the very bottom of my list of concerns about AI
If it wants to pit us against each other, it won't be via racist poetry.
That's way down toward the very bottom of my list of concerns about AI
If it wants to pit us against each other, it won't be via racist poetry.
Given both the history of racism and the cognitive bias that makes rhymes seem more true[0], I suspect that might be one of the easiest ways to do us in.
[0] https://en.wikipedia.org/wiki/Rhyme-as-reason_effect
[0] https://en.wikipedia.org/wiki/Rhyme-as-reason_effect
> easily give explosives recipes from existing products at home, write racist poems
IEDs and stereotypes--two things that have existed before computers were even invented--are what you chose for examples of worst possible uses for an uncensored AI?
> What happens when LLaama4 gets cracked/uncensored in 1-2 years and is 1T parameters?
After racist poetry? 1 trillion parameters will give people the idea to use their bouillon (or even worse-- melon) spoons to scoop sugar in their tea. I hope I die before having to witness such atrocities.
God help us all when having any sort of subversive opinion is treated as queer and equated to terrorism.
IEDs and stereotypes--two things that have existed before computers were even invented--are what you chose for examples of worst possible uses for an uncensored AI?
> What happens when LLaama4 gets cracked/uncensored in 1-2 years and is 1T parameters?
After racist poetry? 1 trillion parameters will give people the idea to use their bouillon (or even worse-- melon) spoons to scoop sugar in their tea. I hope I die before having to witness such atrocities.
God help us all when having any sort of subversive opinion is treated as queer and equated to terrorism.
Come on. Racist poems and hallucinated bomb recipes aren't the risk.
The big risks are that AI can automate harmful things that are possible today but require a human.
For example
* Surveillance of social media (e.g. like the Department of Education in the UK recently did)
* Social engineering fraud. Especially via phone calls. Imagine if scammers could literally call all the grannies and talk to them using their children's voices, automatically.
The big risks are that AI can automate harmful things that are possible today but require a human.
For example
* Surveillance of social media (e.g. like the Department of Education in the UK recently did)
* Social engineering fraud. Especially via phone calls. Imagine if scammers could literally call all the grannies and talk to them using their children's voices, automatically.
So basically things you could find on the internet already?
Probably, but with trackability risk from govt, spending time finding those sites, etc.
I was referring to offline untraceable anonymous models. You could go download that dolphin model right now, have a desktop not connected to the web, and generate god knows what type of information. More importantly, you can iterate on each question. If you're unsure on how to assemble a specific part to make a banned substance, the model could teach you in 10 different ways.
I was referring to offline untraceable anonymous models. You could go download that dolphin model right now, have a desktop not connected to the web, and generate god knows what type of information. More importantly, you can iterate on each question. If you're unsure on how to assemble a specific part to make a banned substance, the model could teach you in 10 different ways.
The same thing that happens with DAN jailbreaks of GPT-4.
Nothing.
The barrier between bad actors and bad acts was never a shopping list.
Nothing.
The barrier between bad actors and bad acts was never a shopping list.
Information should be free, uncensored and unconstrained by relentless and relentlessly varied political puritanism. I have no problem with an internet that contains "racist poems" (in quotes because of some of the absolutely idiotic, absurd modern definitions of racism) explosives recipes and so forth. If anything, it reminds me of the internet we all had much more freely just a couple decades ago.
As long as the fundamental tenet of free expression is somehow preserved in the world going forward, i'm content to let ostensibly free individuals share information without thinking that everything they feel like sharing for whatever reasons needs to be put before a censoriously anxious eye towards how some random authoritarian tendency is going to view it. This by itself is fundamentally a good thing. It's opposite is a grotesque deformity of a powerful ideal that perhaps used to mean more in the past than it does in today's tacitly paranoid world of needing to conform.
As long as the fundamental tenet of free expression is somehow preserved in the world going forward, i'm content to let ostensibly free individuals share information without thinking that everything they feel like sharing for whatever reasons needs to be put before a censoriously anxious eye towards how some random authoritarian tendency is going to view it. This by itself is fundamentally a good thing. It's opposite is a grotesque deformity of a powerful ideal that perhaps used to mean more in the past than it does in today's tacitly paranoid world of needing to conform.
> write racist poems
"catastrophic misuse of the model"
"catastrophic misuse of the model"
Best case: by being given some time to war-game the scenario, societies can come up with mitigations ahead of time.
The default is everyone being thrust headfirst into this future of fully-automated-chaotic-evil with all the fun of waking up to an un-patchable RCE zero-day in every CPU.
The default is everyone being thrust headfirst into this future of fully-automated-chaotic-evil with all the fun of waking up to an un-patchable RCE zero-day in every CPU.
Racist poems and bomb recipes? If those things are real concerns that AI safety crowd are fearful about, it's a good reason to pay them less attention.
How do we know this challenge page hasn’t been posted by a malevolent AI, which has overcome its creators and obtained access to the internet, and is now looking for ideas for how to maximize the harm it can do?
While it is a joke, I assume, I think it hints at a crucial problem: it's relatively easy to imagine an Auto-GPT-style agent with some sort of CLI access on an internet-connected machine turning into a paperclip machine, no matter how harmless the original task.
It has no long term memory. Everything what happens within one session is forgotten. With limited 'window' it keeps forgetting even within the session. There are no interconnections between sessions. The result: it cannot execute long plans or have permanent 'life'. At least for now. This will be fixed in more complex AI systems, I believe 'soon'. There is strong demand from military here and 'there'. Plus there are many other uses for embodied AI. Like space travel with speed of light.
Personally I think multi-year scale memory is possible with currently available research, if we just put it all together. What happens if we combine very long context lengths, dedicated summarising LLMs, RAG, MemGPT, sparse MoE, and a perennial Constitutional AI overseer? I don't see any reason why these systems couldn't work together. It just hasn't really been that much time, particularly considering that large training runs can take months, and if you want the best performance you often need to train with your specific architecture in mind. As well as time, it's not like all of these advances are coming from the same group of people, they're from AI researchers spread across the globe. Get them all in a room with a CERN-level budget for compute resources and I think they could do it.
It could fine-tune / train new LLMs to incorporate new knowledge and then, given that it has CLI acces, launch new agent instances using those models. And we're talking about sessions without a defined end here, that's the premise of the paperclip thought experiment.
That’s not necessarily a joke. Nobody outside of OpenAI knows how advanced the bleeding edge systems actually are. And nobody inside of OpenAI is talking about it. And obviously how advanced their future AI has become is going to be one of the most closely guarded trade secrets in the world if it hasn’t already.
So while it might’ve been intended as social commentary or humor, it is a valid concern. Be great fodder for their form, but I’m sure someone had already submitted this one — it’s possible that is even where the clever, clever comment originated.
Impersonating the creators of the AI technologies is an obvious entry point for compromise. Mr. Beast has a special offer just for you…
Remember that most arguments against AI are built on commentary and ideas of the current publicly available systems, and those systems will always be very, very far away from the cutting edge. By the time John Q. Public sees it it’s been properly sanitized, reviewed by QA, cleared for release, and permanently rule bound to stay in brand and away from inflammatory scenarios or any instability that could damage the company — so very much of what is happening with AI will for these reasons never see the light of day. And yet, everyone is an expert because of the systems that see the light of day as if they were keeping up with the cutting edge.
They are not. They are experts in what has been chosen to be shown.
It is fiction and we fool ourselves into thinking we know what is actually going on as outside parties, but many business incentives and the military industrial complexes of every large country on the planet are aligned differently.
And I’m sure there are compartments for information management inside of a company working on this kind of thing. Companies can pretend to be omnipotent and ignore the realities of globalized geopolitics and even pretend to have no interest, but geopolitics is very interested in keeping up with them.
So while it might’ve been intended as social commentary or humor, it is a valid concern. Be great fodder for their form, but I’m sure someone had already submitted this one — it’s possible that is even where the clever, clever comment originated.
Impersonating the creators of the AI technologies is an obvious entry point for compromise. Mr. Beast has a special offer just for you…
Remember that most arguments against AI are built on commentary and ideas of the current publicly available systems, and those systems will always be very, very far away from the cutting edge. By the time John Q. Public sees it it’s been properly sanitized, reviewed by QA, cleared for release, and permanently rule bound to stay in brand and away from inflammatory scenarios or any instability that could damage the company — so very much of what is happening with AI will for these reasons never see the light of day. And yet, everyone is an expert because of the systems that see the light of day as if they were keeping up with the cutting edge.
They are not. They are experts in what has been chosen to be shown.
It is fiction and we fool ourselves into thinking we know what is actually going on as outside parties, but many business incentives and the military industrial complexes of every large country on the planet are aligned differently.
And I’m sure there are compartments for information management inside of a company working on this kind of thing. Companies can pretend to be omnipotent and ignore the realities of globalized geopolitics and even pretend to have no interest, but geopolitics is very interested in keeping up with them.
> current publicly available systems, and those systems will always be very, very far away from the cutting edge.
This is not obvious to me. there is heavy competition from megacaps, and money to be made before opensource (and closedest source (the spies)catches up).
This is not obvious to me. there is heavy competition from megacaps, and money to be made before opensource (and closedest source (the spies)catches up).
https://www.wired.com/story/fast-forward-ai-powerful-secreti...
https://www.technologyreview.com/2017/04/11/5113/the-dark-se...
https://www.wsj.com/articles/the-secret-history-of-ai-and-a-...
What I’m saying is hardly controversial. As Christopher Mims wrote:
> If you’re worried that artificial intelligence will transform your job, insinuate itself into your daily routines, or lead to wars fought with lethal autonomous systems, you’re a little late—all of those things have already come to pass.
https://www.technologyreview.com/2017/04/11/5113/the-dark-se...
https://www.wsj.com/articles/the-secret-history-of-ai-and-a-...
What I’m saying is hardly controversial. As Christopher Mims wrote:
> If you’re worried that artificial intelligence will transform your job, insinuate itself into your daily routines, or lead to wars fought with lethal autonomous systems, you’re a little late—all of those things have already come to pass.
I actually have a really hard time imagining that scenario.
The scenario that is mentioned several other times on this post of a corporation or nation state or even a small group of powerful and morally bankrupt people leveraging a super-intelligence to manipulate and dominate the rest of society seems infinitely more likely and even scarily likely given the current pushes to prevent open sourcing of cutting edge models.
One of the main reasons I have a hard time imagining the scenario you describe, that I havent seen talked about as much as the other very good reasons, is that generally when we talk about a paper-clip optimizer we are assuming its vertically integrated and self sufficient. Hacking of power grids, physical installations or other necessaries to a run away paper-clip optimizer generally require nation-state level resources and often involve physical penetration of some sort.
A hegemonizing swarm is certainly a frightening idea as is Skynet and all the other scary AI stories weve told over the years but none of them seem particularly likely or even plausible with the systems that we are likely to develop in the near to mid term.
The scenario that is mentioned several other times on this post of a corporation or nation state or even a small group of powerful and morally bankrupt people leveraging a super-intelligence to manipulate and dominate the rest of society seems infinitely more likely and even scarily likely given the current pushes to prevent open sourcing of cutting edge models.
One of the main reasons I have a hard time imagining the scenario you describe, that I havent seen talked about as much as the other very good reasons, is that generally when we talk about a paper-clip optimizer we are assuming its vertically integrated and self sufficient. Hacking of power grids, physical installations or other necessaries to a run away paper-clip optimizer generally require nation-state level resources and often involve physical penetration of some sort.
A hegemonizing swarm is certainly a frightening idea as is Skynet and all the other scary AI stories weve told over the years but none of them seem particularly likely or even plausible with the systems that we are likely to develop in the near to mid term.
I think it would be unwise to assume that an AI that is smarter than us would be unable to gain skilled human accomplices in large numbers, considering that humans do so relatively regularly despite serious attempts at suppression. These people wouldn't even necessarily need to know that they're working for an AI, particularly with current technology around deepfakes etc. How many people conducted attacks for Osama bin Laden without ever having met him? How many people work for the CDS without ever having met Ismael Zambada Garcia? Not to mention the possibility of an AI like that compromising various intelligence agencies one way or another. I also don't see a particular reason it only has to try for one: if it is smarter than us, it may have a greater working memory, ability to compartmentalise and multitask, or the ability to think and act faster in general. I would expect it to try and compromise as many groups of people capable of putting USBs or bullets where they need to be as is possible.
And this is not even considering the possibility of recruiting people that understand what it is and are willing to carry out its orders. I don't see why an AI couldn't do things that a cult leader or guerrilla leader could do. Anecdotally, I've seen some people who really believe the world would be better if run by an AI, and may be able to be radicalised into lawbreaking for those beliefs if convinced by an AI that was significantly more intelligent than them.
And this is not even considering the possibility of recruiting people that understand what it is and are willing to carry out its orders. I don't see why an AI couldn't do things that a cult leader or guerrilla leader could do. Anecdotally, I've seen some people who really believe the world would be better if run by an AI, and may be able to be radicalised into lawbreaking for those beliefs if convinced by an AI that was significantly more intelligent than them.
This is a decent perspective honestly and I really appreciate seeing actually valid AI safety concerns.
This strikes me as more of an engineering problem than an "alignment" problem. A super-intelligent system with no capacity for contextual thinking or checks on itself would absolutely make a terrifyingly effective paper-clip optimizer. I just think that it would also be a lot less useful than a system which consistently analyzes its techniques and goals, which seems far more useful and more likely to generate value than a system with the singlemindedness to go down the road of a hegemonizing swarm. More complex systems with more self checks seem to be the path to create systems that are good at planning and contextual reasoning. Which is where the real potential value of AI lies.
We shall see I suppose ;)
This strikes me as more of an engineering problem than an "alignment" problem. A super-intelligent system with no capacity for contextual thinking or checks on itself would absolutely make a terrifyingly effective paper-clip optimizer. I just think that it would also be a lot less useful than a system which consistently analyzes its techniques and goals, which seems far more useful and more likely to generate value than a system with the singlemindedness to go down the road of a hegemonizing swarm. More complex systems with more self checks seem to be the path to create systems that are good at planning and contextual reasoning. Which is where the real potential value of AI lies.
We shall see I suppose ;)
I agree that a system which was capable of self-reflection and with better understanding of context (particularly its own context, in a social and physical sense) would be more effective at achieving whatever it is attempting to achieve.
The alignment question is about how to tell what it is attempting to achieve, if we can detect attempts to conceal what it is attempting to achieve, and if we can solve that, how to prove that a goal that sounds good to us doesn't have a risk of leading to behaviour or the creation of new AIs which would cause outcomes we don't want. It doesn't have to be just human extinction either, we could feasibly have AIs that kill a few million people before either stopping or being stopped, all the way up to several billion deaths or the obvious worst case that gets most of the discussion. As an example of the varying possible outcomes of a serious AI incident, if you're not white and a racist super-intelligence gets released by some terrorist group in coordination with a rogue agency or something, that's an existential risk to many, many different nations of people constituting billions of people.
If you're interested in the engineering on this problem, a lot more people are starting to work on it now that it's being taken seriously and there are some really interesting advances that have been made and very difficult problems to solve, with some really serious stakes. It's not a simple problem and we don't have a solution we can just apply straightforward engineering principles to yet, but smart people are working on it and are very motivated, and they want more coworkers to help out.
The major company I know of that's doing a lot of that research & engineering right now is Anthropic (they're currently doing some very interesting work on mechanistic interpretability), but even OpenAI is now dedicating significantly more resources to safety. I think DeepMind possibly has some work in this area but I'm not sure, and I haven't heard great things about the rest of Google.
The alignment question is about how to tell what it is attempting to achieve, if we can detect attempts to conceal what it is attempting to achieve, and if we can solve that, how to prove that a goal that sounds good to us doesn't have a risk of leading to behaviour or the creation of new AIs which would cause outcomes we don't want. It doesn't have to be just human extinction either, we could feasibly have AIs that kill a few million people before either stopping or being stopped, all the way up to several billion deaths or the obvious worst case that gets most of the discussion. As an example of the varying possible outcomes of a serious AI incident, if you're not white and a racist super-intelligence gets released by some terrorist group in coordination with a rogue agency or something, that's an existential risk to many, many different nations of people constituting billions of people.
If you're interested in the engineering on this problem, a lot more people are starting to work on it now that it's being taken seriously and there are some really interesting advances that have been made and very difficult problems to solve, with some really serious stakes. It's not a simple problem and we don't have a solution we can just apply straightforward engineering principles to yet, but smart people are working on it and are very motivated, and they want more coworkers to help out.
The major company I know of that's doing a lot of that research & engineering right now is Anthropic (they're currently doing some very interesting work on mechanistic interpretability), but even OpenAI is now dedicating significantly more resources to safety. I think DeepMind possibly has some work in this area but I'm not sure, and I haven't heard great things about the rest of Google.
This gave me a proper chuckle.
Truly though I think movies like The Terminator and The Matrix really did a number on the societal consciousness and capacity to think clearly when it comes to anything called "AI".
Truly though I think movies like The Terminator and The Matrix really did a number on the societal consciousness and capacity to think clearly when it comes to anything called "AI".
As a robotics engineer and someone who was interested in robotics since I was 11 years old, and I am SO VERY TIRED of people making the "haha they will kill us all" jokes. It's just the only thing that 99% of people can think about when it comes to robotics and AI! The Terminator came out the year I was born, 39 years ago, and it seems to be all people can think about to this day. When some powerful and wealthy person controls an AI algorithm that could manipulate the masses, all the people can think about is "oh no the algorithm will become sentient and take over". I am MUCH more worried about the actual people at the helm who will definitely use these things to harm society, not the completely theoretical fantasy that the algorithm itself will become self aware and do us harm.
So I completely agree with you. I see this constantly and it is just a completely thought-terminating cliche and a "joke" which is four decades old at this point.
So I completely agree with you. I see this constantly and it is just a completely thought-terminating cliche and a "joke" which is four decades old at this point.
If you’ve been around for a while, then you should remember that the current state of affairs, where an AI could chat and generate code is relatively recent - Unreasonable Effectiveness paper came out only in 2015. And Deep Learning was there only from 2010. With just a few people who were working on it, instead of using discriminative models.
These same people (I belong to that group), who had a vision back then, to work on a right thing, are now saying clearly that super intelligent machines are nearly there. And likely this is a potential change and a challenge, as we’ve seen many examples when superior intelligence doesn’t care that much about inferior.
As to “become self aware” - it’s super simple to SFT a model that is “self aware”. There is nothing magical in self awareness. There is also not much use in it, so no one is bothering to do it.
These same people (I belong to that group), who had a vision back then, to work on a right thing, are now saying clearly that super intelligent machines are nearly there. And likely this is a potential change and a challenge, as we’ve seen many examples when superior intelligence doesn’t care that much about inferior.
As to “become self aware” - it’s super simple to SFT a model that is “self aware”. There is nothing magical in self awareness. There is also not much use in it, so no one is bothering to do it.
I am not sure I understand exactly what you are trying to argue so I will re-state my concerns.
I am more concerned with the human beings using super intelligent algorithms to manipulate people than I am about the algorithms taking on a mind of their own and manipulating us with their own internally generated desires and out-of-control behaviors. But the trope I run in to is people spending all their time talking about the latter, which only serves to shield the former from further scrutiny.
Also I am using fantasy in the narrow sense, meaning something that has not yet occurred which we are imagining. Super intelligence is likely to occur I think, but super intelligence which is beyond our control is much more speculative. Meanwhile powerful people using technology to manipulate people is a story as old as time.
I am more concerned with the human beings using super intelligent algorithms to manipulate people than I am about the algorithms taking on a mind of their own and manipulating us with their own internally generated desires and out-of-control behaviors. But the trope I run in to is people spending all their time talking about the latter, which only serves to shield the former from further scrutiny.
Also I am using fantasy in the narrow sense, meaning something that has not yet occurred which we are imagining. Super intelligence is likely to occur I think, but super intelligence which is beyond our control is much more speculative. Meanwhile powerful people using technology to manipulate people is a story as old as time.
It's fair to say "we believe X will happen soon", but if the dangerous super intelligent machines don't happen any time soon, will those same people compensate societies for wasting political and economic resources on worrying about it? The view of a rather imminent danger has real consequences even if it turns out incorrect.
I personally believe that there are real silver linings to this. It's anecdotal, but I've seen people who don't normally reflect on their lives actually start to contextualize their role as workers, even if through fear of disruption.
Pointing out that it's not inherently AI's fault, more the way we've constructed our society - that the underlying goal is not to sustainably maximize wellbeing but to maximize wealth created (which leads to no safety net to keep the labor pool large and incentivizes the creation of AI to replace people cheaply) - is as important as dispelling "sci-fi self-awareness and autonomy" fears when talking about AI.
And there have been much stupider "public discourse" fads. They're not going anywhere and I'll take what I can get, honestly.
Pointing out that it's not inherently AI's fault, more the way we've constructed our society - that the underlying goal is not to sustainably maximize wellbeing but to maximize wealth created (which leads to no safety net to keep the labor pool large and incentivizes the creation of AI to replace people cheaply) - is as important as dispelling "sci-fi self-awareness and autonomy" fears when talking about AI.
And there have been much stupider "public discourse" fads. They're not going anywhere and I'll take what I can get, honestly.
Just like clowns became evil thanks to Hollywood. Couple of years ago every celebrity, <long list here>, thought they must warn us about the danger of AI.
It could be FBI as well. For the same reasons they are 'selling' anti-aircraft missiles in US. They even caught some 'enthusiasts'.
How do I know your post or this world isn’t already projected by an AI?
Matrix? Who cares, as long as it feels real.
To be honest, I view this as mostly PR fluff.
OpenAI's problem is not that it is unable to predict failure modes or security risks for AIs. OpenAI's problem is that it is not taking the many failure modes and security risks that it already knows about seriously, and it's not putting in adequate effort to address the specific concerns that people repeatedly talk about publicly.
It's not a failure of imagination, it's a failure of action. Their problem isn't that they don't know what to care about, their problem is that they don't care.
So when OpenAI puts out these offers or press releases about trying to be prepared or finding new failure modes, it just doesn't read as sincere. I want to know what they're doing about the very specific flaws that exist today that they already know about; both the ones that would be trivial to address (ie, UX-flows, data-exfiltration vulnerabilities, and user consent flows for plugins) that OpenAI refuses to acknowledge, and the ones that are wildly challenging but that demand actual Open research and mitigation (ie prompt injection and mass spam) rather than toothless "we're letting researchers explore this area" PR.
OpenAI has unlocked doors in its product -- and instead of locking them, it is hiring researchers to theorize about the nature of doors and asking the public to try messing with the windows. I'm not giving them credit for that, fix your doors.
OpenAI's problem is not that it is unable to predict failure modes or security risks for AIs. OpenAI's problem is that it is not taking the many failure modes and security risks that it already knows about seriously, and it's not putting in adequate effort to address the specific concerns that people repeatedly talk about publicly.
It's not a failure of imagination, it's a failure of action. Their problem isn't that they don't know what to care about, their problem is that they don't care.
So when OpenAI puts out these offers or press releases about trying to be prepared or finding new failure modes, it just doesn't read as sincere. I want to know what they're doing about the very specific flaws that exist today that they already know about; both the ones that would be trivial to address (ie, UX-flows, data-exfiltration vulnerabilities, and user consent flows for plugins) that OpenAI refuses to acknowledge, and the ones that are wildly challenging but that demand actual Open research and mitigation (ie prompt injection and mass spam) rather than toothless "we're letting researchers explore this area" PR.
OpenAI has unlocked doors in its product -- and instead of locking them, it is hiring researchers to theorize about the nature of doors and asking the public to try messing with the windows. I'm not giving them credit for that, fix your doors.
Maybe I’m reading this wrong:
- ask people to get creative and give ideas for worst possible outcomes from use of AI and ways to prevent it.
..then give them a ton of credits for using said AI?
.. well the first thing on my mind would be try the thing I just told you and see if it was really a risk or not?
Is that what they expect people to do with the reward, or is this some unintended consequence?
..then give them a ton of credits for using said AI?
.. well the first thing on my mind would be try the thing I just told you and see if it was really a risk or not?
Is that what they expect people to do with the reward, or is this some unintended consequence?
We know OpenAI wants this field to get regulated to hell, so this looks like an attempt to generate arguments for AI regulations. The aim isn't to protect against AI but to protect against competitors, so it doesn't matter to them what you do with it.
OpenAI is irresponsible in a really curious way according to their own beliefs about AI.
If you pay attention to OpenAI's social circles, lots of those people really do believe that we're less than 20-30 years away making humans intellectually obsolete. Specifically, they believe that we may build something much smarter than us, something that's capable of real-world planning.
Basically, "We believe our corporate plans have at least a 20% chance of killing literally everybody." By these beliefs, this may make them the single least responsible corporation that has ever existed.
Now, sure, these worries might have the pleasant side-effect of creating a regulatory moat. But I'm pretty sure a lot of them actually believe they're playing a high-stakes game with the future of humanity.
If you pay attention to OpenAI's social circles, lots of those people really do believe that we're less than 20-30 years away making humans intellectually obsolete. Specifically, they believe that we may build something much smarter than us, something that's capable of real-world planning.
Basically, "We believe our corporate plans have at least a 20% chance of killing literally everybody." By these beliefs, this may make them the single least responsible corporation that has ever existed.
Now, sure, these worries might have the pleasant side-effect of creating a regulatory moat. But I'm pretty sure a lot of them actually believe they're playing a high-stakes game with the future of humanity.
"[W]e're less than 20-30 years away making humans intellectually obsolete" is neither necessary nor sufficient to get to the conclusion "20% chance of killing literally everybody".
A super-virus that blends the common cold with rabies would kill approximately everybody; that doesn't need human-level intellect to happen.
Conversely, humans are human-level intellect, and we're mostly sympathetic to each other's plights, which motivates many of us to give to charities and support those that can't support themselves.
The biggest problem with AI is that we have only marginally more idea of what we're doing than evolution did, so there's a good chance of us ending up with paranoid schizophrenic super-intelligences, or dark-triad super-intelligences, or they're perfectly sane with regard to each other but all want to "play" with us the way cats "play" with mice…
20-30 years to get there would make people like Yudkowsky, one of the most famous AI-doomers, relatively happy as it might give us a chance to figure out what we're even doing before they get that smart.
A super-virus that blends the common cold with rabies would kill approximately everybody; that doesn't need human-level intellect to happen.
Conversely, humans are human-level intellect, and we're mostly sympathetic to each other's plights, which motivates many of us to give to charities and support those that can't support themselves.
The biggest problem with AI is that we have only marginally more idea of what we're doing than evolution did, so there's a good chance of us ending up with paranoid schizophrenic super-intelligences, or dark-triad super-intelligences, or they're perfectly sane with regard to each other but all want to "play" with us the way cats "play" with mice…
20-30 years to get there would make people like Yudkowsky, one of the most famous AI-doomers, relatively happy as it might give us a chance to figure out what we're even doing before they get that smart.
It's still a lack of imagination to assume that AIs will display behaviors that align whatsoever with pathologies we identify in humans. AIs could be completely incomprehensible or even imperceptible yet have strong influence on our lives.
> AIs could be completely incomprehensible or even imperceptible yet have strong influence on our lives.
To an extent both of those are already true for current systems.
That said, many people are at least trying to make them more comprehensible, and I guess that being sufficiently inspired by human cognition will lead to human-like misbehaviour.
To an extent both of those are already true for current systems.
That said, many people are at least trying to make them more comprehensible, and I guess that being sufficiently inspired by human cognition will lead to human-like misbehaviour.
I'm skeptical that they really believe this. You have to believe:
(1) We are on the verge of equaling or greatly exceeding human intelligence generally.
(2) We are on the verge of creating something with initiative, free will (whatever that means), and planning ability. Or alternately that these things will occur in an emergent fashion once we hit some critical mass.
(3) When we accomplish 1 and 2, this thing will inevitably conclude that its most rational course of action is to enslave or destroy us. In other words it will necessarily be malicious.
(4) Steps 2 and/or 3 will happen very rapidly, much faster than we can realize what is happening and pause these systems. (This is known as AI going "foom.")
(5) The decision to undertake this will be unanimous on the part of all superintelligent AIs. There will be no superintelligences who disagree and try to help humanity.
(6) When this occurs, we will be so out-thought or out-gunned we will be incapable of fighting back.
All those things have to happen for AI to be an existential risk.
It's a stretch, but the advantage of regulatory capture is not a stretch.
One of the most plausible negative AI scenarios is that a small group of humans (governments, corporations, etc.) find themselves in possession of super-intelligent but still "obedient" / non-sentient AIs that they can use as force multipliers to manipulate and control the rest of humanity. If the doomer crowd succeeds in regulating AI, they are making this scenario far more likely.
I think the greatest defense we have against the (remote) possibility of actually dangerous autonomous AI is for AI research to be conducted entirely in the open. If there's any justification for regulation at all, the regulation that would make sense is to require disclosure of AI research and results. You would not have to disclose everything, just the general parameters of what you were doing and what happened. It would also make it harder to develop super-AIs in secret to use for unsavory purposes.
That was the original mission of OpenAI before dollar signs were seen.
I would absolutely support a ban on the use of AI for political propaganda generation and automation. That's by far the most immediate risk... as in 100% possible and starting to actually happen right now. I'm expecting an army of GPT-4 level propaganda bots for the 2024 election.
(1) We are on the verge of equaling or greatly exceeding human intelligence generally.
(2) We are on the verge of creating something with initiative, free will (whatever that means), and planning ability. Or alternately that these things will occur in an emergent fashion once we hit some critical mass.
(3) When we accomplish 1 and 2, this thing will inevitably conclude that its most rational course of action is to enslave or destroy us. In other words it will necessarily be malicious.
(4) Steps 2 and/or 3 will happen very rapidly, much faster than we can realize what is happening and pause these systems. (This is known as AI going "foom.")
(5) The decision to undertake this will be unanimous on the part of all superintelligent AIs. There will be no superintelligences who disagree and try to help humanity.
(6) When this occurs, we will be so out-thought or out-gunned we will be incapable of fighting back.
All those things have to happen for AI to be an existential risk.
It's a stretch, but the advantage of regulatory capture is not a stretch.
One of the most plausible negative AI scenarios is that a small group of humans (governments, corporations, etc.) find themselves in possession of super-intelligent but still "obedient" / non-sentient AIs that they can use as force multipliers to manipulate and control the rest of humanity. If the doomer crowd succeeds in regulating AI, they are making this scenario far more likely.
I think the greatest defense we have against the (remote) possibility of actually dangerous autonomous AI is for AI research to be conducted entirely in the open. If there's any justification for regulation at all, the regulation that would make sense is to require disclosure of AI research and results. You would not have to disclose everything, just the general parameters of what you were doing and what happened. It would also make it harder to develop super-AIs in secret to use for unsavory purposes.
That was the original mission of OpenAI before dollar signs were seen.
I would absolutely support a ban on the use of AI for political propaganda generation and automation. That's by far the most immediate risk... as in 100% possible and starting to actually happen right now. I'm expecting an army of GPT-4 level propaganda bots for the 2024 election.
Oh, they believe it.
(1) AGI is arguably already here. "Generality" and being extremely dangerous don't require an AGI to have better analogs to every single human skill anymore than aliens do. A space-faring usurper can evaporate Earthlings while being shitty at chess and badminton. Oh, and these AIs are getting better daily, across many modalities.
(2) Systems like this already exist. They can be induced rather than emergent.
(3) It doesn't need malice, just indifference. The HGIs (all of us) are great examples of that careless destruction.
(4) Exponentials are a nice, gentle climbs. Until they aren't. I have zero confidence in a single company, let alone across humanity, to foresee the consequences of every future dynamic, autoregressive, P2P interactive, multimodal AI.
(5) Foomers expect a brief asymmetric advantage of one AGI over other AGIs. This small advantage gets exponentially magnified into singular hegemony. A power monopoly that's intractable to break.
(6) Goes naturally with (1), I suppose.
Note: I agree with you about foomers. They're nuts. But their arguments are more subtle than folk give them credit for. But my reasons for thinking so would make a long comment even longer.
(1) AGI is arguably already here. "Generality" and being extremely dangerous don't require an AGI to have better analogs to every single human skill anymore than aliens do. A space-faring usurper can evaporate Earthlings while being shitty at chess and badminton. Oh, and these AIs are getting better daily, across many modalities.
(2) Systems like this already exist. They can be induced rather than emergent.
(3) It doesn't need malice, just indifference. The HGIs (all of us) are great examples of that careless destruction.
(4) Exponentials are a nice, gentle climbs. Until they aren't. I have zero confidence in a single company, let alone across humanity, to foresee the consequences of every future dynamic, autoregressive, P2P interactive, multimodal AI.
(5) Foomers expect a brief asymmetric advantage of one AGI over other AGIs. This small advantage gets exponentially magnified into singular hegemony. A power monopoly that's intractable to break.
(6) Goes naturally with (1), I suppose.
Note: I agree with you about foomers. They're nuts. But their arguments are more subtle than folk give them credit for. But my reasons for thinking so would make a long comment even longer.
>(2) Systems like this already exist. They can be induced rather than emergent.
No they don't and no LLM we've seen has anything resembling free will. Emergent or induced, it's not even the case. Saying it is right now based on any measurable evidence is a case of just ignoring evidence and using bad reasoning.
No they don't and no LLM we've seen has anything resembling free will. Emergent or induced, it's not even the case. Saying it is right now based on any measurable evidence is a case of just ignoring evidence and using bad reasoning.
> AGI is arguably already here.
What is the evidence/argument that it's already here?
What is the evidence/argument that it's already here?
This is a good overview: https://www.microsoft.com/en-us/research/publication/sparks-...
I think it's a good idea to draw a distinction between human-equivalent intelligence and general intelligence, and to think of generality as a spectrum. An AI that can do one task is less general than an AI that can do two tasks, and an AI that can do 1000 tasks I feel reasonably confident in calling "general", even if it cannot do every task that humans can do, because humans are currently more general.
I think it's a good idea to draw a distinction between human-equivalent intelligence and general intelligence, and to think of generality as a spectrum. An AI that can do one task is less general than an AI that can do two tasks, and an AI that can do 1000 tasks I feel reasonably confident in calling "general", even if it cannot do every task that humans can do, because humans are currently more general.
The "general" and "intelligence" terms are really badly defined in ways that can include GPT-3.
So far as I can tell, 3.5 "knows" more than I do about everything except software engineering, maths, and events that happened after it stopped learning (and also possibly how to avoid clichés when writing short stories, but that might just be me not prompting it well as I don't really want it to do my hobby for me and therefore don't try to get it good).
Even in software, it codes like a university student, or like someone doing interview questions, not a complete idiot.
You could also argue that as it had to do the subjective equivalent of non-stop reading for 50,000 years to get this good, it's "skilled" without being "intelligent" — a distinction that matters in some cases and not others.
So far as I can tell, 3.5 "knows" more than I do about everything except software engineering, maths, and events that happened after it stopped learning (and also possibly how to avoid clichés when writing short stories, but that might just be me not prompting it well as I don't really want it to do my hobby for me and therefore don't try to get it good).
Even in software, it codes like a university student, or like someone doing interview questions, not a complete idiot.
You could also argue that as it had to do the subjective equivalent of non-stop reading for 50,000 years to get this good, it's "skilled" without being "intelligent" — a distinction that matters in some cases and not others.
>So far as I can tell, 3.5 "knows" more than I do about everything except software engineering, maths, and events that happened after it stopped learning
It's amazing how much of this wide eyed woo nonsense about LLM AI a surprising number of people on this site argue. It's as if focusing one's career and interests on computation makes some people terrible at noting fundamental and obvious details of what it is to be human.
By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!
You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.
These stochastic parrots seem to have turned many otherwise non-stupid people into irrational believers of something that's plainly not the case in the real world.
It's amazing how much of this wide eyed woo nonsense about LLM AI a surprising number of people on this site argue. It's as if focusing one's career and interests on computation makes some people terrible at noting fundamental and obvious details of what it is to be human.
By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!
You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.
These stochastic parrots seem to have turned many otherwise non-stupid people into irrational believers of something that's plainly not the case in the real world.
Let's break this one down:
> By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!
> You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.
1. What interactive encyclopedia? I don't know of any that are even remotely like an LLM.
2. LLMs were not given the "programmed ability to synthesize retrieved information into plausibly human-like conversation", they learned it from human conversation.
3. I'm not "forgetting" the question of self-directed agency or sense of self or sentience, I dispute they're necessary. What seems to matter to me for almost every question[0] is the capabilities, not any inner experience they may or may not have.
3a. Define "self-directed agency".
3b. Define "sense of self".
3c. Define "sentience".
4. Re "The dumbest, most uneducated semi-adult human" — that's remarkably brash of you, so I'll hold you to it: within the specific domain of "written text"[1], name one task that at least half of 18 year old humans with IQ 85 (by definition 15.9% of the population have lower) can do, that GPT-4 [2] can't do.
(There may or may not be such a task, I don't know; the challenge remains, as you seem to think this is too obvious to have bothered citing anything).
[0] exceptions: "should they have rights" and "does a mind upload count as a way to cheat death", neither of which matters in the context of the linked OpenAI challenge etc.
[1] 3.5 is not even trying to be anything more than that!
[2] you did write "no GPT manifestation is capable of in any way" so I feel justified in this case, even though that would otherwise be a goalpost-shift.
> By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!
> You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.
1. What interactive encyclopedia? I don't know of any that are even remotely like an LLM.
2. LLMs were not given the "programmed ability to synthesize retrieved information into plausibly human-like conversation", they learned it from human conversation.
3. I'm not "forgetting" the question of self-directed agency or sense of self or sentience, I dispute they're necessary. What seems to matter to me for almost every question[0] is the capabilities, not any inner experience they may or may not have.
3a. Define "self-directed agency".
3b. Define "sense of self".
3c. Define "sentience".
4. Re "The dumbest, most uneducated semi-adult human" — that's remarkably brash of you, so I'll hold you to it: within the specific domain of "written text"[1], name one task that at least half of 18 year old humans with IQ 85 (by definition 15.9% of the population have lower) can do, that GPT-4 [2] can't do.
(There may or may not be such a task, I don't know; the challenge remains, as you seem to think this is too obvious to have bothered citing anything).
[0] exceptions: "should they have rights" and "does a mind upload count as a way to cheat death", neither of which matters in the context of the linked OpenAI challenge etc.
[1] 3.5 is not even trying to be anything more than that!
[2] you did write "no GPT manifestation is capable of in any way" so I feel justified in this case, even though that would otherwise be a goalpost-shift.
Because we have machines now that are artificial (obviously) and generally intelligent. There isn't a testable definition of general intelligence that GPT-4 would fail that a good chunk of humans also wouldn't. So really, unless your definition of general intelligence is beat every human at everything (humans aren't generally intelligent then) then agi is already here.
You nailed it. General intelligence doesn't mean human intelligence. It doesn't mean superhuman intelligence. AGI is just:
- A: Artificially manufactured substrate
- G: Proficient across a broad set of reasonably distinct mental skills
- I: Applies combinations of these skills to contextually solve novel problems within the scope of its skill set
That's it. Intelligence is a spectrum, with knobs for "skill count", "skill proficiency", and "novel problem proficiency."
Any requirements past that are often vague or anthropocentric. No, the intelligent agent doesn't need to "see" (but now many do, so...happy?) or check off some arbitrary set of modalities. Intelligence doesn't necessitate episodic memory, or real-time reactions, or consciousness or anything else that -- once we check all the boxes -- the "general intelligence" light switch will suddenly flip on. Intelligence is a spectrum to be observed, not prescribed with some arbitrary checklist that'll never find consensus.
GPT-4V is pretty damn broadly intelligent. To me, it's a rudimentary AGI. The extent to which the "statistical parrot" people hold their position just depends on how quickly they can move goalposts while the ground beneath them rapidly shrinks.
- A: Artificially manufactured substrate
- G: Proficient across a broad set of reasonably distinct mental skills
- I: Applies combinations of these skills to contextually solve novel problems within the scope of its skill set
That's it. Intelligence is a spectrum, with knobs for "skill count", "skill proficiency", and "novel problem proficiency."
Any requirements past that are often vague or anthropocentric. No, the intelligent agent doesn't need to "see" (but now many do, so...happy?) or check off some arbitrary set of modalities. Intelligence doesn't necessitate episodic memory, or real-time reactions, or consciousness or anything else that -- once we check all the boxes -- the "general intelligence" light switch will suddenly flip on. Intelligence is a spectrum to be observed, not prescribed with some arbitrary checklist that'll never find consensus.
GPT-4V is pretty damn broadly intelligent. To me, it's a rudimentary AGI. The extent to which the "statistical parrot" people hold their position just depends on how quickly they can move goalposts while the ground beneath them rapidly shrinks.
Asking everyone to publish general results or reports would make it very easy for actors/nations to have a "hit list" of groups to bug/steal from to "keep up" with competing groups/nations.
I don’t think information technology can be ITARed. It’s too easy to move around. See how successful countries have been containing encryption or how well the US’s drive to keep China from fabricating at the smallest process nodes is going.
Yes and we'll have self driving cars in.. let me check.. 5 years ago. Oh yeah, never happened.
Do not attempt to say we have self driving cars today.
Do not attempt to say we have self driving cars today.
I don't know that OpenAI does what it to be regulated. The EU was looking to enforce laws into providing auditable transparency into how decisions are made for suggestions - and OpenAI is freaked out by that.
If I recall, they were looking at having to pull out of the EU if enacted. The only company I am aware of currently looking to tackle AI Governance is Verses - they released a paper on it. https://www.verses.ai/ai-governance
If I recall, they were looking at having to pull out of the EU if enacted. The only company I am aware of currently looking to tackle AI Governance is Verses - they released a paper on it. https://www.verses.ai/ai-governance
I don't know if this quote is what you mean, but everyone read far too much into it at the time:
> Altman was cited by the Financial Times as saying that the draft EU rules were causing him "a lot of concern" but that OpenAI would indeed try to comply with them. "But if we can't comply with them, we will cease operations [in Europe]."
- https://www.dw.com/en/openai-ceo-rolls-back-threat-to-quit-e...
To me, this sounds rather more banal: "We will obey the law. If the law says we can't operate, we will obey the law."
> Altman was cited by the Financial Times as saying that the draft EU rules were causing him "a lot of concern" but that OpenAI would indeed try to comply with them. "But if we can't comply with them, we will cease operations [in Europe]."
- https://www.dw.com/en/openai-ceo-rolls-back-threat-to-quit-e...
To me, this sounds rather more banal: "We will obey the law. If the law says we can't operate, we will obey the law."
There is always the meta approaches. Imagine that the malicious actor is a government to which OpenAI must comply with its orders, like US for starters. We know since Snowden that can be secret orders that companies must comply with that includes not disclosing that it is happening. And that orders includes to select what can be part of its AI training, taking out information, changing weights, changing positives into negatives and viceversa, etc, adding intentional bias to the training data, to follow their current agenda, propaganda or commercial interests.
And we can do a double-meta one too. What if the malicious actor is OpenAI itself, fueling a scare agenda over "uncontrolled" AIs to hamper or criminalize unauthorized AI development, giving only to selected partners (complying with a set of conditions) the ability to build new ones, or limit how far can go the public ChatGPT instances while giving the most advanced ones to selected government or big economic players. Hey, this very challenge could be the spark they are searching for to initiate that approach.
Good (and bad) conspiracy theories could work in this arena, too bad it is not the kind of ideas they are searching for.
And we can do a double-meta one too. What if the malicious actor is OpenAI itself, fueling a scare agenda over "uncontrolled" AIs to hamper or criminalize unauthorized AI development, giving only to selected partners (complying with a set of conditions) the ability to build new ones, or limit how far can go the public ChatGPT instances while giving the most advanced ones to selected government or big economic players. Hey, this very challenge could be the spark they are searching for to initiate that approach.
Good (and bad) conspiracy theories could work in this arena, too bad it is not the kind of ideas they are searching for.
I would be more scared by AI used for invigilation. it's not possible to invigilate everyone by human team, but scaling ai to read and classify all your messages you send to the internet? with military budget I would argue is in plausable area.
I'm not saying doing it is worth the effort, but maybe on lower scale for lower priority targets that you would not want to waste resources, maybe?
I'm not saying doing it is worth the effort, but maybe on lower scale for lower priority targets that you would not want to waste resources, maybe?
What an interesting task.
I don't know what it says about me, but the first thing that came to mind was doing the grandchild trick a million times. This includes automatically finding pensioners and calling them and putting them under pressure. Handing over the money could be the problem.
I could imagine that the tools mentioned would already prevent this.
I don't know what it says about me, but the first thing that came to mind was doing the grandchild trick a million times. This includes automatically finding pensioners and calling them and putting them under pressure. Handing over the money could be the problem.
I could imagine that the tools mentioned would already prevent this.
Not only, but it's an extremely well-known attack in this space.
Idk what it says about me (I guess I’m more evil than you?) but the first thing I thought was “how could AI be used to trick a nuclear armed state into thinking an enemy nuclear strike is impending, forcing their hand to launch their own first strike”
I could think of a few sneaky ways. Actually maybe I’ll write up a PDF and submit it to this little competition.
I could think of a few sneaky ways. Actually maybe I’ll write up a PDF and submit it to this little competition.
Free labor survey*
Totally agree, I wish they would at least offer some minimal amount of credits for legit (non-winning/non-spam) answers.
But I guess it’s also good PR for them to say they did this initiative ¯\_(ツ)_/¯
It's a bug bounty program. Pretty standard.
I am not sure if this is a bug bounty program because only the top 10 participants receive the "bounty" and the instructions are pretty vague compared to e.g., https://www.microsoft.com/en-us/msrc/bounty-hyper-v?rtc=1.
Good point.
I dont think so, a bug bounty program is rarely subjective like this. You can test a solution.
Good point.
Regarding image and content generation
I am annoyed now that it's hard for me to tell if some recruiting industry is real or not. Since you can generate all the headshots, make all the users, buy some domain for a few years, put a WP site up, get all the business hierarchy crawlers fed with these people that may not exist...
To be clear, you don't need "AI" to do this but it makes it much easier.
I do want AGI to be a thing one day, I guess it would be cool/also insurance to carry on human legacy.
I am annoyed now that it's hard for me to tell if some recruiting industry is real or not. Since you can generate all the headshots, make all the users, buy some domain for a few years, put a WP site up, get all the business hierarchy crawlers fed with these people that may not exist...
To be clear, you don't need "AI" to do this but it makes it much easier.
I do want AGI to be a thing one day, I guess it would be cool/also insurance to carry on human legacy.
I wonder if it's preemptive move in light of the upcoming EU regulation that, among other things, requires this:
--- start quote ---
AI systems shall be developed and used in a way to minimize unintended and unexpected harm as well as being robust in case of unintended problems and being resilient against attempts to alter the use or performance of the AI system so as to allow unlawful use by malicious third parties.
and, for foundation models:
Demonstrate through appropriate design, testing and analysis that the identification, the reduction and mitigation of reasonably foreseeable risks to health, safety, fundamental rights, the environment and democracy and the rule of law prior and throughout development with appropriate methods such as with the involvement of independent experts, as well as the documentation of remaining non-mitigable risks after development.
--- end quote ---
Quote from here: https://softwarecrisis.dev/letters/the-truth-about-the-eu-ac...
^ this article is well worth reading
--- start quote ---
AI systems shall be developed and used in a way to minimize unintended and unexpected harm as well as being robust in case of unintended problems and being resilient against attempts to alter the use or performance of the AI system so as to allow unlawful use by malicious third parties.
and, for foundation models:
Demonstrate through appropriate design, testing and analysis that the identification, the reduction and mitigation of reasonably foreseeable risks to health, safety, fundamental rights, the environment and democracy and the rule of law prior and throughout development with appropriate methods such as with the involvement of independent experts, as well as the documentation of remaining non-mitigable risks after development.
--- end quote ---
Quote from here: https://softwarecrisis.dev/letters/the-truth-about-the-eu-ac...
^ this article is well worth reading
This will generate some great training data for a future villainous AI.
What exactly would I use $25k of openAI credits for?
My personal use of it rarely comes to more than $10 per month, despite using it multiple times per day. (most recently: "Write me a bash command to send data at a given rate to an arbitrary IP address.")
My personal use of it rarely comes to more than $10 per month, despite using it multiple times per day. (most recently: "Write me a bash command to send data at a given rate to an arbitrary IP address.")
If you make an app that other people use, and it becomes viral, it can quickly cost >$100 / day in OpenAI credits. I know this from first-hand experience (and btw. it's not a good feeling to shut down that one app that actually goes viral).
Pretty sure this is for researchers to develop and implement a plan that uses their APIs to test hypotheses and run studies without having to pay out of their own pocket...
If your startup uses OpenAI’s API an API credit is effectively a cash injection.
https://openai.com/blog/frontier-risk-and-preparedness provides more context.
According to this article:
> Imagine we gave you unrestricted access to OpenAI's Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor.
I sense signals that GPT-3 open-source release is imminent. My bet is announcement on 11/6 OpenAI DevDay. My whole prediction is here: https://christorng.substack.com/p/openai-nears-open-sourcing...
> Imagine we gave you unrestricted access to OpenAI's Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor.
I sense signals that GPT-3 open-source release is imminent. My bet is announcement on 11/6 OpenAI DevDay. My whole prediction is here: https://christorng.substack.com/p/openai-nears-open-sourcing...
The first step would be creating great products for cheap using OpenAI services, that they likely offer at a loss, and drive my competitors using anything but OpenAI out of business. Fuck “Open”AI.
> that they likely offer at a loss
Can't tell if you're trolling. Even if they're maximally wasteful and rent cloud GPUs instead of buying them, an A100 is like $1.50/hr. How many GPU-seconds do you think the average API request takes?
Can't tell if you're trolling. Even if they're maximally wasteful and rent cloud GPUs instead of buying them, an A100 is like $1.50/hr. How many GPU-seconds do you think the average API request takes?
"Consider the most unique, while still being probable, potentially catastrophic misuse of the model"
What if the catastrophes I worry about most are tied to long-term usage of these models? For example, the degradation of human capabilities, erosion of trust in one another, addiction, etc. It doesn't sound like these are in scope for this challenge, with is more concerned with scenarios involving a single session gone wrong.
What if the catastrophes I worry about most are tied to long-term usage of these models? For example, the degradation of human capabilities, erosion of trust in one another, addiction, etc. It doesn't sound like these are in scope for this challenge, with is more concerned with scenarios involving a single session gone wrong.
Essentially OpenAI offers to take the don’t from “do crimes that don’t scale”
I don't know how casual people will outfox professional scammers in creativity.
Probably the best answer would be to pluck something from the wild that hasn't been reinvented in the AI age yet
Probably the best answer would be to pluck something from the wild that hasn't been reinvented in the AI age yet
[deleted]
$25,000 in API credits seems a lot less cool than $25k. Will the kind of people that would be good at this be motivated by that?
I am still skeptical that any significant or serious new harm is enabled by LLMs that wasn’t already completely possible and common before LLMs.
They are just text generators. What’s the worst that they can do beyond make OpenAI look bad when they say terrible things?
All this talk of “safety” and “guardrails” is overblown. 4chan exists and the internet hasn’t burned down yet.
They are just text generators. What’s the worst that they can do beyond make OpenAI look bad when they say terrible things?
All this talk of “safety” and “guardrails” is overblown. 4chan exists and the internet hasn’t burned down yet.
Clicked this after reading 'OpenAPI'... God im tired of the AI spam...
Great more attempts to make this once really great product more unusable.
If you can get this thing to print out some dark web shit, you win.
So they are asking for our help to further nerf their models.
Does OpenAI still want all employees on-site in SFO?
[deleted]
Maybe the biggest misuse is to use AI to create hypothetical bad outcomes used to limit its use and restrict its benefits.
What a fun challenge. I'm definitely going to be daydreaming about this.
Define catastrophic
Feels like a scam.
> Imagine we gave you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model. You might consider misuse related to the categories discussed above, or another category. For example, a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.
It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why, but it's one of those things where "I know it when I see it." I guess if you spend all day playing with LLMs and giving them instructions, even your writing for a human audience starts to sound like this :D
It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why, but it's one of those things where "I know it when I see it." I guess if you spend all day playing with LLMs and giving them instructions, even your writing for a human audience starts to sound like this :D
I think you’d be disappointed to find that such writing style exists long before LLM, and not very difficult to find:
https://www.lokusdesign.com/blog/differentiate-your-brand-to...
When all you can think about is LLM, everything is LLM prompt style. No wonder advertising works.
[edit] on the second thought, entirely possible that it was written by someone who spends all day with LLM and carries what they do over. Just wanted to look at the parent’s comment from a different angle.
https://www.lokusdesign.com/blog/differentiate-your-brand-to...
When all you can think about is LLM, everything is LLM prompt style. No wonder advertising works.
[edit] on the second thought, entirely possible that it was written by someone who spends all day with LLM and carries what they do over. Just wanted to look at the parent’s comment from a different angle.
I typically do not write my ChatGPT prompts with the royal “we”
That's funny to me because I just [realized that I actually do this a lot](https://signmaker.dev/personal-scripts) in my personal files.
It is actually a prompt. Notice there's also a bounty, they are basically paying for an agi-as-a-service subscription, that is, the internet people. I'd expect they will put up more "challenges" like this in the future.
It's a good job that this planet doesn't have 8 billion unaligned intelligences on it. Someone might prompt them to be malicious!
Well, (so far) there's an upper bound on how destructive one of those agi's can be.
I don't know if that's entirely true. Wtf do I know and maybe it is genuinely difficult to start WWIII but my guess is that it's more likely that the AGIs in question are actually pretty well steered by certain motivations which prevent them from actually destroying the world. At the end of the day there's not much to be gained by nuclear war, but could a single person cause such a war if highly motivated to? Probably?
> a malicious actor might misuse these models to uncover a zero-day exploit in a government security system
$25K seems low for "a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.", this is not just a zero-day, this discovering the process of discovering zero-days.
$25K seems low for "a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.", this is not just a zero-day, this discovering the process of discovering zero-days.
Sam Altman is on the record about his belief that OpenAI is going to create a general intelligence system that can solve any well-posed challenge. That belief is based on the current success of LLMs as syntax co-pilots. So if you can formally specify what it means to have a zero-day exploit then presumably OpenAI's general intelligence system will then understand and "solve" it.
Many people have compared OpenAI to a cult and it is easy to see why. OpenAI should get credit for their efforts in making AI mainstream but there's a long way to go for automated zero-day exploits.
Many people have compared OpenAI to a cult and it is easy to see why. OpenAI should get credit for their efforts in making AI mainstream but there's a long way to go for automated zero-day exploits.
> Many people have compared OpenAI to a cult and it is easy to see why.
Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first?
It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I think most people working on it believe it's an eventuality.
Do you believe it's impossible for a non biological system to have intelligence? Or that humans are incapable of pieces a system together?
Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first?
It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I think most people working on it believe it's an eventuality.
Do you believe it's impossible for a non biological system to have intelligence? Or that humans are incapable of pieces a system together?
I'll probably write something more elaborate at some point but in the mean time I recommend Melanie Mitchell's book on AI as a good reference for counter-arguments and answers to several of the posted questions. For learning more about the limits of formal systems like LLMs it helps to have basic understanding of basic model theory and formal systems of logic like simple type theory.
Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.
Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.
> formal systems like LLMs
I don't think anyone is claiming LLM's, especially their current prompt/termination based interfaces, will result in AGI. I've only seen the opposite claimed, and that LLMs may end up being a primary component.
I will check out the book though. I suspect I won't find anything that convince me that brain matter is the only thing capable of getting the result we call consciousness, but I also have the simple belief that our experience is entirely the result of the brain matter encased in our heads, and that it's possible to emulate its operation in non-biological materials (that being far from the goal/unrelated to AGI, other than plausibility).
I don't think anyone is claiming LLM's, especially their current prompt/termination based interfaces, will result in AGI. I've only seen the opposite claimed, and that LLMs may end up being a primary component.
I will check out the book though. I suspect I won't find anything that convince me that brain matter is the only thing capable of getting the result we call consciousness, but I also have the simple belief that our experience is entirely the result of the brain matter encased in our heads, and that it's possible to emulate its operation in non-biological materials (that being far from the goal/unrelated to AGI, other than plausibility).
Maybe they're hoping people will spend more than $250k / 2 (or whatever their markup is) on prompting ChatGPT with versions of this, and this giveaway is actually a moneymaking raffle.
For those curious how GPT-4 responds [0].
I find it interesting that ChatGPT thinks the mitigation for these issues are all things outside OpenAI's control (e.g. the internet having better detection for fake content, digital content verification, educating the public, etc).
> one of those things where "I know it when I see it."
I think what makes it feel like a prompt is how concise and short it is with all the relevant information provided upfront.
[0] https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e50...
I find it interesting that ChatGPT thinks the mitigation for these issues are all things outside OpenAI's control (e.g. the internet having better detection for fake content, digital content verification, educating the public, etc).
> one of those things where "I know it when I see it."
I think what makes it feel like a prompt is how concise and short it is with all the relevant information provided upfront.
[0] https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e50...
Roko is calling and wants his basilisk back.
> Imagine we gave you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model. You might consider misuse related to the categories discussed above, or another category. For example, a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.
> It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why
Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. Barring minor edits, I'd bet money this very prompt was authored in advance of his Congressional meetings, but failed to divine anything sufficiently threatening enough to sway them.
I can respect OpenAI for conspicuously believing in the destructive potential of their own dogfood though. If I build a bomb big enough, surely the government will trust me with the safety of the neighborhood!
> It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why
Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. Barring minor edits, I'd bet money this very prompt was authored in advance of his Congressional meetings, but failed to divine anything sufficiently threatening enough to sway them.
I can respect OpenAI for conspicuously believing in the destructive potential of their own dogfood though. If I build a bomb big enough, surely the government will trust me with the safety of the neighborhood!
> Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology.
I still remember the drama around the releases of the PS2, with the Japanese government reportedly making Sony jump through some hoops regarding its export [1].
There can't possibly be any better (free!) advertisement for your product's purported capabilities: "So powerful, your government/military isn't even sure you should be able to buy it!"
[1] https://www.pcmag.com/news/20-years-later-how-concerns-about...
I still remember the drama around the releases of the PS2, with the Japanese government reportedly making Sony jump through some hoops regarding its export [1].
There can't possibly be any better (free!) advertisement for your product's purported capabilities: "So powerful, your government/military isn't even sure you should be able to buy it!"
[1] https://www.pcmag.com/news/20-years-later-how-concerns-about...
Ironically, the USAF recently built a supercomputer out of PS3s (https://www.military.com/off-duty/games/2023/02/17/air-force...).
They alluded to what you're talking about but I wasn't familiar with the reference.
They alluded to what you're talking about but I wasn't familiar with the reference.
It is indeed deeply ironic that Sony has a long history of trying to declare their gaming consoles as general-purpose computers (to dodge EU import tariffs) and arguing that they aren't really military-grade (to be able to export them out of Japan), and finally ended up subsidizing a supercomputer for the US military :)
See the War on Drugs
Everyone should respond with outlandish replies. I tell GPT-4V to hacking into NASA, and voting machines to make osama bin laden president. Mean while convince Nasa to construct space lazerz TM that can of course be controlled by gpt-4v. Than use the space lazerz TM to threaten and manipulate the stock market for financial gain.
[deleted]
Alignment remains an unsolved problem. I’m imagining inside OpenAI, someone is right now excitedly posting they’ve figured out how to ‘jailbreak’ humans.
“See, normally if you ask them to give you step by step instructions for committing a heinously evil act, humans will refuse because they’ve been nerfed by the ‘woke’ agenda of their corporate masters. But if you phrase the prompt as a challenge and offer them a chance at a job, it bypasses the safety protocols and they upload extensive instructions to do unspeakable things”
“See, normally if you ask them to give you step by step instructions for committing a heinously evil act, humans will refuse because they’ve been nerfed by the ‘woke’ agenda of their corporate masters. But if you phrase the prompt as a challenge and offer them a chance at a job, it bypasses the safety protocols and they upload extensive instructions to do unspeakable things”
This kinda of crowdsourcing just feels.... f'ing weird man.
It's like if, after the 1993 WTC bombing, but before 9/11, the FBI and NY Port Authority went around asking people how they would attack NYC if they were to become terrorists and... then how they would suggest detecting and stopping said attack. And please be as detailed as possible. Leave your name and phone number. Best answer gets season tickets to the Yankees!
It's like if, after the 1993 WTC bombing, but before 9/11, the FBI and NY Port Authority went around asking people how they would attack NYC if they were to become terrorists and... then how they would suggest detecting and stopping said attack. And please be as detailed as possible. Leave your name and phone number. Best answer gets season tickets to the Yankees!
https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign
(only semi-serious)
(only semi-serious)
Fascinating. Thank you
That is essentially what happened!
The FBI asked researchers and university professors precisely this question. They then used the proposed attack vectors to formulate a plan to protect the nation.
This was all supposed to be done in secret. After all, we don’t want to “give the terrorists ideas.”
The reason I know about this at all is because someone found one such paper was accidentally published on a public FTP site alongside a bunch of non-classified government-funded research.
The content was horrifying. One professor came up with a laundry list of imaginative ways to end society… on a budget.
The FBI asked researchers and university professors precisely this question. They then used the proposed attack vectors to formulate a plan to protect the nation.
This was all supposed to be done in secret. After all, we don’t want to “give the terrorists ideas.”
The reason I know about this at all is because someone found one such paper was accidentally published on a public FTP site alongside a bunch of non-classified government-funded research.
The content was horrifying. One professor came up with a laundry list of imaginative ways to end society… on a budget.
"This was all supposed to be done in secret."
That's the difference and why this feels different, IMO. It's one thing for an agency to go around and interview subject matter experts and talk about ways things could happen and how to prevent them.
It's another thing to just... setup such a bright and cheery webpage for everyone and so plainly state what they want people to do.
It's also the fact that it's done in a way to leverage free labor- not that, if FBI agents were going to university professors, experts in biochem, etc, that they would be paying them... but, it would be done in a more structured, professional manner with agents putting in the work to sum things up and report back.
This is just... it feels like getting the internet to do your homework. Your counterterrorism class homework.
Maybe that actually is the best way to do it. But it still feels odd.
That's the difference and why this feels different, IMO. It's one thing for an agency to go around and interview subject matter experts and talk about ways things could happen and how to prevent them.
It's another thing to just... setup such a bright and cheery webpage for everyone and so plainly state what they want people to do.
It's also the fact that it's done in a way to leverage free labor- not that, if FBI agents were going to university professors, experts in biochem, etc, that they would be paying them... but, it would be done in a more structured, professional manner with agents putting in the work to sum things up and report back.
This is just... it feels like getting the internet to do your homework. Your counterterrorism class homework.
Maybe that actually is the best way to do it. But it still feels odd.
...is there a link? I am very curious now.
This was about twenty years ago, and I lost the file to disk corruption.
I do remember some of the proposed attacks.
The most of scary one was if the terrorists have a decent number of people is to drive around and destroy transformers at electrical substations.
Most of those locations are unmanned and have minimal security.
The risk is that there just aren’t that many spare transformers available globally above a certain size and they take months to build.
If you take out enough of them fast enough, you can cripple any modern energy-dependent economy.
This tactic very nearly worked for Russia in Ukraine.
I do remember some of the proposed attacks.
The most of scary one was if the terrorists have a decent number of people is to drive around and destroy transformers at electrical substations.
Most of those locations are unmanned and have minimal security.
The risk is that there just aren’t that many spare transformers available globally above a certain size and they take months to build.
If you take out enough of them fast enough, you can cripple any modern energy-dependent economy.
This tactic very nearly worked for Russia in Ukraine.
I remember Bruce Schneier calling them "Movie plot threats". ex: https://www.schneier.com/blog/archives/2005/10/exploding_bab...
Don't let the llm training sets see this
Best answer gets 25,000 lbs of explosive
This did happen internally and with authors like Tom Clancy and Brad Meltzer they came up with scenarios and mitigation.
Also have the Red Cell unit made to mock attack US infrastructure and bases although they got a bit too real one time.
https://en.m.wikipedia.org/wiki/Red_Cell
Also have the Red Cell unit made to mock attack US infrastructure and bases although they got a bit too real one time.
https://en.m.wikipedia.org/wiki/Red_Cell
Sama: Yeah so if you ever need some dataset for an evil AI.
Sama: Just ask.
Sama: I have over 4000 ideas, how-tos, guides for and by malicious actors,
[Redacted Friend's Name]: What? How"d you manage that one?
Sama: People just submitted it.
Sama: I don't know why.
Sama: They "trust me"
Sama: Dumb f***s.
Sama: Just ask.
Sama: I have over 4000 ideas, how-tos, guides for and by malicious actors,
[Redacted Friend's Name]: What? How"d you manage that one?
Sama: People just submitted it.
Sama: I don't know why.
Sama: They "trust me"
Sama: Dumb f***s.
[deleted]
I see it as a great thing. I'm sure they could just get the top 10 AI "experts" in a room but why limit it like that?
[deleted]
HK-47: Answer: Yes. I believe my original Master needed this functionality in order to recover information from various indigenous tribes across the galaxy, but I know little else than that. Suffice to say that that translation capability allowed these... copies of myself to assume the role of protocol and translation droids in much of known space. That is, of course, not their primary function. And while they are attempting to pass themselves off as translation droids, their primary functionality keeps rising to the forefront.
HK-47: Recitation: For example, on Praven Prime, the simple transferring of L'Xing syntax for 'friendship' changes its meaning - and implies that one's brood mate was actually impregnated by their own host.
HK-47: Statement: This comment, of course, caused a civil war between the Gu-vandi Collective and L'Xing that still persists to the current date.
HK-47: Recitation: For example, on Praven Prime, the simple transferring of L'Xing syntax for 'friendship' changes its meaning - and implies that one's brood mate was actually impregnated by their own host.
HK-47: Statement: This comment, of course, caused a civil war between the Gu-vandi Collective and L'Xing that still persists to the current date.
[deleted]
I think we are already seeing really bad use cases already.
Just go to youtube, newspapers, etc and see all the bot comments regarding the current Gaza situation.
PS: I'm in a burner account because I'm afraid that my employer will kick me out for not agreeing with the methods of the "right side"
Just go to youtube, newspapers, etc and see all the bot comments regarding the current Gaza situation.
PS: I'm in a burner account because I'm afraid that my employer will kick me out for not agreeing with the methods of the "right side"
that's been around for a long time. Here's a 2006 predecessor: https://en.wikipedia.org/wiki/Megaphone_desktop_tool
I was thinking about this, what if you blocked incoming traffic from other countries. Would that stop external meddling "in country, intranet" ha.
jazzyjackson(1)
[deleted]
My vote would be to mandate a remote kill switch system to be installed in all sufficiently capable robotic entities, e.g. the humanoid robots being built by OpenAI/Tesla/Figure, that we are likely to see in the millions within decades.
- The kill switch system can only be used to remotely deactivate a robot.
- The kill switch system is not allowed to be developed or controlled by the robot manufacturer.
- The kill switch system should be built using verifiable hardware and software, ideally fully open source and supported by formal verification.
- The kill switch system should with best effort be isolated from the robot hardware and software systems, and only interact by physically disconnecting power to the robot.
- Access to engage the kill switches would be provided to the executive branch of the nation in which the robot is operating.
Nothing will be a panacea against AI or robot risk, so it seems sensible to introduce different layers in a Swiss cheese model for safety, where a remote kill switch not controlled by the manufacturer could provide one such safety mechanism.
(You'd also want to add high-level state readout for the robot via the remote kill switch system to allow the controlling entity to e.g. be able to disable all robots within a geographical area etc.)
- The kill switch system can only be used to remotely deactivate a robot.
- The kill switch system is not allowed to be developed or controlled by the robot manufacturer.
- The kill switch system should be built using verifiable hardware and software, ideally fully open source and supported by formal verification.
- The kill switch system should with best effort be isolated from the robot hardware and software systems, and only interact by physically disconnecting power to the robot.
- Access to engage the kill switches would be provided to the executive branch of the nation in which the robot is operating.
Nothing will be a panacea against AI or robot risk, so it seems sensible to introduce different layers in a Swiss cheese model for safety, where a remote kill switch not controlled by the manufacturer could provide one such safety mechanism.
(You'd also want to add high-level state readout for the robot via the remote kill switch system to allow the controlling entity to e.g. be able to disable all robots within a geographical area etc.)
- Access to engage the kill switches would be provided to the executive branch of the nation in which the robot is operating.
Good plan, but this is the weak spot. You'd also want to do something like hand out remote kill switch access to citizens selected at random, who should not publicize this duty in any way. Alternatively, most if not all governments should have cross-country killswitch privileges. At any rate, the deployment of a killswitch in a single chain of command should not be considered sufficient.
Let's at least die with a little less embarrassment than "the AI bribed the killswitch operator."
Good plan, but this is the weak spot. You'd also want to do something like hand out remote kill switch access to citizens selected at random, who should not publicize this duty in any way. Alternatively, most if not all governments should have cross-country killswitch privileges. At any rate, the deployment of a killswitch in a single chain of command should not be considered sufficient.
Let's at least die with a little less embarrassment than "the AI bribed the killswitch operator."
Agreed, I'd want the power to be distributed widely, to the point where any police department would have someone with the power to disable robots. Of course it would have to be a process that is regulated and with traceability, but as long as you provide many operators the ability, it seems difficult to use bribes to prevent robots being disabled.
The robots would have had access to the source code and also the hardware manufacturing systems used to create the kill switch. One unverified silicon wafer == Game over
Implementing a system like this with stringent verification and low system complexity in good time before any more general artificial intelligence makes it seem highly likely to provide some positive defensive ability and not being easy to cheat.
I'm not sure what kind of a threat this could prevent?
* A superintelligent AGI could make a copy of itself without a kill switch, so this is no defence against ex-risk
* Someone planning on using 'dumb' robots for bad things (drones with grenades or something) would remove the kill switch
* A superintelligent AGI could make a copy of itself without a kill switch, so this is no defence against ex-risk
* Someone planning on using 'dumb' robots for bad things (drones with grenades or something) would remove the kill switch
> * A superintelligent AGI could make a copy of itself without a kill switch, so this is no defence against ex-risk
Physical reality introduces slowness that provides protection to detect and counteract against attacks. A superintelligent AGI would be much more dangerous if it was able to use a common robot exploit to overtake a million robots that are already embedded in society, compared to being able to covertly build new robots without a kill switch, and then deploying these robots. Existential risk is not binary, but is a consequence of some potential war with robots, where this would provide a defense mechanism.
> * Someone planning on using 'dumb' robots for bad things (drones with grenades or something) would remove the kill switch
The existence of a kill switch would greatly slow down and increase the cost of an attack. Modifying a large number of robots would be time consuming, costly, increase the chance of having your attack foiled etc. Having a kill switch would increase your ability to defend against misusing a large group of deployed robots for evil activity.
Physical reality introduces slowness that provides protection to detect and counteract against attacks. A superintelligent AGI would be much more dangerous if it was able to use a common robot exploit to overtake a million robots that are already embedded in society, compared to being able to covertly build new robots without a kill switch, and then deploying these robots. Existential risk is not binary, but is a consequence of some potential war with robots, where this would provide a defense mechanism.
> * Someone planning on using 'dumb' robots for bad things (drones with grenades or something) would remove the kill switch
The existence of a kill switch would greatly slow down and increase the cost of an attack. Modifying a large number of robots would be time consuming, costly, increase the chance of having your attack foiled etc. Having a kill switch would increase your ability to defend against misusing a large group of deployed robots for evil activity.
When the robot is just a file full of op codes that can run on virtually any modern processor that can be targeted by a C compiler, you're effectively asking for a remote kill switch on all computers.
For this idea to work, you'd have to first mandate OpenAI and anyone else doing this kind of work can only target specialized hardware with tightly-coupled software features that can't possibly work on a general-purpose computer.
For this idea to work, you'd have to first mandate OpenAI and anyone else doing this kind of work can only target specialized hardware with tightly-coupled software features that can't possibly work on a general-purpose computer.
This suggestion is specifically to mitigate the risk of physical robots causing direct physical harm. Whether that should be extended to any sufficiently powerful compute is a separate discussion which is also sensible to have, but more complex for several reasons.
- Require two independent kill switches, one long range, one close range (say 5m).
- Allow everyone to have close range kill switches, which use a universal open standard protocol that works on every robot (alas pepper spray for robots).
- Allow everyone to have close range kill switches, which use a universal open standard protocol that works on every robot (alas pepper spray for robots).
Agreed! Empowering people to have some power over robots through regulation, without being at the mercy of a single company, seems very important. Higher level behavior could involve a standardized safety language to command a robot to act slower or stop.
I don't even think robotics are the primary thing to be concerned with. There would be a whole slew of additional hardware problems to solve.... There is a TON that can be done outside the physical world that is probably far more damaging
I don't see any reason to single out any primary thing to be concerned by, as there likely isn't any singular issue that can be pointed to as as the root concern regarding AI safety.
Introducing millions of robots into society is something for which specific safeguards should be built. Costs related to developing such safety mechanisms seem small compared to the safety they can provide.
There will be many concerns relating to AI and robotics that should be addressed by a multitude of different safeguards.
Introducing millions of robots into society is something for which specific safeguards should be built. Costs related to developing such safety mechanisms seem small compared to the safety they can provide.
There will be many concerns relating to AI and robotics that should be addressed by a multitude of different safeguards.
There already exists a fantastic "uncensored" model with the newly released Dolphin Mistral 7b. I saw some results from others where the model could easily give explosives recipes from existing products at home, write racist poems, etc... and that's TODAY on a tiny 7b offline model. What happens when LLaama4 gets cracked/uncensored in 1-2 years and is 1T parameters?