Dual use of artificial-intelligence-powered drug discovery(nature.com)
nature.com
Dual use of artificial-intelligence-powered drug discovery
https://www.nature.com/articles/s42256-022-00465-9
70 comments
Quite an interesting paper in it's impact study, but the questions it asks all boil down to "the masses are not to be trusted with our vaunted knowledge, because harm can be done. How can we prevent them from moving outside the boundaries of knowledge we like" and then the example used is how GPT-3 is managed. This is without a doubt a terrific example of bad technological stewardship: Any malicious state actor gets to use what you make for evil, while at the same time keeping it away from the folks you're making it for. I wish they had had a more substantial and innovative thing to say about it, but as it stands it reads as regressive.
I disagree with your interpretation.
Would it be better if nobody had access to easier ways to produce new and innovative nerve agents? Yes, of course it would be better.
Do I prefer a world where only state actors can build nerve agents, vs a world that includes state actors plus more? Yes, less sources of nerve agents is more good.
You might as well complain how pointless it is to block proliferation of nuclear weapons when malicious state actors already have them.
The world wont be safer by spreading nuclear weapons to more people, and similarly for nerve agents.
Fortunately, this sounds similar to nuclear weapons in another way, designing the weapon is not the hard part that limits them to state actors, its producing them that's hard.
Still, nuclear powers don't tend to publish their weapon plans publicly, for good reason, and the same logic should apply here.
I'm not using the comparison to imply this is as serious as nuclear proliferation btw, just that it's a useful example of knowledge that really should be kept limited.
Would it be better if nobody had access to easier ways to produce new and innovative nerve agents? Yes, of course it would be better.
Do I prefer a world where only state actors can build nerve agents, vs a world that includes state actors plus more? Yes, less sources of nerve agents is more good.
You might as well complain how pointless it is to block proliferation of nuclear weapons when malicious state actors already have them.
The world wont be safer by spreading nuclear weapons to more people, and similarly for nerve agents.
Fortunately, this sounds similar to nuclear weapons in another way, designing the weapon is not the hard part that limits them to state actors, its producing them that's hard.
Still, nuclear powers don't tend to publish their weapon plans publicly, for good reason, and the same logic should apply here.
I'm not using the comparison to imply this is as serious as nuclear proliferation btw, just that it's a useful example of knowledge that really should be kept limited.
> "the masses are not to be trusted with our vaunted knowledge ..."
The masses will use AI to do bioweapon design? That doesn't sound like something most people could do.
The masses will use AI to do bioweapon design? That doesn't sound like something most people could do.
Most people also wouldn’t shoot up a school, yet here we are
Knowledge and resources necessary to shoot up a school: location of dad's AR-15, bullets, and how to load.
Knowledge and resources necessary to develop AI-proposed toxins: multiple PhDs, thousands of square feet (probably 10s to low 100s of thousands) of lab space, including nasty stuff, like bromides and heavy metals, and testing and evaluation of various delivery vehicles, ranging from canes with needles to air-launched warheads with atomizers.
Knowledge and resources necessary to develop AI-proposed toxins: multiple PhDs, thousands of square feet (probably 10s to low 100s of thousands) of lab space, including nasty stuff, like bromides and heavy metals, and testing and evaluation of various delivery vehicles, ranging from canes with needles to air-launched warheads with atomizers.
My point is that the 'masses' have no interest in this knowledge.
But ..
"Risk of misuse The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting."!!!
Operating in the rarefied air of 'science' they were unaware of the risk of misuse!
I don't think they can even conceive of the idea that the government might misuse this technology.
"Risk of misuse The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting."!!!
Operating in the rarefied air of 'science' they were unaware of the risk of misuse!
I don't think they can even conceive of the idea that the government might misuse this technology.
They wrote a paper on it. They weren't just aware, they were demonstrably aware. Or are you suggesting that we blame people for having a finite mental processing speed?
I'm saying its a funny to say that they were unaware of the risks when they started on the research. I mean, its not beyond imagination that AI could be/is being used for nefarious ends.
I think we agree...??
I think we agree...??
> Operation in the rarefied air of 'science' they were unaware of the risk of misuse!
The thundering irony of this concept in a timeline where a (group of) scientist(s) discovered fission cannot be overstated.
The thundering irony of this concept in a timeline where a (group of) scientist(s) discovered fission cannot be overstated.
The thundering irony of blaming the group of people who made you aware of a problem for being unaware of the problem is even more difficult to overstate.
[deleted]
Do you think everybody should have the right to design a Sars-Cov-4 in their kitchen just because it's soon going to be technologically possible and free speech?
From the paper:
> In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.
> In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.
This is more evidence for why I think the "Vulnerable World Hypothesis" is not only a real framework, but that it describes the most eminent danger to society. It's far more dangerous than nuclear MAD, in my opinion.
We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it.
You don't even have to attack humans. Our society is dependent on a lot of assumptions.
And just as VWH states, I don't think it can be defended against. It's scary.
We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it.
You don't even have to attack humans. Our society is dependent on a lot of assumptions.
And just as VWH states, I don't think it can be defended against. It's scary.
I don't even think you need a hostile actor, you could just have an incompetent actor who flips the sign on a value, and instead of designing a biological agent to maximize wheat production, designs one that minimizes wheat production, and ends up exterminating all global wheat.
To be fair, at least for your example, you'd need a really long chain of incompetent actors to kill off wheat.
We should really be scared of the lone and bored grad student making a wheat killer and accidentally dropping the test tube.
We should really be scared of the lone and bored grad student making a wheat killer and accidentally dropping the test tube.
I'd be more worried about someone developing a plastic-recycling microbe that works too well. There's already lots of viruses that attack plants, but the plants have immune systems for that.
> I'd be more worried about someone developing a plastic-recycling microbe that works too well.
See the 2007 novel Ill Wind by Kevin Anderson and Doug Beason for an exploration of this scenario:
https://books.google.com/books/about/Ill_Wind.html?id=wWIgO7...
See the 2007 novel Ill Wind by Kevin Anderson and Doug Beason for an exploration of this scenario:
https://books.google.com/books/about/Ill_Wind.html?id=wWIgO7...
Degrading plastics would be a nasty one.
On the topic of plants, instead of attacking crops directly, attack mycorrhiza. Plants can't get nutrients without them.
On the topic of plants, instead of attacking crops directly, attack mycorrhiza. Plants can't get nutrients without them.
It's not trivial to synthesize VX-like compounds without killing yourself. You would definitely need an "above average" educated/trained person.
That is really in the attack of captain obvious territory. Next we can expect - we turn our algorithm to find energy storage molecules and what we got is explosives.
Researchers seemed surprised by the results.
And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
This sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.
Freedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory.
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
The United States has had "born secret" laws about nuclear weapons details for decades [1], but it's unclear if they are actually compatible with the Constitution. There was a case in 1979 that could have established whether "born secret" nuclear information is a valid exception to the First Amendment, but the government dropped its case before the courts could rule on it [2]. The "born secret" concept does not apply to biological or chemical weapons.
Your nearest research university library probably holds publications about the effects and synthesis of chemical warfare agents, like Some aspects of the chemistry and toxic action of organic compounds containing phosphorus and fluorine by Bernard Charles Saunders: https://catalog.lib.uchicago.edu/vufind/Record/587946
Which is also scanned and available online: https://archive.org/details/B-001-026-884-ALL
[1] https://en.wikipedia.org/wiki/Born_secret
[2] https://en.wikipedia.org/wiki/United_States_v._Progressive,_....
Your nearest research university library probably holds publications about the effects and synthesis of chemical warfare agents, like Some aspects of the chemistry and toxic action of organic compounds containing phosphorus and fluorine by Bernard Charles Saunders: https://catalog.lib.uchicago.edu/vufind/Record/587946
Which is also scanned and available online: https://archive.org/details/B-001-026-884-ALL
[1] https://en.wikipedia.org/wiki/Born_secret
[2] https://en.wikipedia.org/wiki/United_States_v._Progressive,_....
Classification only applies to people who've agreed to keep the classified info secret; everyone else is protected by the 1st amendment. See United States v. Progressive, Inc. (which did _not_ rule this, as it was dismissed first.)
> It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Certainly. However, here they have identified and published not the discovered molecules but only the identification process (and only in fairly broad strokes, at that), and shied away from working further on synthesis (we know enough to state that synthesis would be possible and practical for nearly any candidate molecule).
Sure, as a society we can expand ethical guidelines to some sort of "thou shall not optimize for toxicity", but enforcing it poses some serious challenges, not least of which is the fact that this has demonstrated that the process works even with fairly innocuous and public training data.
Expect to see a bunch of funding for research into technologies around toxin identification and detection, as well as the rapid creation and synthesis of antitoxins and other prophylactic measures (cheaper filtering, maybe). Perhaps even eventual (as in a decade out at least) "hardening" of organisms against toxicological weapons (possibly positioned as research into the mechanisms of acquiring pesticide resistance and similar).
Certainly. However, here they have identified and published not the discovered molecules but only the identification process (and only in fairly broad strokes, at that), and shied away from working further on synthesis (we know enough to state that synthesis would be possible and practical for nearly any candidate molecule).
Sure, as a society we can expand ethical guidelines to some sort of "thou shall not optimize for toxicity", but enforcing it poses some serious challenges, not least of which is the fact that this has demonstrated that the process works even with fairly innocuous and public training data.
Expect to see a bunch of funding for research into technologies around toxin identification and detection, as well as the rapid creation and synthesis of antitoxins and other prophylactic measures (cheaper filtering, maybe). Perhaps even eventual (as in a decade out at least) "hardening" of organisms against toxicological weapons (possibly positioned as research into the mechanisms of acquiring pesticide resistance and similar).
> No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs.
I'm pretty sure that whatever a private individual figures out on his own is not "classified", since it's not a government secret. Is there actually a US law banning people from even talking about WMDs?
I'm pretty sure that whatever a private individual figures out on his own is not "classified", since it's not a government secret. Is there actually a US law banning people from even talking about WMDs?
Yeah but how is this going to be enforced?
Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them.
Is that enough?
Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them.
Is that enough?
No, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge.
Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
> classified speech
That applies to government employees, who willingly agree to be bound by classification. I think that if someone outside had the information, they could publish it (probably a very bad idea).
Also, the press has a right to publish classified information.
That applies to government employees, who willingly agree to be bound by classification. I think that if someone outside had the information, they could publish it (probably a very bad idea).
Also, the press has a right to publish classified information.
Right, just don't take the plea deal, make bail, get a lawyer that can appeal every motion, and stay solvent through the first appeals court, the supreme court, the remand back to trial court, and the subsequent ruling, hope the prosecutor is bored enough to avoid a retrail, and don't get into crime during those 8 years so you get your bail back, and hope you don't get tried at the state level
If you can actually afford your rights, be my guest!
If you can actually afford your rights, be my guest!
It is illegal to conspire to commit crimes, at least many of them.
No one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
Synthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents.
For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well.
Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated.
Someone else in the thread asked about next steps. Those would be good next steps.
For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well.
Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated.
Someone else in the thread asked about next steps. Those would be good next steps.
> there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well
The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
> There is a long list of deadly terrorist attacks using homemade explosives
But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
IIRC Germany experienced some rather horrific accidents in their facilities for manufacturing chemical weapons.
They've certainly given this some thought:
> We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
> We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
What do you mean, "permitted"? It's not illegal to run these sorts of computational jobs or publish the results.
I would rather than policymakers hear about the destructive potential of this technology before, and not after, an actual bad actor uses it.
[deleted]
There's a major difference between drafting and manufacturing.
[deleted]
Someone somewhere is now generating new psychoactive drug candidates.
Well, we can definitely hope so.
Try a nociception, drug design or pharmacy group at your local university. Chances are they’ve been doing this for years.
This is also currently being discussed here: https://news.ycombinator.com/item?id=30698803
(Top of the front page, 56 comments.)
(Top of the front page, 56 comments.)
...heh, can't believe it we're not yet at "Siri, use the TensorFlow 4.6 OmegaFold model to design potentially stable variants of COVID-25 with active BSE prionic inserts. ...DONE. Great, now place an order of the best generated sequence to the cheapest gene synthesis provider in <unregulared country x>. Air-mail delivery."
You should read "Oryx and Crake" by Atwood
Have you read the sequel? I've been debating picking it up.
Siri, print 10B vaccines
Unfortunately things are not that symmetrical... just as it is much easier to make a missile that can get through any given missile defense system than it was to make that defense system, it's many orders of magnitude easier/cheaper (if at all practically possible) to create stuff that causes an incurable lethal condition than it is to create an antidote. (And the root causes of it all are very likely quite fundamental and unbypassable, consequences of the 2nd law bubbling up the matter complexity chain...)
I'd advise against doing more search on these topics if you want to sleep well at night and/or if you don't think you're already on one "naughty list" or another ;)
I'd advise against doing more search on these topics if you want to sleep well at night and/or if you don't think you're already on one "naughty list" or another ;)
[deleted]
This particular issue, toxins, doesn't seem like a huge problem in practice.
You can already buy extremely potent toxins that also work against humans at the hardware store (some pesticides/insecticides/rat poisons).
New toxins might be interesting for covert operations, for example killing someone without being detected by a toxicological report, but this seem to have limited usage.
You can already buy extremely potent toxins that also work against humans at the hardware store (some pesticides/insecticides/rat poisons).
New toxins might be interesting for covert operations, for example killing someone without being detected by a toxicological report, but this seem to have limited usage.
I believe we are heading towards personal safety bubbles, where dangerous molecular detection and warning is done in realtime via smartphone or other personal device such as glasses or implant. It might take awhile coming, but that seems the natural conclusion to having supercomputers available for individuals. Would the detection be possible via advanced mmwave/field detection, or would it have to have a molecular mini lab system of some sort ala pcr? Great market opportunity here.
This seems inevitable because the rise of various threats is also inevitable via technology, and us humans like to be safe.
This seems inevitable because the rise of various threats is also inevitable via technology, and us humans like to be safe.
Honestly, it sounded too over-dramatic and did not offer meaningful actionable consequences. The actual data is thin and not surprising at all, and it's also unclear how difficult it would be to design such harmful compounds without AI. I thought any capable chemist could easily come up with hundreds of harmful substances, old and new. I guess the AI is removing the chemist from the equation?
Quantity is a quality all of its own.
Being able to identify many such compounds very quickly implies some new threat models that are a bit disturbing.
Of the 40000 substances identified, perhaps one of them has another interesting characteristic we've never seen in a toxic compound before? Perhaps such characteristics could be idenfified and filtered for, and perhaps someone could pin down something entirely unique and outside of any threat-assessment anyone's produced.
>and not surprising at all it's obvious in retrospect, but they claim that the threat vector had not been seriously concidered before, and I see no reason to doubt their assessment. Many things aren't surprising once they've been pointed out, but were missed for far too long beforehand, and I personally had never concidered this before, despite seeing multiple articles about AI-generated substances for positive purposes.
Being able to identify many such compounds very quickly implies some new threat models that are a bit disturbing.
Of the 40000 substances identified, perhaps one of them has another interesting characteristic we've never seen in a toxic compound before? Perhaps such characteristics could be idenfified and filtered for, and perhaps someone could pin down something entirely unique and outside of any threat-assessment anyone's produced.
>and not surprising at all it's obvious in retrospect, but they claim that the threat vector had not been seriously concidered before, and I see no reason to doubt their assessment. Many things aren't surprising once they've been pointed out, but were missed for far too long beforehand, and I personally had never concidered this before, despite seeing multiple articles about AI-generated substances for positive purposes.
For example, is one of the super VX alternatives a good candidate for a binary type delivery? Where two or more compounds react on deployment to create the toxic version. Now you can have people safely (for some value of safe when performing organic chemistry) producing the components in many more laboratories
It is trivial that you can flip a sign on any network optimizing for something good and now it’s optimizing for something bad…
But it is still interesting that the network generalized to these toxic nerve agents without having them in the training set
But it is still interesting that the network generalized to these toxic nerve agents without having them in the training set
Use of these drugs would go against the Geneva Protocol.
The people who would use them don't care about the Geneva Protocol.
> The people who would use them don't care about the Geneva Protocol.
Unless you mean that, by definition, anyone who violates the Geneva Conventions doesn't care about them, you are overlooking that most major militaries observe the Geneva Conventions to a great extent. They are cleverly written to make it in everyone's interests.
Unless you mean that, by definition, anyone who violates the Geneva Conventions doesn't care about them, you are overlooking that most major militaries observe the Geneva Conventions to a great extent. They are cleverly written to make it in everyone's interests.
[deleted]