Stephen Fry Warns About the Dangers of Voice Clones(jurgengravestein.substack.com)
jurgengravestein.substack.com
Stephen Fry Warns About the Dangers of Voice Clones
https://jurgengravestein.substack.com/p/stephen-fry-issues-a-stark-warning
81 comments
Good one. I will ask my mother when she's in doubt to tell an old joke of my dad (she to me or me to her), including word plays. We know all of these inside out, including the very way he'd tell these. Only insiders know these jokes, though my wife's father is from the same year and some jokes my dad had her dad had as well. It took some magic away, for sure, but he was still my dad who was chronically ill and kept using humor to deal with life. A gifted talent I don't seem to possess.
[deleted]
All of this can be resolved by adopting a wider principle of self ownership. Something society does not actually want to do on a wide scale because that amount of liberty often means people do with that liberty things we do not like and wish to bar them from
However a overriding principle of self ownership would mean I own my voice, I own my face, I own my body, I own my labor, I own my data, I own everything about me, and I ... and I alone get to control how that is used.
However a overriding principle of self ownership would mean I own my voice, I own my face, I own my body, I own my labor, I own my data, I own everything about me, and I ... and I alone get to control how that is used.
And yet this makes little to no sense on its own merit. Humans are not animals that are hatched from an egg ready to live as a rugged individual in the wilds. You are a social creature that learns everything you know by stealing bits of information from the world around you (or were you born with a sack of gold and you paid your parents to raise you?).
You look and act and sound like thousands of others out there. You bleed information into the wider world around you, and now you seek to restrict others access to it.
You look and act and sound like thousands of others out there. You bleed information into the wider world around you, and now you seek to restrict others access to it.
>>You are a social creature
In actuality I am an anti-social creature, I am contrarian, and very ODD
>>that learns everything you know by stealing bits of information from the world around you
No primary I learn everything I know by accessing information freely given to me by others in the world around me via voluntary exchange. Either freely accessible knowledge people put out for all to access, or by exchanging currency or other value with the holder of said knowledge for which they share that knowledge with me
In actuality I am an anti-social creature, I am contrarian, and very ODD
>>that learns everything you know by stealing bits of information from the world around you
No primary I learn everything I know by accessing information freely given to me by others in the world around me via voluntary exchange. Either freely accessible knowledge people put out for all to access, or by exchanging currency or other value with the holder of said knowledge for which they share that knowledge with me
>In actuality I am an anti-social creature,
"Libertarians are like house cats: absolutely convinced of their fierce independence while utterly dependent on a system they don't appreciate or understand."
"Libertarians are like house cats: absolutely convinced of their fierce independence while utterly dependent on a system they don't appreciate or understand."
I don't understand how this solves the issue of others wanting to own you. People often want this, and the only way to achieve this is not to convince people they should have self-ownership because that does not preclude people owning things that are not theirs.
It would be in the system of laws
We do not have this self ownership principle in our system of laws. It should be something on the order of the US 1st Amendment in weight and scope codified in the very foundation of the legal system
We do not have this self ownership principle in our system of laws. It should be something on the order of the US 1st Amendment in weight and scope codified in the very foundation of the legal system
[deleted]
'I know, let's make a technology that is pretty much only useful to criminals illegal' - just FYI, they aren't going to follow the law.
I've played with voice cloners extensively and my view is that, except for very rare voices, they are repeated in different people - sometimes with the same intonation and prosody - even originating from different languages.
Who owns your voice? Well, who owns your face? Such things are covered by copyright law. You dont have total control over your face. I can use Fry's face in a parody film or critisism of him as a person. Even the cited article uses an image of his face (the embedded youtube clip). The rules for parody, critisim and nominative uses go far.
Deepfakes have already been to the supreme court (see Hustler v. Falwell). Cutting back on such uses risks cutting deeply into freedom of speech. As with Falwell, the answers involve educating the public so that they can tell fake from real, not giving famous people the ability to blackball every work they feel too similar to themselves. Lots of people talk like Stephen Fry. He doesnt get to own that accent or mode of speech just because he is the most famous any more than Lindsay Lohan can own blonde valley girl (she sued gta V over that image).
Deepfakes have already been to the supreme court (see Hustler v. Falwell). Cutting back on such uses risks cutting deeply into freedom of speech. As with Falwell, the answers involve educating the public so that they can tell fake from real, not giving famous people the ability to blackball every work they feel too similar to themselves. Lots of people talk like Stephen Fry. He doesnt get to own that accent or mode of speech just because he is the most famous any more than Lindsay Lohan can own blonde valley girl (she sued gta V over that image).
> the answers involve educating the public so that they can tell fake from real
At some point, it will become impossible to tell the difference.
At some point, it will become impossible to tell the difference.
I wonder if at that point people will rightfully lose confidence in the authenticity of digital media, or if people are doomed to believe what they see...
Or famous people can maintain authoritative lists of things they have done and said. Something as simple as Imdb can do much in debunking fakery.
In principle, any digital artifact may be digitally signed, just like web pages are (using CA's).
That way, you can set up a chain of trust, to track where some information is coming from.
This works as long as there is at least a top level signature provider that is uncorrupted. That will be hard in authoritarian societies and maybe even harder in radically polarized societies. Imagine a world where the CPP, DNC and GOP (and others) each controlled their own CA's, and had the ability to present a fake view of reality to their supporters.
That way, you can set up a chain of trust, to track where some information is coming from.
This works as long as there is at least a top level signature provider that is uncorrupted. That will be hard in authoritarian societies and maybe even harder in radically polarized societies. Imagine a world where the CPP, DNC and GOP (and others) each controlled their own CA's, and had the ability to present a fake view of reality to their supporters.
There’s information about me on IMDb that isn’t accurate. Not once ever have I added information about me on IMDb, so where exactly is this authoritative information on IMDb coming from? Requests by me to have information corrected goes no where. IMDb as an authoritative source is pure folly at best
Something as simple as IMBD, not literally IMDB. It is possible for famous people to maintain curated lists of their official work. Presidents especially do this, eventually putting it all into presidential libraries. Or it isn't possible to verify any information online because [reasons] and then our entire society falls apart as nothing online can ever be trusted. We aren't anywhere near that point.
> Or famous people can maintain authoritative lists of things they have done and said.
Which can easily omit things they did say but wish they hadn't.
Which can easily omit things they did say but wish they hadn't.
Why differentiate between digital and offline media do you believe the gro isn't produced digitally?
"Digital" doesn't mean anything more.
I live in Japan where Instax instant camera are super popular. Fujifilm makes more money off of making and selling Polaroid-style analog instant film than any single other business. The Impossible Project is cute and fun for hipsters but Instax has been there making film all along.
But they realized that young women today aren't happy how they look, and they want to use Instagram-style beauty filters on themselves, so they started selling Instax Bluetooth printers. They take analog instant film cartridges, but they use an LCD to imprint the photo onto them. So now you get the nostalgia of Polaroid photos, but you can edit your face as much as you want. Is this digital or analog?
I live in Japan where Instax instant camera are super popular. Fujifilm makes more money off of making and selling Polaroid-style analog instant film than any single other business. The Impossible Project is cute and fun for hipsters but Instax has been there making film all along.
But they realized that young women today aren't happy how they look, and they want to use Instagram-style beauty filters on themselves, so they started selling Instax Bluetooth printers. They take analog instant film cartridges, but they use an LCD to imprint the photo onto them. So now you get the nostalgia of Polaroid photos, but you can edit your face as much as you want. Is this digital or analog?
>> use an LCD to imprint the photo onto them.
It's just blurry digital. The polaroid film is acting as a really lossy medium for the digital information. Look close enough and you will likely still find artifacts of digital photography in the "film" printout.
It's just blurry digital. The polaroid film is acting as a really lossy medium for the digital information. Look close enough and you will likely still find artifacts of digital photography in the "film" printout.
The real argument is around degrees of reconstruction in image. Cameras never depict light fully faithfully in the analog world.
And even in the fully analog days there was plenty of doctoring of photos, it just wasn't as accessible to the common man without the equipment. Famously: https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...
This is one use case I can see for immutable and distributed public ledgers aka blockchains.
By having a publicly associated wallet which can sign the release/link/buffer of an audio file one would be able to assure legitimacy of the output and the rest of the world can carry on without worrying they are being bamboozled by a deepfake.
However, blockchains have been going so deep down the path of cryptocurrencies and get rich quick schemes that their real use cases are being ignored for random tokens.
The blockchain I envision being used for this purpose I stated above would have no token, it would just be… a blockchain.
I guess you could have the ‘mining/POW’ part take place during the signing of the data and would be done by the signee.
¯\_(ツ)_/¯
By having a publicly associated wallet which can sign the release/link/buffer of an audio file one would be able to assure legitimacy of the output and the rest of the world can carry on without worrying they are being bamboozled by a deepfake.
However, blockchains have been going so deep down the path of cryptocurrencies and get rich quick schemes that their real use cases are being ignored for random tokens.
The blockchain I envision being used for this purpose I stated above would have no token, it would just be… a blockchain.
I guess you could have the ‘mining/POW’ part take place during the signing of the data and would be done by the signee.
¯\_(ツ)_/¯
Genuinely curious -- what does a blockchain provide here that a normal public key signing doesn't?
To be honest the main benefit I can see is that the entire DB would be publicly distributed among a large number of independent users which would make the data a lot harder to falsify/hack/corrupt.
However, any system which can achieve the same goal is perfectly valid. This was just my initial thought — probably from my large exposure to blockchains etc
However, any system which can achieve the same goal is perfectly valid. This was just my initial thought — probably from my large exposure to blockchains etc
Timestamping and future immutability.
A boost to GPU sales.
You mean all those Warhammer 40k lore videos with David Attenborough weren't legit? :)
It is crazy if you need a good example of what Stephen Fry id talking about look at the "Attenborough Lore" channel. I honestly can't tell other than I doubt he'd do voice overs for dozens of 40k videos and the narrator is listed as "AI".
It is crazy if you need a good example of what Stephen Fry id talking about look at the "Attenborough Lore" channel. I honestly can't tell other than I doubt he'd do voice overs for dozens of 40k videos and the narrator is listed as "AI".
Attenborough is an interesting example because he had a real life near-analogue. He had a famous brother, Richard Attenborough (John Hammond in Jurassic Park). Their voices were extremely similar and they of course looked much the same. David is more famous now but back in the 1990s Richard was more recognizable. Who owns the voice? Does it default to the person who is most famous at a given moment? How often to we revisit that? Do the brothers battle it out in court to see which is allowed and which forbidden to do voiceovers using the "Attenborough" voice? And the Olsen sisters... that's a legal quagmire.
> I can use Fry's face in a parody film or critisism of him as a person.
This might feel like nitpicking but given the topic... there are places where there is no notion of 'fair use' like this. There are a whole variety of answers to the question of how much of your identity is yours (and how much is even separable from yourself). One shouldn't take whatever IP regime you're familiar with as the only possible truth, cuz there’s a world of possibilities
This might feel like nitpicking but given the topic... there are places where there is no notion of 'fair use' like this. There are a whole variety of answers to the question of how much of your identity is yours (and how much is even separable from yourself). One shouldn't take whatever IP regime you're familiar with as the only possible truth, cuz there’s a world of possibilities
Surely, who "owns a face" is not governed by copyright law because a face is not an intentionally created work of art, or anything similar?
There is “right of publicity”.
If I’m right, if you take a photo of Leonardo DiCaprio you can’t just use it in a commercial to promote a product without his permission even though you own the copyright of the photo.
Nor can you place an actor in a movie without their permission - they did that with Crispin Glover in Back to the Future 2, he sued the studio and I believe won or the studio settled.
If I’m right, if you take a photo of Leonardo DiCaprio you can’t just use it in a commercial to promote a product without his permission even though you own the copyright of the photo.
Nor can you place an actor in a movie without their permission - they did that with Crispin Glover in Back to the Future 2, he sued the studio and I believe won or the studio settled.
What if you found some random joe that looks just like DiCaprio, and paid him $100 to use his likeness in a movie?
That's always been possible, but in the past it would be clear that random Joe isn't a good actor and DiCaprio is. But if the whole thing is CGI anyway, you can make him as good or bad of an actor as you want.
That's always been possible, but in the past it would be clear that random Joe isn't a good actor and DiCaprio is. But if the whole thing is CGI anyway, you can make him as good or bad of an actor as you want.
One might argue that this would be no different from some artist (or AI) painting a painting in the style of Picasso. As long as you make no attempt at passing it off as an original Picasso, it may be seen as ok, unless we extend intellectual property rights to cover style....
... which we do already do, of course, if the protected item is something like an iPhone.
In both cases, though, if you falsely claim that the actor/phone IS the real thing, then it's clearly forgery.
And, I suppose, if you specifically bake too much of a similarity into it (ie a phone that looks exactly like an iphone, except the logo), one may argue that the likness effectively misleads the same way a forgery would, and that it can be illegal for that reason.
... which we do already do, of course, if the protected item is something like an iPhone.
In both cases, though, if you falsely claim that the actor/phone IS the real thing, then it's clearly forgery.
And, I suppose, if you specifically bake too much of a similarity into it (ie a phone that looks exactly like an iphone, except the logo), one may argue that the likness effectively misleads the same way a forgery would, and that it can be illegal for that reason.
DiCaprio can probably sue you and win especially if you are deliberately trying to misrepresent his involvement in your project.
As mentioned in my previous post, in Back to the Future 2, the studio hired another actor to play Crispin Glover's role by wearing a mask of his face to look like him. Glover sued and got a nice payout.
This would not have been a problem if they just recasted the role like they did with Claudia Wells (who was replaced by Elisabeth Shue).
As mentioned in my previous post, in Back to the Future 2, the studio hired another actor to play Crispin Glover's role by wearing a mask of his face to look like him. Glover sued and got a nice payout.
This would not have been a problem if they just recasted the role like they did with Claudia Wells (who was replaced by Elisabeth Shue).
There's a movie Beach Babes from Beyond (not great wouldn't recommend) with Stallone, Swayze, Travolta, and Sheen in it.
Unfortunately that's Jackie, Don, Joey,and Joe instead of Sylvester, Patrick, John, and Martin.
So it's no surefire thing.
Unfortunately that's Jackie, Don, Joey,and Joe instead of Sylvester, Patrick, John, and Martin.
So it's no surefire thing.
You could do that but your cast will not include the name DiCaprio, which is usually a big thing that draws viewership.
Tell that to all the celebs that take great efforts to manage and alter their faces. Plastic surgery, exercise and healthy living do alter and "create" faces. Tell that to makeup artists who get payed to paint faces.
[deleted]
Natural voices and faces don't seem like copyrightable works. A face resulting from plastic surgery, or a voice you made up for a role; they seem like they could be copyrightable, though also they seem closely analogous too materials you use to create a work, rather than a work in themselves.
> Lots of people talk like Stephen Fry
Difference between "someone sounds like Fry" and "have trained a model on his voice and hoped he won't notice". In second version his rights to his voice would be violated
Difference between "someone sounds like Fry" and "have trained a model on his voice and hoped he won't notice". In second version his rights to his voice would be violated
So i have to stop doing my Morgan Freeman impersonation at parties? That impersonation has been trained on his movies and speeches.
Doing an impression of someone in private, for fun, with no reasonable possibility of being confused for the original… seems very different to creating voice model outputs with the intent to deceive for financial or political gain.
This is something I feel like gets forgotten in this AI debates too often.
It matters not what you could potentially do with a tool but rather what you actually did with the tool and the consequences of what you did. "Fraud" charges already can capture a lot of these things without additional regulating around what models you can legally train in the first place.
It matters not what you could potentially do with a tool but rather what you actually did with the tool and the consequences of what you did. "Fraud" charges already can capture a lot of these things without additional regulating around what models you can legally train in the first place.
The Late Late Show with Craig Ferguson had an on-stage robot voiced by a Morgan Freeman impersonation, a very good one. That show was created to make money and I am sure that a great many people up that late were occasionally fooled by the impersonation. Saturday night live had Tina Fay impersonate Sarah Palin. She did it so well that it even fooled the real Sarah Palin's media team into using images from SNL as images of the actual candidate. Saturday Night live created this fake for profit and, arguably, for political reasons.
https://www.smh.com.au/entertainment/tv-and-radio/fox-news-m...
https://www.smh.com.au/entertainment/tv-and-radio/fox-news-m...
I imagine the comedic context made it clear there was no intent to deceive.
Presumably if you were saying you were Morgan Freeman and acting on behalf of Morgan Freeman and damaging the image of Morgan Freeman, I would imagine you would be asked to stop rather quickly yes.
If your party trick is to call Morgan Freeman's bank and wire a few thousand dollars to the birthday boy^Wperson, yes you should stop doing that.
If that works, he really needs to switch to a better bank.
Fun at a party is not running a commercial ML system. Maybe when ML model starts having fun at parties you can revisit this question.
I read your comment in his voice.
Don't actors trademark their likeness because copyright isn't sufficient?
Yes, but trademark has very similar rules to copyright. You can use trademarks in much the same way, such as saying "I hate Coca Cola" (nominative use) or commenting "This security vulnerability brought to you by Windows Vista, the most secure operating system ever" (parody use).
The use of their likeness is protected under “right of publicity” if I’m correct.
Could you cite statute in whatever jurisdiction you're referring to please.
Not a lawyer so don't speak lawyer but if you googled "right of publicity" a few well known cases come up like Vanna White vs Samsung. Vanna White won.
There was also the case of Crispin Glover in Back to the Future 2. Glover also got paid.
There was also the case of Crispin Glover in Back to the Future 2. Glover also got paid.
Not GP, so I don’t know what jurisdiction he was imagining. But, the ‘right to publicity’ is a jurisprudential invention (1958), but it has since been codified in maybe half the states in the US. The case used most frequently in law school is Carson v. Here’s Johnny Portable Toilets, Inc (Carson was found to have right to his catchphrase even without a trademark). For a statute for instance in California Civil Code 3344 (which protects natural living persons’ likeness, voice, photographs, etc.
— Also, none of the above is legal advice.
— Also, none of the above is legal advice.
or the danger of photoshop... I think we managed to live with the idea that you can't trust a random photo fairly well. AI is just doing that to audio (which was never very reliable anyway) and video.
Which is healthy, people are too trusting of videos. It is particularly easy to manipulate people in a video by adding music, selective editing / context omission, etc.
Which is healthy, people are too trusting of videos. It is particularly easy to manipulate people in a video by adding music, selective editing / context omission, etc.
I believe trademark is the best framework for protection here. A voice is part of a brand. If you imply that your product is from a specific brand then you may be in violation of their rights. I cannot add a swoosh to my new shoes if it too closely resembles the Nike brand because it is both false advertising and a trademark violation. No one should be able to use a specific voice without appropriate permission.
Unfortunately the ultimate test here is "confusion." Will a reasonable person be confused as to the origin of a product. Satire must be sufficiently obvious to receive protection and branding sufficiently different as to not cause confusion.
These tests are well established in the visual realm, they need to be consistently applied to audio as well.
Unfortunately the ultimate test here is "confusion." Will a reasonable person be confused as to the origin of a product. Satire must be sufficiently obvious to receive protection and branding sufficiently different as to not cause confusion.
These tests are well established in the visual realm, they need to be consistently applied to audio as well.
> No one should be able to use a specific voice without appropriate permission.
You can’t use their name or endorsement without permission. The challenge is if I do an impression and never say “this is Stephen Fry” then I’m not using their likeness, I’m just emulating.
Does Oasis need the Beatles permission for sounding like them? No. But if they performed as the Beatles, then that’s a problem.
You can’t use their name or endorsement without permission. The challenge is if I do an impression and never say “this is Stephen Fry” then I’m not using their likeness, I’m just emulating.
Does Oasis need the Beatles permission for sounding like them? No. But if they performed as the Beatles, then that’s a problem.
> No one should be able to use a specific voice
What is this thing "specific voice"? Is there a scientific definition, something beyond them just sounding similar to a layperson? There are literally billions of voices on this planet. And a person's voice changes drastically across their lifetime. Any such rules would need exact definitions with enough granularity of data to ensure nobody gains rights over the "specific voice" owned by some other person.
What is this thing "specific voice"? Is there a scientific definition, something beyond them just sounding similar to a layperson? There are literally billions of voices on this planet. And a person's voice changes drastically across their lifetime. Any such rules would need exact definitions with enough granularity of data to ensure nobody gains rights over the "specific voice" owned by some other person.
The post you're replying to answered your question.
>Unfortunately the ultimate test here is "confusion." Will a reasonable person be confused as to the origin of a product. Satire must be sufficiently obvious to receive protection and branding sufficiently different as to not cause confusion.
>Unfortunately the ultimate test here is "confusion." Will a reasonable person be confused as to the origin of a product. Satire must be sufficiently obvious to receive protection and branding sufficiently different as to not cause confusion.
It makes me really sad to be contributing to these “advances” in tech. Should actually create an exit plan and find something else to do.
You could take solace in this part from the end, maybe?
“Ironically, perhaps, it is AI, for all it’s threats, that will offer the best chance for a solution to itself. And, it is to be hoped, to the climate apocalypse, too.”
“Ironically, perhaps, it is AI, for all it’s threats, that will offer the best chance for a solution to itself. And, it is to be hoped, to the climate apocalypse, too.”
I think we should less worried about people believing deep-fakes are real and more worried that politicians (and others) will be able to claim things which they actually said are deep fakes.
That depends on the relative frequency of the two modes. If 90% of videos of each precidential candidate were a deepfake good enough that you cannot tell if it's real or not, then that is a worse problem than the possibility that one real video can be denied.
In both cases, though, the solution would be to find a way to allow people to tell deepfakes from real videos. For instance by having some sort of certificate provider agencies digitally sign them.
This would still make it hard to tell a deepfake from an unsigned video, of course.
In both cases, though, the solution would be to find a way to allow people to tell deepfakes from real videos. For instance by having some sort of certificate provider agencies digitally sign them.
This would still make it hard to tell a deepfake from an unsigned video, of course.
Another good premise for an article written in the appallingly casual style I’ve come to expect from tech writers. I switched off after “no more”.
It is an interesting point though - do I own the copyright to my own voice? What if it’s modulated slightly to be “close enough”, but not a direct copy?
It is an interesting point though - do I own the copyright to my own voice? What if it’s modulated slightly to be “close enough”, but not a direct copy?
Anyone can paint an image in the style of van Gogh. But if you then try create perception that it's an original van Gogh, you are guilty of forgery.
The same goes for a signature. Anyone can write the name of someone like Elon Musk in a way that resembles his signature. But if you write it in the signature field of a contract, for instance, then it is also forgery.
One can discuss whether people should be free to create deepfake audio and/or video of other people, including politicians and celebrities. But if you do so for the purpose of benefitting by people falsely identifying the deepfake/voice clone as coming from the real person, it should also be handled as a forgery, just as a piece of art or a signature.
The same goes for a signature. Anyone can write the name of someone like Elon Musk in a way that resembles his signature. But if you write it in the signature field of a contract, for instance, then it is also forgery.
One can discuss whether people should be free to create deepfake audio and/or video of other people, including politicians and celebrities. But if you do so for the purpose of benefitting by people falsely identifying the deepfake/voice clone as coming from the real person, it should also be handled as a forgery, just as a piece of art or a signature.
The world is typically filled with more complicated scenarios than this...
It's not "can you sign like Elon Musk" it's "What if you make a machine that can perfectly sign Elon Musk's name then someone else uses it to forge documents, do you have legal liability in some jurisdictions?"
It's not "can you sign like Elon Musk" it's "What if you make a machine that can perfectly sign Elon Musk's name then someone else uses it to forge documents, do you have legal liability in some jurisdictions?"
There will always be gray zones, and there always have been. Otherwise legal actions can be illegal if the intent is to support a crime.
It's legal to drive your friend to the bank. But not if you know she intends to rob it.
It's legal to drive your friend to the bank. But not if you know she intends to rob it.
Oh I thought this was about Futurama…
Perhaps everybody in the world needs some sort of signing key, so they can officially "authorize" public media that contains their likeness.
...Or does he?
Danger to his bank account?
g232089(1)
i'll warn you about the danger of stephen fry with just as much ferocity
I also taught my whole family a passphrase to verify that any call “from family” is actually that family member and not a shakedown scam.
Super easy precautions against really painful consequences.