AI voice actors sound more human than ever–and are ready to hire(technologyreview.com)
technologyreview.com
AI voice actors sound more human than ever–and are ready to hire
https://www.technologyreview.com/2021/07/09/1028140/ai-voice-actors-sound-human/
44 comments
If you haven't heard it yet, 15.ai had a ton of voices. IIRC it was a single person team, and they were working on an mechanism to build a decent voice model from minimal training.
They had trained on just about every character in "My Little Pony: Friendship is magic" and various characters from other games (Portal 2, etc), and it was remarkably good.
It's been down for months though, claiming that they're having a hard time training the latest iteration of the model :-/
They had trained on just about every character in "My Little Pony: Friendship is magic" and various characters from other games (Portal 2, etc), and it was remarkably good.
It's been down for months though, claiming that they're having a hard time training the latest iteration of the model :-/
How does this compare with Vocaloid?[1]
[1] http://www.vocaloid.com/en/
[1] http://www.vocaloid.com/en/
It's probably better to compare it with Voiceroid, which focuses on plain TTS than singing.
Here is one of the characters that is available in both Vocaloid and Voiceroid.
https://www.ah-soft.com/voiceroid/yukari/index.html
You can see there are 6 tunables. The first three, "Speed", "Pitch", "Intonation intensity", seems pretty standard to TTS engines.
For "Anger", "Sadness" and "Joyfulness" though, I think very few other engines provides settings to those. (Can anyone else name one?)
The point is there is no one right way to speak a sentence. There is way more context than the text itself. Like the use of strange intonation in synthesized train announcements used in Japan to increate legibility in noisy environments, as another commenter mentioned.
Here is one of the characters that is available in both Vocaloid and Voiceroid.
https://www.ah-soft.com/voiceroid/yukari/index.html
You can see there are 6 tunables. The first three, "Speed", "Pitch", "Intonation intensity", seems pretty standard to TTS engines.
For "Anger", "Sadness" and "Joyfulness" though, I think very few other engines provides settings to those. (Can anyone else name one?)
The point is there is no one right way to speak a sentence. There is way more context than the text itself. Like the use of strange intonation in synthesized train announcements used in Japan to increate legibility in noisy environments, as another commenter mentioned.
Nope, still sounds artificial to me. Inflections are all over the map on the 2 naration samples and make it sound like bad editing. The pacing is just odd in the VocalID ad samples.
Sorry, I might be a bit biased as I know several voice actors, and have routinely work with vocal tracks in the edit bay. We would be having a field day with the jokes if asked to use any of this.
Sorry, I might be a bit biased as I know several voice actors, and have routinely work with vocal tracks in the edit bay. We would be having a field day with the jokes if asked to use any of this.
Totally agree. I have 0 experience in this but they sound super robotic to me. No character, no charm, no excitement or emotion or anything human. No ummmms or uhs or laughs or pauses to breathe.
It's a better robot voice sure, why not sell it as such? Selling this as human is harmful to the industry reputation and also plainly ridiculous.
It's a better robot voice sure, why not sell it as such? Selling this as human is harmful to the industry reputation and also plainly ridiculous.
https://www.sonantic.io/ voices which Obsidian used in The Outer Worlds sound pretty decent to me. I wonder, would that be affordable in a small indie project?
They don’t offer any pricing on their website, I wonder how much the stuff cost.
AI appears to be more going after creative work then it does relieving us of the mundane.
Yes! I guess this is sort of the opposite of what people would normally expect, but I assume it's largely because an algorithmic mistake won't kill someone such as in self driving cars
This shines in the mundane text that is still useful read aloud. For example, the Financial Times uses an AI voice actor to read their articles as soon as they're published. It's not as smooth as a professional human, but it doesn't need to be if I just want to listen to a news analysis instead of read it by eye.
A few years ago, I noticed that train companies in Japan had started using synthesized voices for some announcements on trains and at stations. (Probably the same thing is happening in other places, but I haven't travelled outside Japan for a long time.) The voices were clearly artificial and sometimes made ridiculous mistakes—such as reading the name of the train company JR East in English as “junior east”—but over time I realized that they also had advantages over prerecorded human voices.
Once, I was riding a two-car local train through a rural part of Kyushu when the train was delayed by heavy rain up ahead. Soon there were audio announcements with artificial voices about the delay in Japanese, English, Chinese, and Korean. Both the Japanese and English sounded a bit strange—the English especially because it was clearly machine translated—but the meaning was clear and they served their purpose. (I couldn't assess the Chinese or Korean.)
The pitch of the artificial voices must have been chosen for good audibility in a noisy environment, as they were easier to understand than most live announcements by human conductors on trains.
Professional human voice actors certainly would have been better, but there is no way they could have been arranged at a reasonable cost for up-to-the-minute train delay announcements in even one language, let alone four.
Once, I was riding a two-car local train through a rural part of Kyushu when the train was delayed by heavy rain up ahead. Soon there were audio announcements with artificial voices about the delay in Japanese, English, Chinese, and Korean. Both the Japanese and English sounded a bit strange—the English especially because it was clearly machine translated—but the meaning was clear and they served their purpose. (I couldn't assess the Chinese or Korean.)
The pitch of the artificial voices must have been chosen for good audibility in a noisy environment, as they were easier to understand than most live announcements by human conductors on trains.
Professional human voice actors certainly would have been better, but there is no way they could have been arranged at a reasonable cost for up-to-the-minute train delay announcements in even one language, let alone four.
I had the same sort of situation the other day. I wanted to produce a "corporate video" for our company that explained our ontology, and it's benefits. Using Blender I managed to come to a semi-pro result, but getting the narrating voice was not quite so easy. I didn't have access to a good mic, and my French / Dutch accent would have distracted and lowered the production value.
In my opnioning this is a good example of how AI is slowly gaining foot. It's a bit like the old adage of a camera phone vs a DSLR. Sure, it's inferior, but it's always better than a superior version that I don't have access to. It's difficult to compete on quality with something that's free.
In my opnioning this is a good example of how AI is slowly gaining foot. It's a bit like the old adage of a camera phone vs a DSLR. Sure, it's inferior, but it's always better than a superior version that I don't have access to. It's difficult to compete on quality with something that's free.
> it's inferior, but it's always better than a superior version that I don't have access to.
Yes. Machine translation is like that, too. It makes mistakes and can be hard to understand and easily misunderstood, but compared with the time and effort required to learn another language, or the expense and bother of finding a human translator or interpreter, it clearly has its advantages.
Yes. Machine translation is like that, too. It makes mistakes and can be hard to understand and easily misunderstood, but compared with the time and effort required to learn another language, or the expense and bother of finding a human translator or interpreter, it clearly has its advantages.
Wow. That's a very good use case I would never have thought of.
It's those unexpected use cases that often become the killer app of advancements like this.
It's those unexpected use cases that often become the killer app of advancements like this.
I'm pretty sure Amtrak/VRE in the US uses synthesized voices at some stations, the one at RVR sounds almost exactly like GLaDOS.
[deleted]
They'll get it right. Eventually.
10 years? 50 years? 100 years?
Only a matter of time now.
10 years? 50 years? 100 years?
Only a matter of time now.
I’m waiting impatiently for an accessible musical vocal synthesizer in a hardware box. I.e.: one that would have both piano and computer keyboards, or midi and usb inputs.
There are so many hardware lead/bass/drum synthesizers being released these days, and hardly any (hardware) vocal synthesizers that I know of.
There are so many hardware lead/bass/drum synthesizers being released these days, and hardly any (hardware) vocal synthesizers that I know of.
I've been working in this area for a little over a year, so perhaps the artifacts in the generated speech are more apparent to me than to others. With that said, I don't find the displayed voices to be very compelling. All of them have artifacts which are apparent on a very localized scale - similar to compression artifacts, but I believe they are better described as phase discontinuities which result from the types of neural networks used to generate the waveforms in parallel. If you see a speech synthesis startup adding backing tracks to their speech samples, it's to cover up these artifacts. Removing these artifacts and improving the audio fidelity is a solvable problem, and you can look at the latest research for promising approaches. However, improving the naturalness and expressiveness of the speech at a high level is much further from being solved. I'm not caught up with the literature in this area entirely, but I have yet to see any examples of expressive text-to-speech that has natural intonation and prosody. The TTS researchers at Google have a bunch of research[1] which has essentially solved the task of monotone voice-assistant text-to-speech, but they haven't had as much success when it comes to more expressive speech. Best I've seen on this front is from an older paper[2], but the audio fidelity is quite poor there so it still leaves a lot to be desired.
[1] https://google.github.io/tacotron/
[2] https://audio-samples.github.io/#section-4
[1] https://google.github.io/tacotron/
[2] https://audio-samples.github.io/#section-4
All of these sound really good to me, I think your ear being trained on these issues has a lot to do with it. We're very close to the point of it being indistinguishable from human speech, in fact with a few of these short clips I wouldn't be able to tell if I didn't know it's artificial speech.
All recordings are of low fidelity, like they sampled someone talking over GSM over voip over analog.
I heard the artifacts too and was not impressed. I do not work with that industry.
> Several now use a profit-sharing model to pay actors every time a client licenses their specific synthetic voice, which has opened up a new stream of passive income.
Sounds pretty bad. This means we are going to make it artificially expensive for no other good reason than rent-seeking. It's not like the "actor" has to do extra work when someone licenses their voice.
I hope the answer will be to make true AI-based (as in, deep learning voices) that do not exist and screw all the rent-seeking opportunities.
Sounds pretty bad. This means we are going to make it artificially expensive for no other good reason than rent-seeking. It's not like the "actor" has to do extra work when someone licenses their voice.
I hope the answer will be to make true AI-based (as in, deep learning voices) that do not exist and screw all the rent-seeking opportunities.
A font designer doesn't do more work when their font faces get used in different works. Stock photo creators don't do extra work when their images get used in new media. Musicians don't do extra work to have their songs used in commercials and movies.
Virtually anyone can use their own voice (or an entirely artificial one) in some work. If folks are willing to pay money to use one particular person's voice, what's the harm in that? If they don't want to pay to use that voice, they have literally billions of other options.
Virtually anyone can use their own voice (or an entirely artificial one) in some work. If folks are willing to pay money to use one particular person's voice, what's the harm in that? If they don't want to pay to use that voice, they have literally billions of other options.
[deleted]
I really enjoy the robotic ones ...
Although there wasn't plain sampling+cut&paste involved, those examples are still a product of human voice actors who gave consent for their voice to be used as the source.
Now I wonder what happens a few years from now should someone produce a new song with a singer who sounds exactly say like Elvis, same tone, same style, although Elvis isn't mentioned anywhere.
Elvis is dead, so it can't be him, and the track wouldn't be a copy of anything Elvis sang during his life. Also demonstrating that Elvis voice has been used to train an AI would be hard (assuming the voice wasn't created entirely from scratch).
A lawsuit from the Elvis Estate would probably follow, but on what grounds?
We might see some deep modifications to copyright/trademark laws when this happens.
It's still obviously synthesized but it may be good enough for many applications.
Alternatively, voice actors are generally pretty cheap and they're usually not the limiting factor of much at all, though it does depend.
The cost of no-name actors is almost an insignificant part of production cost.
As someone note above - things like the voice enabled stop announcements for bus/rail/subway stops etc. - this would be ideal.
Alternatively, voice actors are generally pretty cheap and they're usually not the limiting factor of much at all, though it does depend.
The cost of no-name actors is almost an insignificant part of production cost.
As someone note above - things like the voice enabled stop announcements for bus/rail/subway stops etc. - this would be ideal.
Curious, is there any promising open source work in this area?
When this works 10 of billions of dollars of media can be cheaply dubbed.
You are talking English media to billions of new people, and massive libraries coming back to English speakers.
And dubbed way better than anything at the moment. You can mess with the speech to lips. Not just movies, also things like Khan Academy.
This is something that should be Moonshot.
You are talking English media to billions of new people, and massive libraries coming back to English speakers.
And dubbed way better than anything at the moment. You can mess with the speech to lips. Not just movies, also things like Khan Academy.
This is something that should be Moonshot.
10 of billions of dollars of media can be cheaply dubbed and altered.
I guess.
I'd just be happy with some Russian Scifi dubbed for the first time to begin.
On Wish Dragon I was disappointed they referenced the game as chess, when clearly that was pandering to the English audience. It would be nice to fix that, it doesn't feel like that was the directors decision.
But sure, George Lucas could stuff a lot of dialogue up.
To me the concerns that original dialog will be lost are simply not real. If anything they can be restored if you have just the subtitles/original script
I think fan fiction re-dubbing could be really interesting. I think some movies could be turned around on the dialogue alone.
But to fear the authenticity of movies is at stake while spreading the worlds media across cultures, no, to many data hoarders to be losing original audio these days, I think we are safe.
And obviously on Khan Academy etc you do want to alter it. That's the point. Fix mistakes. Change things for different cultures. Speak a little clearer.
I'd just be happy with some Russian Scifi dubbed for the first time to begin.
On Wish Dragon I was disappointed they referenced the game as chess, when clearly that was pandering to the English audience. It would be nice to fix that, it doesn't feel like that was the directors decision.
But sure, George Lucas could stuff a lot of dialogue up.
To me the concerns that original dialog will be lost are simply not real. If anything they can be restored if you have just the subtitles/original script
I think fan fiction re-dubbing could be really interesting. I think some movies could be turned around on the dialogue alone.
But to fear the authenticity of movies is at stake while spreading the worlds media across cultures, no, to many data hoarders to be losing original audio these days, I think we are safe.
And obviously on Khan Academy etc you do want to alter it. That's the point. Fix mistakes. Change things for different cultures. Speak a little clearer.
This will be great for dubbing, no doubt.
I was thinking more about the negative aspects, though. Altering historical recordings to spread misinformation, fake audio recordings "exposing" politicians getting leaked to the public,... things like that.
Those are wonderful. I listen to quite a bit of audio books and while some voice actors are superb, some are just terrible for my taste and ruin the book.
If I can dial in the pace of speech then I would almost prefer this to real actors, right now.
If I can dial in the pace of speech then I would almost prefer this to real actors, right now.
Local government broadcaster, the ever more bloated and ugly NRK, have started using synthetic voices for translations and it is so jarring I cringe in my seat.
Especially gross is it when you hear the passionate voice of someone crying out - I particularly remember this when Myanmar was on the agenda - and then that perfectly understandable English voice fades out and is replaced with this all-artifacts-no-emotion-at-all voice.
Especially gross is it when you hear the passionate voice of someone crying out - I particularly remember this when Myanmar was on the agenda - and then that perfectly understandable English voice fades out and is replaced with this all-artifacts-no-emotion-at-all voice.
There is a lot of work going on in this space right now and there have been huge advances in the past few years. The neural voices are starting to sound pretty good.
I am working on a product right now that's leveraging text to speech. I am presently using Microsoft and Google TTS because they seem to have models trained in a number of languages.
I'd be curious to hear what others are using, especially in the case where support for a number of languages is required.
I am working on a product right now that's leveraging text to speech. I am presently using Microsoft and Google TTS because they seem to have models trained in a number of languages.
I'd be curious to hear what others are using, especially in the case where support for a number of languages is required.
The ability to convert ASCII text into legible phonemes has always been such a trivial part of a voice actor's job it's the part that is considered "free". After all, for decades already, you've been able to go on Mechanical Turk or Fiverr and get someone to do a quite good voice conversion for 1/10th the price of a voice actor and yet that market has remained quite small.
The real skill of a voice actor is to collaborate with the writer on deciding the specific vocal choices to make during a performance to give it specific character and intentionality. There's been no work in this area and, indeed, it's hard to imagine how you would even design a system to annotate text with specific intonation cues that isn't many times slower than just saying the thing out loud.
eg: Imagine you had a WebMD page and you gave it to two different voice actors and you told one to try and make this drug sounds as promising as possible and another to make it sound as scary as possible. One piece of text can produce two vastly different audio products depending on intentionality. But with AI tools, there isn't even a setting for getting different audio from the same text and it's not even clear how you would build the setting that would allow you to convey what intentionality you want from the AI.
In short, there's a market for just handing a raw piece of text over to a human to convert into audio with no other interaction and that market is amenable to be taken over by AI but is already quite cheap and quite a small market. There's an entirely separate market where the voice actor sits down with the author and talks over what the intentions of the text are and how best to convert them to audio and nobody is thinking about building AIs that can understand intentionality any better than a human would.
The real skill of a voice actor is to collaborate with the writer on deciding the specific vocal choices to make during a performance to give it specific character and intentionality. There's been no work in this area and, indeed, it's hard to imagine how you would even design a system to annotate text with specific intonation cues that isn't many times slower than just saying the thing out loud.
eg: Imagine you had a WebMD page and you gave it to two different voice actors and you told one to try and make this drug sounds as promising as possible and another to make it sound as scary as possible. One piece of text can produce two vastly different audio products depending on intentionality. But with AI tools, there isn't even a setting for getting different audio from the same text and it's not even clear how you would build the setting that would allow you to convey what intentionality you want from the AI.
In short, there's a market for just handing a raw piece of text over to a human to convert into audio with no other interaction and that market is amenable to be taken over by AI but is already quite cheap and quite a small market. There's an entirely separate market where the voice actor sits down with the author and talks over what the intentions of the text are and how best to convert them to audio and nobody is thinking about building AIs that can understand intentionality any better than a human would.
The first two audio clips in the article are the same text with different intentionality, so they have the 'intent' setting figured out already.
Intent is not a binary choice, it's an infinite gradation.
This isn't about just text-to-voice AI but about how we build AI systems in general.
The AI we have now is good for
a) problems with an unambiguously correct answer that AIs can perform better than humans (like camera tracking a scene).
b) problems where the deficiencies of AI performance are made up for by other characteristics (like machine translation which is often bad but at least its instant)
Where we have no real advances in AI are problems in which there are multiple valid answers and how to coach and give feedback to an AI to improve their answer more in accordance to your preferences. People who make pollyannaish predictions about AI takeover fail to understand this distinction.
eg: It's relatively easy to make an AI that can spit out a bunch of brand new recipes. But we don't know how to right now take an 85% good recipe that an AI generates, make it, taste it and then give AI feedback that would allow it to improve that recipe. Like, we don't even know the UI for that. About the best UI we have for it right now is to have the AI show you two versions side by side and you pick the better one until it gradient descents into what you want but that's unbelievably clunky for most tasks.
Where humans excel is at the interface and the interface makes up a surprisingly large part of many tasks, especially ones that white collar workers mistakenly regard as "menial".
This isn't about just text-to-voice AI but about how we build AI systems in general.
The AI we have now is good for
a) problems with an unambiguously correct answer that AIs can perform better than humans (like camera tracking a scene).
b) problems where the deficiencies of AI performance are made up for by other characteristics (like machine translation which is often bad but at least its instant)
Where we have no real advances in AI are problems in which there are multiple valid answers and how to coach and give feedback to an AI to improve their answer more in accordance to your preferences. People who make pollyannaish predictions about AI takeover fail to understand this distinction.
eg: It's relatively easy to make an AI that can spit out a bunch of brand new recipes. But we don't know how to right now take an 85% good recipe that an AI generates, make it, taste it and then give AI feedback that would allow it to improve that recipe. Like, we don't even know the UI for that. About the best UI we have for it right now is to have the AI show you two versions side by side and you pick the better one until it gradient descents into what you want but that's unbelievably clunky for most tasks.
Where humans excel is at the interface and the interface makes up a surprisingly large part of many tasks, especially ones that white collar workers mistakenly regard as "menial".
In terms of practical examples, modding communities are definitely taking advantage already: https://www.nexusmods.com/skyrimspecialedition/mods/44184
In the bay area there's "Junípero Serra" which is usually read by text-to-speech systems as "June i pair oh sara" instead of "Wun ip airoh sara" (and I don't even know that's correct and I'm not good at phonetics to show where the accents go. I only know "June i pair oh sara" is not correct. It would be like if they said San Jose" as "San Joe's" as in like "This ball belongs to Joe. It's Joe's ball" -> "San Joe's"
In Hawaii there's Kalakaua Parkway, the main street through Waikiki. It's pronounced "Kah lah kah oo ah", not "kah lah cow ah" which is what all the navigation systems say.
It really feels like these languages are being destroyed by attrition. Each year a few percent less people know the correct way to pronounce something. I'm not into the whole French level of language protection but can't we at least put some effort into this?