Neural Network Enabled Filmmaking(flawlessai.com)
flawlessai.com
Neural Network Enabled Filmmaking
https://www.flawlessai.com/
43 comments
FWIW the french bits are unimpressive at best. I’d stick with the subtitles
Yeah, it's worse than a good dub for french. The lips move kind of at the right moment, but that's it. The lips don't move the way they should to produce the sound.
In short, it's not even uncanny valley level.
In short, it's not even uncanny valley level.
I suspect this is bound to happen when your team is composed mostly of people that are not native speakers of the target languages.
I have no idea if that's the case in this specific instance, but it would explain why they think it's "flawless" when, to a native speaker, it's everything but.
But sure, they can get there with the right expertise and some more time.
Once there, they will start to face the uncanny valley of other cultural differences: e.g. a Japanese person/actor behaves/acts in a way that is different from the way a westerner behaves/acts. That will really be "peak" uncanny valley. Even just the Forest Gump clip in Japanese gave me all kinds of weird feelings for its incongruity - and I am not even a Japanese native.
I have no idea if that's the case in this specific instance, but it would explain why they think it's "flawless" when, to a native speaker, it's everything but.
But sure, they can get there with the right expertise and some more time.
Once there, they will start to face the uncanny valley of other cultural differences: e.g. a Japanese person/actor behaves/acts in a way that is different from the way a westerner behaves/acts. That will really be "peak" uncanny valley. Even just the Forest Gump clip in Japanese gave me all kinds of weird feelings for its incongruity - and I am not even a Japanese native.
The "A few good men" movie sequence is not really good.
First I cannot understand Jack Nicholson's answer and could not rely on the lips movements to get it. It is supposed to be the "you can't handle the truth" translation, but I miss the verb "vous ne ... pas la vérité".
Also the lips and mouth position to pronounce "la vérité" are not realistic, Jack seems to add a syllable at the very end.
Honest request: re-do Short Circuit so the “Indian” character played by a white guy just looks and speaks more like a white guy. I tried watching that movie for fun but the blackface or whatever you would call it really tanked it for me.
Also in your promo video you show “traditional dubbing” to English and I really expected that to be followed by your system on the same clip but instead you moved to other languages, which I can’t really grasp.
Edit: I see I’ve gotten a downvote. You can hear from the actor himself how he feels about playing that role: https://www.dailywire.com/news/fisher-stevens-regrets-playin...
Also in your promo video you show “traditional dubbing” to English and I really expected that to be followed by your system on the same clip but instead you moved to other languages, which I can’t really grasp.
Edit: I see I’ve gotten a downvote. You can hear from the actor himself how he feels about playing that role: https://www.dailywire.com/news/fisher-stevens-regrets-playin...
It would be interesting to see it fully redubbed with the actor’s own unaccented voice.
Yes I think actually this could be a PR win for the publisher. I’d say this opens a can of worms but these companies love to recycle old content. With studio support, the same actor, and some of this high powered movie magic they could really convincingly replace the offensive character.
[deleted]
dubbing, yech
TLDR: something like FaceGAN to force artificial lip sync for a dub into the video frames
That way, you can dub videos however you like without having to rewrite your dub text to match the visible mouth movements. I like the idea, but in my opinion finding good dub speakers is the challenge, not figuring out what you want them to say.
That way, you can dub videos however you like without having to rewrite your dub text to match the visible mouth movements. I like the idea, but in my opinion finding good dub speakers is the challenge, not figuring out what you want them to say.
I wonder...
The problem with dubbing is that it inherently sucks. That's a poor starting point for making something good. Any kind of connoisseur would watch the original anyway. Not many actors would be enthusiastic about being the voice of Renee Zellweger for the unwashed masses.
Animated films, OTOH, tend to have great dubbing. They hire famous actors, put in the effort... just like they do for the original language.
If dubbing didn't suck, it might start a whole chain reaction towards good.
The problem with dubbing is that it inherently sucks. That's a poor starting point for making something good. Any kind of connoisseur would watch the original anyway. Not many actors would be enthusiastic about being the voice of Renee Zellweger for the unwashed masses.
Animated films, OTOH, tend to have great dubbing. They hire famous actors, put in the effort... just like they do for the original language.
If dubbing didn't suck, it might start a whole chain reaction towards good.
Nevertheless, in some places in the world it is a mandatory standard to dub and it is a costly process distributors must handle to enter the market. I imagine VOD’s could also be interested in this as they have large libraries. Optimizing dubbing for them can really shift the proportion of localized content which can be useful when seeking regulatory approval and what not. I
There are also probably many other applications for something like this, albeit I’m with you that dubbing is super lame.
I like this direction. There is still more work needed. They should post-process the frames similar to the restyle-encoder: https://github.com/yuval-alaluf/restyle-encoder
The over the top marketing on the landing page is so hilarious to me.
Having looked at some Hollywood-focused products and services recently, this type video is similar to a “reel,” and from what I can tell this kind of presentation is typical of the biz.
“Neural Network Lip Sync For Translations” feels like a more honest tagline than “Neural Network Enabled Filmmaking”. I get that you’re probably aiming for filmmakers, and they can definitely make good use of this tech, but I’ve gotta imagine the “whales” in this market are companies with hundreds of hours of corporate training videos that need to be localized.
But no matter the audience, cutting to the chase about what the product does is super important.
But no matter the audience, cutting to the chase about what the product does is super important.
This entire site suffers from overselling and overdramatizing what they have massively. Gotta tone it down a little.
Looks cool to me. I don’t speak these other languages, but I like the way the actors’ facial expressions are maintained through the dub.
I wonder how longer or shorter phrasing would affect clips or cut length and how backgrounds might also have to accommodate script line length.
This last year’s best picture went to Parasite, a “foreign” film with English subtitles.
While film people would probably always prefer the original, I bet most Americans would have watched Parasite dubbed with this technology and not even realized the actors didn’t use English in the creation.
Hollywood does butts-in-seats and I think this would put more butts in seats.
I wonder how longer or shorter phrasing would affect clips or cut length and how backgrounds might also have to accommodate script line length.
This last year’s best picture went to Parasite, a “foreign” film with English subtitles.
While film people would probably always prefer the original, I bet most Americans would have watched Parasite dubbed with this technology and not even realized the actors didn’t use English in the creation.
Hollywood does butts-in-seats and I think this would put more butts in seats.
I was expecting something to do with actual filmmaking - like writing, production optimization, etc. This is... dubbing?
As someone who grew up with dubs I think this looks terrible and distracting.
Firmly in the uncanny valley. I’d rather watch the original with subtitles (and that’s almost always the case, not only in terms of preserving acting, but to be exposed to other languages and cultures). But is a cool idea for the future. I imagine in 10 years this could be pretty amazing.
I wasn't expecting this to be lip syncing, and so many negative comments here about quality and "I'd rather watch dubs".
To me, this is "good enough" to watch. I don't like dubs because the voice is out of sync with the character, and removes you from the story. I don't like subtitles because I can read faster than they talk, and then I'm left with words blocking part of the scene, and I feel like I know what the person has said, yet their still talking.
This is a great start, and they'll continue to improve in the future.
To me, this is "good enough" to watch. I don't like dubs because the voice is out of sync with the character, and removes you from the story. I don't like subtitles because I can read faster than they talk, and then I'm left with words blocking part of the scene, and I feel like I know what the person has said, yet their still talking.
This is a great start, and they'll continue to improve in the future.
> I don't like dubs because the voice is out of sync with the character, and removes you from the story.
What about having to watch a slightly mismatched mouth say "pe pe pe ba ba ba", doesn't that remove you from the story?
What about having to watch a slightly mismatched mouth say "pe pe pe ba ba ba", doesn't that remove you from the story?
Have you watched stuff with subtitles much? It doesn't sound like it. After not very long you hardly notice they're there, unless they're not well done. And you have just acquired a magical ability to understand every language on earth.
The only mildly frustrating thing is that after watching a lot of subtitled movies in a particular language, I still can't speak the language, after having the illusion I understand it perfectly!
The only mildly frustrating thing is that after watching a lot of subtitled movies in a particular language, I still can't speak the language, after having the illusion I understand it perfectly!
You can kind of forget that you are reading subtitles but occasionally you might be reading when something happens on screen that you miss. Being able to spend all time (fully) watching the movie is not something you can deny makes for a better experience.
What makes subtitles even worse for me is that I usually watch at higher speeds often 200-300%. Then reading becomes even more annoying and you have to pay very close attention making you miss out even more on the visuals.
What makes subtitles even worse for me is that I usually watch at higher speeds often 200-300%. Then reading becomes even more annoying and you have to pay very close attention making you miss out even more on the visuals.
A couple of times I’ve stepped into the kitchen to grab a drink forgetting that I can’t understand the dialog if I'm not watching.
[deleted]
Contrary to some posts here, I think this approach shows promise though it would clearly benefit from further refinement.
The criticisms here seem to break down into three groups:
1. "I don't mind reading subtitles and that will always be better."
2. "Any real film purist will prefer watching the original."
3. "The implementation is kind of uncanny at some points."
My thoughts:
#1: Sure, I actually agree but I also know that there are a lot of people out there that, for whatever reason, find reading subtitles breaks their immersion. Even though that's not me, dismissing that audience's experience isn't very inclusive.
#2: Yep, As a film maker I totally get that but us purists and aficionados aren't that big of a market. So yeah, Criterion should never use any tech like this for definitive remaster releases as it doesn't preserve artistic fidelity. But just because they should keep this away from Citizen Kane doesn't mean there aren't suitable classes of content. In the era of high-fidelity streaming media, additional versions don't necessarily take distribution availability from the original (which should always be available).
#3: Though I'm not an AI engineer I've done some work with AI engineers on artistic content and my guess is the current gaps we see might be sufficiently surmountable with refinement. It may never be fully automatic but redubbing is human intensive anyway so an interactive UI that allows an ADR-experienced editor to tune problematic output may help a lot.
I think the team should keep going. Once it's good enough, there is a lot of content that might reach new audiences through this tech.
The criticisms here seem to break down into three groups:
1. "I don't mind reading subtitles and that will always be better."
2. "Any real film purist will prefer watching the original."
3. "The implementation is kind of uncanny at some points."
My thoughts:
#1: Sure, I actually agree but I also know that there are a lot of people out there that, for whatever reason, find reading subtitles breaks their immersion. Even though that's not me, dismissing that audience's experience isn't very inclusive.
#2: Yep, As a film maker I totally get that but us purists and aficionados aren't that big of a market. So yeah, Criterion should never use any tech like this for definitive remaster releases as it doesn't preserve artistic fidelity. But just because they should keep this away from Citizen Kane doesn't mean there aren't suitable classes of content. In the era of high-fidelity streaming media, additional versions don't necessarily take distribution availability from the original (which should always be available).
#3: Though I'm not an AI engineer I've done some work with AI engineers on artistic content and my guess is the current gaps we see might be sufficiently surmountable with refinement. It may never be fully automatic but redubbing is human intensive anyway so an interactive UI that allows an ADR-experienced editor to tune problematic output may help a lot.
I think the team should keep going. Once it's good enough, there is a lot of content that might reach new audiences through this tech.
Also, 1 and 2 might be valid for the film industry, but the advertising/tv industry are not bothered by that. There are so many languages in the EU, I can see how this could be a very useful tech.
This is awesome!
Deepfake but real-ish lip-sync edition.
With things as they are now, I prefer original dialog with subtitles usually because the lip-sync is so bad, forced, and doesn't sound like the original actor. (Anyone remember how bad the sync was on some Spaghetti Westerns?) If this can do "deepfake" alternate language dialog using AI processing to change the language and the accent but in the same voice as the actor, I'm all for it. Although, the talent should get compensated because it's taking money out of their pockets if their jobs are being replaced digitally.
With things as they are now, I prefer original dialog with subtitles usually because the lip-sync is so bad, forced, and doesn't sound like the original actor. (Anyone remember how bad the sync was on some Spaghetti Westerns?) If this can do "deepfake" alternate language dialog using AI processing to change the language and the accent but in the same voice as the actor, I'm all for it. Although, the talent should get compensated because it's taking money out of their pockets if their jobs are being replaced digitally.
It fails because the face is real but the mouth movement is AI. It would need to be perfect otherwise you are in uncanny valley. It reminded me of the talking horse TV shows from years ago.
I think this is maybe the only good/beneficial use case for DeepFakes
Do they do the translation and voice synthesizing or just the mouthing?
Given the quality of Google translate, “every language” sounds like they are overselling it.
Given the quality of Google translate, “every language” sounds like they are overselling it.
You dub it conventionally, then you have AI-assisted "mouth visual redubbing"