How close are we to full movie generation from a technical standpoint?
With DALL-E and similar things popping up, the ability to generate things accurately based on text prompts, etc. how soon will we be able to generate things like entire screenplays that are indistinguishable from others written by people, sophisticated acting performances or just straight up entire movies being generated? What will that progression look like? Are we talking years? Decades?
18 comments
As someone working in the field, I'm surprised at all the comments here saying that it's not too far off. I disagree.
It really comes down to what your tolerance for error is. If you want something that kind of looks like a movie on the surface, but comes with a bunch of of nonsensical plot points and/or image artifacts then yes, I don't think we're too far off, maybe a few years or so. But you'll easily be able to tell that it's not made by humans. I'm taking about long-form movies here, not short video clips, which IMO aren't too far off and are probably possible roughly with what we have today.
But for something that's truly indistinguishable from a human-made long-form movie, I don't think we can get there by improving the current technology, not even in 1-2 decades. Yes, DALL-E and GPT are impressive, but once you go past cherry-picked examples to something that requires long-term cohesion and reasoning it falls apart rather quickly because you're completely out-of-sample. You're not going to get this kind of model produce long-form content without any kind of incohesion.
It really comes down to what your tolerance for error is. If you want something that kind of looks like a movie on the surface, but comes with a bunch of of nonsensical plot points and/or image artifacts then yes, I don't think we're too far off, maybe a few years or so. But you'll easily be able to tell that it's not made by humans. I'm taking about long-form movies here, not short video clips, which IMO aren't too far off and are probably possible roughly with what we have today.
But for something that's truly indistinguishable from a human-made long-form movie, I don't think we can get there by improving the current technology, not even in 1-2 decades. Yes, DALL-E and GPT are impressive, but once you go past cherry-picked examples to something that requires long-term cohesion and reasoning it falls apart rather quickly because you're completely out-of-sample. You're not going to get this kind of model produce long-form content without any kind of incohesion.
What is worrying about this comment, is that you say short video clips aren’t far off. There is a certain illusion to long form artworks that they are a whole, when I would argue that instead they’re a composite of many short form works. A movie is typically split into three acts, and act is made up of multiple sequences, a sequence is made up of multiple scenes and then a scene is made up of multiple shots. If we are close to short form video being a reality, then we are almost certainly close to long form video.
Yes you probably won’t be able to just write one sentence and out plops a cinematic blockbuster. But you will be able to write “Tom and Hannah discuss the death of Charlie and what it means to them whilst eating food in a diner” to spit out part of a script. Once you’ve tweaked your prompts and got your full script, you’d then theoretically put it in HOLL-Y and say “a close up shot of Charlie reading this part of the script in the diner in between eating his burger”, “a wide shot of this part of the script set in the same diner as the last video” etc.
String those videos together with some ai generated transition shots to set the scene and you’ve got yourself a movie.
Yes you probably won’t be able to just write one sentence and out plops a cinematic blockbuster. But you will be able to write “Tom and Hannah discuss the death of Charlie and what it means to them whilst eating food in a diner” to spit out part of a script. Once you’ve tweaked your prompts and got your full script, you’d then theoretically put it in HOLL-Y and say “a close up shot of Charlie reading this part of the script in the diner in between eating his burger”, “a wide shot of this part of the script set in the same diner as the last video” etc.
String those videos together with some ai generated transition shots to set the scene and you’ve got yourself a movie.
> If we are close to short form video being a reality, then we are almost certainly close to long form video
No, because a short 5-second clip requires mostly pattern matching that can be learned from existing training data that's in-sample, while long-form content requires symbolic reasoning that's out-of-sample.
> Yes you probably won’t be able to just write one sentence and out plops a cinematic blockbuster. But you will be able to write “Tom and Hannah discuss the death of Charlie and what it means to them whilst eating food in a diner” to spit out part of a script
I agree with this, but that's not how I understood the original question, which was having an AI model generate a movie script. If you have humans in-the-loop at every step that need to fine-tune and iterate on each part, that's different and I agree it's not nearly as far far away. But then you can already do something similar today by crowdsourcing the whole process behind a bunch of API calls. AI models will make this process cheaper, perhaps you can get a 10x reduction on workers, but you still need a huge amount of human input for the final output. Nothing wrong with that, I just don't think that's what the OP was asking.
No, because a short 5-second clip requires mostly pattern matching that can be learned from existing training data that's in-sample, while long-form content requires symbolic reasoning that's out-of-sample.
> Yes you probably won’t be able to just write one sentence and out plops a cinematic blockbuster. But you will be able to write “Tom and Hannah discuss the death of Charlie and what it means to them whilst eating food in a diner” to spit out part of a script
I agree with this, but that's not how I understood the original question, which was having an AI model generate a movie script. If you have humans in-the-loop at every step that need to fine-tune and iterate on each part, that's different and I agree it's not nearly as far far away. But then you can already do something similar today by crowdsourcing the whole process behind a bunch of API calls. AI models will make this process cheaper, perhaps you can get a 10x reduction on workers, but you still need a huge amount of human input for the final output. Nothing wrong with that, I just don't think that's what the OP was asking.
> I agree with this, but that's not how I understood the original question, which was having an AI model generate a movie script. If you have humans in-the-loop at every step that need to fine-tune and iterate on each part, that's different and I agree it's not nearly as far far away.
I think you’re right in that the OP probably just wants a “one a click” prompt -> script/movie which is decades or more off.
Thinking out loud:
The end product exists to effect human emotions and I’m not convinced you can create a test suite for a computer that measures that. I think the closest you could get would be to strap people up to some sensors and have it measure their responses such as heart rate, breathing rate, sweat rate etc and have the test subjects answer questions about their perception of the film and then have the computer make tweaks and refinements based on that which would be about as close as you can get and pretty much how the industry operates now I believe except the producer stands in for the role of the computer.
Somewhere down the line there’s got to be human input. I kind of think maybe what I’m trying to say is that for video there’s an extra data classification step compared to 2D images. That human review part of each scene of the script and each shot of the film are part of a “meta level” to the workflow. I’m guessing this is what you’re referring to by “long form content requires symbolic reasoning”?
Maybe if enough “meta level” data was gathered about which of these tweaks resulted in successful movies the AI might start developing an “intuition” for what makes a great movie but I think you’d need decades of data for it to work?
I think you’re right in that the OP probably just wants a “one a click” prompt -> script/movie which is decades or more off.
Thinking out loud:
The end product exists to effect human emotions and I’m not convinced you can create a test suite for a computer that measures that. I think the closest you could get would be to strap people up to some sensors and have it measure their responses such as heart rate, breathing rate, sweat rate etc and have the test subjects answer questions about their perception of the film and then have the computer make tweaks and refinements based on that which would be about as close as you can get and pretty much how the industry operates now I believe except the producer stands in for the role of the computer.
Somewhere down the line there’s got to be human input. I kind of think maybe what I’m trying to say is that for video there’s an extra data classification step compared to 2D images. That human review part of each scene of the script and each shot of the film are part of a “meta level” to the workflow. I’m guessing this is what you’re referring to by “long form content requires symbolic reasoning”?
Maybe if enough “meta level” data was gathered about which of these tweaks resulted in successful movies the AI might start developing an “intuition” for what makes a great movie but I think you’d need decades of data for it to work?
This month is the 10th year anniversary of imagenet2012.
I don't think it is reasonable to make predictions of how well the technology will be in 10 years. I don't think any experts in the field would say Dall-e was viable ten years ago, let alone 20 years ago.
I don't think it is reasonable to make predictions of how well the technology will be in 10 years. I don't think any experts in the field would say Dall-e was viable ten years ago, let alone 20 years ago.
IMO we're pretty far off from building a single model that can generate a whole film at the press of a button.
But I think a more interesting distinction is a model that allows a human to generate professional quality without spending the time or skill it used to take. We've got most of the building blocks -- a language model could generate a 3 act summary, could go from a description of a scene to the text of it, can go from a text description of a character to an image, from an image to a 3d models, etc.
Every step needs human intervention and curation, but the difference is it can be done by anyone. As the tools get better they get more control. I think in under a year, anyone will be able to create a 30-page graphic novel that is indistinguishable from human created work without close inspection. It will still take a lot of time and creative energy but they will not need to have an art or writing background.
In 5 years, creating a pixar level feature animated film will still take a lot of work, but it will be achievable by, say, a high school movie-making club over the course of a semester, rather than needing a studio of hundreds of specialists over two years.
But I think a more interesting distinction is a model that allows a human to generate professional quality without spending the time or skill it used to take. We've got most of the building blocks -- a language model could generate a 3 act summary, could go from a description of a scene to the text of it, can go from a text description of a character to an image, from an image to a 3d models, etc.
Every step needs human intervention and curation, but the difference is it can be done by anyone. As the tools get better they get more control. I think in under a year, anyone will be able to create a 30-page graphic novel that is indistinguishable from human created work without close inspection. It will still take a lot of time and creative energy but they will not need to have an art or writing background.
In 5 years, creating a pixar level feature animated film will still take a lot of work, but it will be achievable by, say, a high school movie-making club over the course of a semester, rather than needing a studio of hundreds of specialists over two years.
> I think in under a year, anyone will be able to create a 30-page graphic novel that is indistinguishable from human created work without close inspection.
I don’t even think you will be able to distinguish with close inspection. There’s already been AI generated artwork submitted to art competitions without informing the judges of its origin that has gone on to win the competition.
I don’t even think you will be able to distinguish with close inspection. There’s already been AI generated artwork submitted to art competitions without informing the judges of its origin that has gone on to win the competition.
> how soon will we be able to generate things like entire screenplays that are indistinguishable from others written by people
Writing a typical movie that's indistinguishable from Hollywood output is probably not too far off.
But that's a problem to me because most Hollywood movies are derivative trash sequels or remakes. I'm sure AI can pull off the typical cape-shit sequel or remake.
But can AI create original characters that people feel an emotional connection to and create new stories? Would an AI get people to see the love that's been put into some characters? That's what I'm not so sure about.
Writing a typical movie that's indistinguishable from Hollywood output is probably not too far off.
But that's a problem to me because most Hollywood movies are derivative trash sequels or remakes. I'm sure AI can pull off the typical cape-shit sequel or remake.
But can AI create original characters that people feel an emotional connection to and create new stories? Would an AI get people to see the love that's been put into some characters? That's what I'm not so sure about.
> But can AI create original characters that people feel an emotional connection to and create new stories? Would an AI get people to see the love that's been put into some characters? That's what I'm not so sure about.
I don’t think this would be as hard as you think. In the screenwriting world there’s a very famous book called “Save the cat” which is named after the authors assertion that in the first 5 minutes of the movie the protagonist has got to “save a cat” or some other altruistic act to make the audience like them. To make likeable characters you’d just have to feed in a few prompts for some scenes where they put the needs of others above themselves and one where they vulnerably discuss a tragic event that’s left them psychologically wounded with another character. Boom, one likeable character served on a plate.
Bonus points if the character makes bad decisions throughout the first half of the movie based on this psychological wound (called “the lie they’re telling themselves”) and then in the second half of the movie they make a change culminating in a showdown with the antagonist (who represents what the character would have turned into if they hadn’t changed). Classic hero’s journey.
I don’t think this would be as hard as you think. In the screenwriting world there’s a very famous book called “Save the cat” which is named after the authors assertion that in the first 5 minutes of the movie the protagonist has got to “save a cat” or some other altruistic act to make the audience like them. To make likeable characters you’d just have to feed in a few prompts for some scenes where they put the needs of others above themselves and one where they vulnerably discuss a tragic event that’s left them psychologically wounded with another character. Boom, one likeable character served on a plate.
Bonus points if the character makes bad decisions throughout the first half of the movie based on this psychological wound (called “the lie they’re telling themselves”) and then in the second half of the movie they make a change culminating in a showdown with the antagonist (who represents what the character would have turned into if they hadn’t changed). Classic hero’s journey.
Need an open source project that tries to build it.
(There is a lot of hard problems to solve but it is worth the effort)
Need to start with stick figures in 2d worlds.
Next upgrade to voxel 3d worlds.
Finally upgrade to 3d worlds rendered in unity/blender/etc.
(It will take too long to get Disney/Marvel quality in the first release).
Need create a ViT/LLM trained on scripts and existing movies.
Can use future versions of AI image generation to get to high quality.
Probably 5+ years unless there a significant compute or theoretical break through. Video is orders of magnitude more data than still images, and there aren’t a lot of good labeled data sets to train on.
TikTok and instagram probably have the only meaningful (short form, highly annotated and quality marked) datasets and they won’t be sharing.
Maintaining long term coherence over time is an unsolved problem.
..but something else is going happen sooner (and it’s already happening, so the timescale is significantly shorter); and that is frame-by-frame mapping of existing video using diffusion models.
The issue is coherence between frames, but people are already addressing that by using carefully crafted “key frames” and the previous frame as an img2img prompt for the next frame.
There are already people taking crude “grey box” rendered animations and feeding them frame by frame into models to generate stylised video with “roughly this structure”.
Coherence over time isn’t there… but, it’ll get better.
I’ll be stunned if you don’t see this generating good quality videos before the end of the year.
Further more, since this basically renders the work of many VFX artists mostly irrelevant (from do everything to fix up little bits that didn’t make sense) you can bet on movie studios pouring a tonne of capital into it. You’ll see change in this space very quickly.
TikTok and instagram probably have the only meaningful (short form, highly annotated and quality marked) datasets and they won’t be sharing.
Maintaining long term coherence over time is an unsolved problem.
..but something else is going happen sooner (and it’s already happening, so the timescale is significantly shorter); and that is frame-by-frame mapping of existing video using diffusion models.
The issue is coherence between frames, but people are already addressing that by using carefully crafted “key frames” and the previous frame as an img2img prompt for the next frame.
There are already people taking crude “grey box” rendered animations and feeding them frame by frame into models to generate stylised video with “roughly this structure”.
Coherence over time isn’t there… but, it’ll get better.
I’ll be stunned if you don’t see this generating good quality videos before the end of the year.
Further more, since this basically renders the work of many VFX artists mostly irrelevant (from do everything to fix up little bits that didn’t make sense) you can bet on movie studios pouring a tonne of capital into it. You’ll see change in this space very quickly.
If the current method we use turns out to actually lead to producing content that has a history and can regenerate characters then it will likely be within 5 years.
But right now it is possible it is only a nice math trick that is already near its limits. GPT3 and Stable Diffusion are great for developing one moment / conversation and that may even turn into one scene but beyond that it is extremely difficult to have something coherent.
There is a really exciting model That may mostly solve that but it’s too early right now: https://github.com/rinongal/textual_inversion
For fun I tried to make an entire animated music video but it took over one week of processing and basically fell apart coherently by 30 seconds so just did one third:
https://youtu.be/f3GfUKJBUYA
But right now it is possible it is only a nice math trick that is already near its limits. GPT3 and Stable Diffusion are great for developing one moment / conversation and that may even turn into one scene but beyond that it is extremely difficult to have something coherent.
There is a really exciting model That may mostly solve that but it’s too early right now: https://github.com/rinongal/textual_inversion
For fun I tried to make an entire animated music video but it took over one week of processing and basically fell apart coherently by 30 seconds so just did one third:
https://youtu.be/f3GfUKJBUYA
The answer really depends on whether you can look over the uncanny hollowness of affect that all AI art seems to induce. Technically I can believe all the image and motion parts will fall into place in a few years. Im becoming increasingly unconvinced by ML output to conjure satisfyingly rich meaningful experiences in the audience. That may be due partly to an emotional detachment caused by the observer knowing that the output was created by automation. The end effect is the same. The (good) artists I know are becoming less threatened by ML replacing them and more curious about how it might help them create references.
Obviously itll be amazing for creating daytime TV and general YouTube filler.
Obviously itll be amazing for creating daytime TV and general YouTube filler.
I thought about this a few days ago and I thought it might start of as a new movie type were people will accept a more dreamlike style of movie. Coherent enough to see it as a movie but with a new form of artifacts. Like small details change but it's still okay for us.
On the other hand, I have seen enough papers who also address aspects like coherence in style, object tracking and other aspects.
I don't think we are far away at all.
More like how long until performance for quick feedback is fast enough, cheap enough etc.
On the other hand, I have seen enough papers who also address aspects like coherence in style, object tracking and other aspects.
I don't think we are far away at all.
More like how long until performance for quick feedback is fast enough, cheap enough etc.
DALLE-2 can't even do "The quick brown fox jumps over the lay dog" properly: https://np.reddit.com/r/dalle2/comments/xaxbpv/the_quick_bro...
Context is hard, so not for a while if the tech is to fool people on a full movie scale.
Context is hard, so not for a while if the tech is to fool people on a full movie scale.
Think of a plot outline from the perspective of various characters' locations in a diagram like this https://www.reddit.com/r/interestingasfuck/comments/9p9xoi/t...
Or one can have character analysis "files" on each character: where do they come from, what are their abilities, what are they trying to do. And what are the relations between characters - and how they change during the story.
A bad movie doesn't have any of that continuity or relatability of course. But the AI would need to generate such and multiple other higher level structures and then match the visual imagery with it - maybe the pixel level visuals are actually the easy part.
Google translate from say Mongolian to Finnish doesn't work by applying pairs learned from reading Mongolian-Finnish translations. It also doesn't work by translating first to English and then to Finnish. It translates all languages first to an internal representation, and then to the target language.
This kind of structured internal representation of the world and human life is what's still missing from an AI, and it can't be learned just from watching movies.
Or one can have character analysis "files" on each character: where do they come from, what are their abilities, what are they trying to do. And what are the relations between characters - and how they change during the story.
A bad movie doesn't have any of that continuity or relatability of course. But the AI would need to generate such and multiple other higher level structures and then match the visual imagery with it - maybe the pixel level visuals are actually the easy part.
Google translate from say Mongolian to Finnish doesn't work by applying pairs learned from reading Mongolian-Finnish translations. It also doesn't work by translating first to English and then to Finnish. It translates all languages first to an internal representation, and then to the target language.
This kind of structured internal representation of the world and human life is what's still missing from an AI, and it can't be learned just from watching movies.
Aren't we talking about something AGI-like with this last sentence?
Maybe I'm crazy but I think it's already possible to make a basic movie, if you have dozens of GPUs or 100K$ to burn on cloud instances