Previously, I thought that since Seedance 2 can reference images, videos, and audio, it should be possible to input everything at once, especially the song itself, allowing the model to generate both lip-syncing and dance movements directly. In practice, the model did reference the singing music I provided, but it reinterpreted it on its own. What I actually wanted was for it to match the movements and lip-syncing based on the song’s content, but the execution was even worse than when I didn’t provide any music as a reference.
So, I tried the following approaches:
For the second attempt, I set strict requirements, asking the model not to reinterpret the music I provided but to use it directly for lip-syncing reference. However, the results were still unsatisfactory.
For the third attempt, I already had decent dance movements but lacked proper lip-syncing. I uploaded the video along with the corresponding song clip, hoping the model would use both as references to generate accurate lip-syncing. Unfortunately, the outcome was still not ideal.
As a result, I ended up reverting to my initial video production method: generating the dance based solely on text and images, then editing and refining it before using an open-source model for lip-syncing adjustments.
However, this approach has its drawbacks. During the secondary generation, the character’s appearance might change—for example, shifting from an Asian face to a European one—or unexpected elements might appear, like suddenly holding a microphone or wearing headphones. While some of these issues can be mitigated with specific prompts, others remain, such as clothing patterns being altered or the style of the outfit changing inconsistently across different angles. There are quite a few downsides to secondary generation, and I won’t list all the problems I encountered here.
Of course, it’s also possible that I’m not using the tool correctly. If anyone knows how to solve these issues, feel free to leave a comment and share your insights. Thank you, everyone!
The Seedance 2 model is incredibly powerful, completely overshadowing all other models. This is an original video I created in just one day, though the music was previously made using Suno. In the past, producing a video like this would have taken me at least a week, and the quality wouldn’t have been nearly as good. Hollywood really needs to start rethinking its approach to content creation.
Using the latest Seedance 2 model, which is incredibly powerful, you can input a reference image along with detailed descriptions of beat timings and dance moves, and it generates high-quality shots with a director’s sense of framing. I hardly had to do any rerolls, especially considering the length of the song.
Each segment can generate up to 15 seconds, but I made a silly mistake! It turns out the "full reference" feature supports all media formats—I could have input the music along with the visuals and generated lip-syncing in one go… I ended up overcomplicating things and had to manually sync the lip movements afterward. Still, I’m pretty happy with how it turned out.
To clarify, I didn’t use any real human dance footage as reference for this video—everything was generated and then edited together. Each segment of my video is based on prompts that generally include the following elements:1. Overall atmosphere description
2. Key actions
3. Scene description: starting pose, mid-sequence body/hand movements over time, and ending pose
4. Dialogue/lyrics/sound effects at specific timestamps
Seedance 2 automatically designs camera angles based on the content, though you can also specify camera movements precisely. In the raw clip below, I didn’t describe camera angles. After generating the clips, I edited them by adding lip-sync, syncing them with the music, and adjusting the speed of some segments to match the beat.
This was a habitual mistake I made while working on this video. Initially, I followed the traditional workflow for video models: first generating reference images, then describing the actions, and so on. However, Seedance supports up to 9 images, 3 video clips, and 3 audio clips as reference materials simultaneously for each generated segment.
This multimodal reference capability is quite rare among current AI video tools. In theory, I could have directly provided the model with edited music or voice clips along with reference images for generation. But for this project, I generated the clips first and then re-generated them to add lip-sync.
So, I tried the following approaches: For the second attempt, I set strict requirements, asking the model not to reinterpret the music I provided but to use it directly for lip-syncing reference. However, the results were still unsatisfactory.
For the third attempt, I already had decent dance movements but lacked proper lip-syncing. I uploaded the video along with the corresponding song clip, hoping the model would use both as references to generate accurate lip-syncing. Unfortunately, the outcome was still not ideal.
As a result, I ended up reverting to my initial video production method: generating the dance based solely on text and images, then editing and refining it before using an open-source model for lip-syncing adjustments.
However, this approach has its drawbacks. During the secondary generation, the character’s appearance might change—for example, shifting from an Asian face to a European one—or unexpected elements might appear, like suddenly holding a microphone or wearing headphones. While some of these issues can be mitigated with specific prompts, others remain, such as clothing patterns being altered or the style of the outfit changing inconsistently across different angles. There are quite a few downsides to secondary generation, and I won’t list all the problems I encountered here.
Of course, it’s also possible that I’m not using the tool correctly. If anyone knows how to solve these issues, feel free to leave a comment and share your insights. Thank you, everyone!