Anam Cara-3: Why we think AI needs a face
5 comments
Really interesting architecture choice separating motion from rendering. That feels like the right abstraction boundary if you want identity generalisation without retraining.
The latency numbers are what stood out to me though. ~70ms time-to-first-frame is genuinely impressive for an interactive loop. In real conversations, responsiveness dominates perceived realism way more than visual fidelity, so that correlation result makes intuitive sense.
Curious how robust the audio-to-motion mapping is under messy real-world input (overlapping speech, accents, background noise, etc.). Does the flow-matching variant help mostly with stability during training, or also temporal consistency during inference?
The latency numbers are what stood out to me though. ~70ms time-to-first-frame is genuinely impressive for an interactive loop. In real conversations, responsiveness dominates perceived realism way more than visual fidelity, so that correlation result makes intuitive sense.
Curious how robust the audio-to-motion mapping is under messy real-world input (overlapping speech, accents, background noise, etc.). Does the flow-matching variant help mostly with stability during training, or also temporal consistency during inference?
Thanks for the question.
audio-to-motion is fairly robust to noisy TTS and differing languages / accents. It doesn't use the raw audio as input, we first embed the audio using a pretrained wav2vec-style embedder, trained on millions of audio samples.
Saying this, we haven't properly evaluated in multiple languages, and we have heard from customers that lip-sync isn't always as good in non-english. For Cara 4 we're training on more diverse data, which will hopefully close this gap.
audio-to-motion is fairly robust to noisy TTS and differing languages / accents. It doesn't use the raw audio as input, we first embed the audio using a pretrained wav2vec-style embedder, trained on millions of audio samples.
Saying this, we haven't properly evaluated in multiple languages, and we have heard from customers that lip-sync isn't always as good in non-english. For Cara 4 we're training on more diverse data, which will hopefully close this gap.
One of the backend developers at Anam here, one of the hardest parts of developing this has been monitoring and analytics.
Most off the shelf solutions, or existing platforms heavily skew towards the normal http web service world. However, the bulk of our interactions happen over webrtc in long-running sessions, where the existing solutions for in-depth metrics and monitoring are much less mature and well documented.
Currently we're using influxdb, prometheus, grafana and some hand rolled monitoring code alongside the stats that webrtc offers itself. Would be interested to know how anyone out there is monitoring conversational flows, and webrtc traffic.
Most off the shelf solutions, or existing platforms heavily skew towards the normal http web service world. However, the bulk of our interactions happen over webrtc in long-running sessions, where the existing solutions for in-depth metrics and monitoring are much less mature and well documented.
Currently we're using influxdb, prometheus, grafana and some hand rolled monitoring code alongside the stats that webrtc offers itself. Would be interested to know how anyone out there is monitoring conversational flows, and webrtc traffic.
Full technical blog here: https://anam.ai/blog/cara-3-interactive-avatars
Very clever and quite frightening. Well done.
Personally I like using LLMs for getting information (not chat) or solving problems, and I like the fact it's text and I can read it quicker than a normal conversation, and don't need to look for facial cues when ingesting the information provided (am I autistic?), but I might be a minority...
Some people might really find this useful.
Personally I like using LLMs for getting information (not chat) or solving problems, and I like the fact it's text and I can read it quicker than a normal conversation, and don't need to look for facial cues when ingesting the information provided (am I autistic?), but I might be a minority...
Some people might really find this useful.
Some context on why we're working on this: faces carry emotional signal that text and voice don't. Almost half the human brain is devoted to visual processing, and it's one of the first things we learn as babies. It's also a more accessible medium. Anam started, in part, from Ben watching his gran struggle with her iPad and thinking there should be a face she could just talk to.
cara-3 uses a two-stage pipeline: a diffusion transformer converts audio to motion embeddings (head position, eye gaze, lip shape, expression), then a rendering model applies those to a reference image to produce video frames. Separating motion from rendering means we can animate any face without retraining. The two models run in sequence within ~70ms time-to-first-frame on an H200, so we can run many concurrent avatar sessions on a single GPU.
The core of audio-to-motion is flow matching, but we found off-the-shelf formulations weren't stable enough for this task, so we developed a novel variant. We also built our own training data pipeline (and recently open-sourced the backbone: Metaxy) because existing frameworks made it hard to iterate without rerunning expensive steps.
We commissioned an independent blind evaluation comparing interactive avatars from Anam with HeyGen, Tavus and D-ID. Hundreds of participants played 20 Questions with the different offerings and cara-3 scored highest on every metric (p < 0.001), 24% above the closest competitor on average. What surprised us most: responsiveness correlated with overall experience (Spearman 0.697) far more than visual quality (0.473). In interactive settings, how fast you respond matters more than how good you look.
Ask us anything!