The Moshi model by Kyutai is incredible, and the paper they published is very dense. We spent some time internally studying it and created an internal writeup (which we are now making public) based on feedback from peers that it helped save them a couple hours & helped make the architecture click faster.
Frisson Labs | Founding ML Engineer - General, Founding ML Engineer - Audio/Speech | San Francisco, CA ONSITE | Full-time | https://www.frisson-labs.com
Frisson Labs is building AI players that play like Discord friends. The hard part is making them respond with low latency, persistent personality, voice timing, and actual game competence: understanding what is happening, deciding what to do, and executing in real time.
Open roles:
- Founding ML Engineer - General: own the full stack for how AI companions play games: game-state understanding, learned policies/controllers, world models, action representation, MLP/policy heads, memory, evals, data pipelines, inference, and product iteration. Strong fit if you have shipped applied ML systems and want to own messy model behavior all the way from research prototype to live gameplay.
- Founding ML Engineer - Audio/Speech: explore and ship low-latency speech systems for real-time play: duplex/streaming voice models, speech understanding, turn-taking, interruption handling, latency/quality tradeoffs, prosody/emotion, and conversational evals. Strong fit if you have deep audio/speech ML experience and care about making voice feel socially present during gameplay.
Comp:
- ML Engineer roles: $150K-$200K + 1.0-2.0% equity
Apply: email [email protected] with the role in the subject, plus your resume/GitHub/research/portfolio and a short note on why AI companions + gaming interests you.
I wanted to get my hands dirty with the Gemma model and try out some model surgery. This is a small smoke test, not a production voice model, but the wiring was fun enough to write up.
Gemma can take in audio, images, and text, but only talks back in text. Mimi can turn codec tokens back into speech. So I froze both sides and trained a small graft in the middle: Gemma hidden states -> Mimi audio tokens.
I've enjoyed playing with this because the bad audio outputs have sounded hilarious
It's open source, runs in your browser and there is no backend. So far I've been using this myself and found a handful of audio researchers to follow on X/HF.
Author here. Mimi is Kyutai's. I built a browser UI around it because I wanted to hear what each codebook level does, not just read about it. Curious what you find when you toggle them.
Matches my experience and that of my vibe coding community. I built claudedumb.com to help track these sorts of anecdotes. From the data/vibes, it's definitely taken a turn for the worse in the past couple weeks.
How confident are you in the opus 4.6 model size? I've always assumed it was a beefier model with more active params that Qwen397B (17B active on the forward pass)