> It runs but any audio input (you will need to provide wav not mp3's) I tried (tried 20s/40s/300s) I get just one short sentence returned in target language that seems not related at all to my audio input (i.e. Tous les humains sont créés égaux).
You might want to open an issue on github for that one.
The model is made to work on short utterances, if you have a long speech, you'll want to segment it first.
I've tried "tous les humains sont créés égaux" on the demo: https://seamless.metademolab.com/expressive (which runs the same code as in the repo) and the output was correct. Maybe there is something wrong going on in the conversion of the input audio?
> Oh and why is Whisper a dependency? Seems not need if FB has their own model?
Whisper is a dependency as it's used as a baseline for evaluation. You can check out the paper for explanations.
Meta employee here. The system is not perfect, or it would not "hallucinate", while it's pretty good, it does sometime make errors (not just hallucination, maybe just some mistranslation due to noise in the training data). What we want is to avoid these errors to introduce toxicity (think swear words) that weren't in the input as this could be very bad for the user. There is a separate system that double checks the output (compared to the input) and tells the translation model to try again if it's too bad.
it only kicks-in if the output is more "toxic" than the input. If the input has a lot of swear words and the output has the same amount, then it will be left alone.
Sorry if I wasn't clear, internally we've been talking about it a lot, but I forgot that it doesn't have such a solid definition outside of our work. Thankfully, we try to define it in section 7.3 of the NLLB paper: https://arxiv.org/pdf/2207.04672.pdf
The tldr is that if you say:
"Thank you for this job offer." you wouldn't want it to be (mis)translated as "Go F*k yourself.". But if you do say "Go F yourself", you still want it to be translated as that.
Our goal is to have a good recall, sometimes to the detriment of precision, so for words with multiple meanings, it might consider them toxic when in the actual context they are used in, they are not. The toxicity mitigation algorithm will search for alternative translations that have the correct meaning but not the potentially toxic word so that there is no added toxicity in the output. This means that sometimes the model might prefer a less coloquial phrasing than what a human would.
You can find details on how the multi-language creation of the toxicity lists was done in section 7.3 of the NLLB paper: https://arxiv.org/pdf/2207.04672.pdf. TLDR: it's not just a translation of a base English list, even if we started from that, each language has a curated list that was built by professional translators.
Hi, I work on seamless. What this refers to is added toxicity mitigation. We try to detect the level of toxicity in the input and make sure that the output toxicity level is not higher. This protects the model from doing egregious errors in the translation.
There are more details in the paper if you want and the mitigation code is all open source if you want to check what it actually does.
• 800 dynamic scenarios across ten realistic universes
• Tests adaptability, robustness to failure, and time sensitivity
• Moves beyond static benchmarks to evaluate real-world agent capabilities
- Agents Research Environments (ARE): A simulation platform for agents research
• Dynamic, evolving environments that mirror real-world complexity
• Built-in reward signals and comprehensive evaluation tools
• Realistic apps (email, calendar, file system, messaging) with realistic data
• Event-driven architecture that creates dynamic scenarios for multi-turn tasks