The alignment between source and target is automatically inferred, basically by searching when the uncertainty over a given output word reduces the most once enough input words are seen. This is then lifted to the audio domain. In theory the same trick should work even with longer grammatical inversions between languages, although this will lead to larger delays. To be tested!
Adding more languages is definitely planned! This was Tom (the first author) master’s internship project with Kyutai, and it was easier to prototype the idea with a single pair. Also he will be presenting this work at ICML in two weeks if anyone is around and wants to learn more.
Dr. Jean Remi King is leading the research effort at Meta AI Research for computational neuroscience, including this work. You should check out his website to learn more : https://kingjr.github.io/
Amazing scientist, previously at ENS Cognitive studies department in Paris.