I dont actually see any tokens used in the model. It seems like the model actually predicts latents and then VAE converts back to audio. More like Tortoise or XTTS
Can one of the authors explain what this actually means from the post?
hertz-vae: a 1.8 billion parameter transformer decoder which acts as a learned prior for the audio VAE. The model uses a context of 8192 sampled latent representations (17 minutes) and predicts the next encoded audio frame as a mixture of gaussians. 15 bits of quantized information from the next token act as semantic scaffolding to steer the generation in a streamable manner.
I've been using Coqui Studio for a while now to generate AI voices for my video game project. It delivers most that is in the blogpost. I like it but I'd ask for a better API support if anyone sees this from Coqui.