Riffusion - Generative AI for Music | Research Scientist, Research Engineer | San Francisco | Full-time
Riffusion is a small team training foundation models for music generation and building products that create more musicians in the world. We strive to create and deploy models that are expressive, fast, controllable, and inspiring at scale.
We’re establishing our founding research team and looking for individuals who love music and are excited to build a more creative future with us. Experience with large scale generative model training and diffusion architectures is preferred. Very strong software engineering and computer science fundamentals required. We’re backed by top investors and have substantial compute at the ready.
You can make music with us https://riffusion.com and reach me at { seth at riffusion dot com }.
Riffusion - Generative AI for Music | Research Scientist, Research Engineer | San Francisco | Full-time
Riffusion is a small team training foundation models for music generation and building products that create more musicians in the world. We strive to create and deploy models that are expressive, fast, controllable, and inspiring at scale.
We’re establishing our founding research team and looking for individuals who love music and are excited to build a more creative future with us. Experience with large scale generative model training and diffusion architectures is preferred. Very strong software engineering and computer science fundamentals required. We’re backed by top investors and have substantial compute at the ready.
You can make music with us https://riffusion.com and reach me at { seth at riffusion dot com }.
Nice! It is best at pronouncing in English, but we've had a bunch of fun trying to get other languages too. Sometimes you can make things happen phonetically.
Even for english words that it doesn't get right the first time haha
It's still really early innings for this technology, so we're happy to be learning and building fun technology helps people to do creative things. Main thing we're focused on is turning it from a toy you enjoy once, to something you come back to and dive deeper into (even if it's still just for fun).
Author here: fwiw we are running the app on a10g GPUs, which generally can turn around a 512x512 in 3.5s with 50 inference steps. This time includes converting the image into audio which should be done on the GPU as well for real-time purposes. We did some optimization such as a traced unet, fp16 and removing autocast. There are lots of ways it could be sped up further I'm sure!
Author here: We were blown away too. This project started with a question in our minds about whether it was even possible for the stable diffusion model architecture to output something with the level of fidelity needed for the resulting audio to sound reasonable.
Author here: It can certainly be applied to voice, but the model would need deeper training to speak intelligibly. If you want to hear more singing, you can try a prompt like "female voice", and increase the denoising parameter in the settings of the app.
That said, our GPUs are still getting slammed today so you might face a delay in getting responses. Working on it!
Author here: Indeed we are using Griffin-Lim. Would be exciting to swap it out with something faster and better though. In the real-time app we are running the conversion from spectrogram to audio on the GPU as well because it is a nontrivial part of the time it takes to generate a new audio clip. Any speed up there is helpful.
We’re establishing our founding research team and looking for individuals who love music and are excited to build a more creative future with us. Experience with large scale generative model training and diffusion architectures is preferred. Very strong software engineering and computer science fundamentals required. We’re backed by top investors and have substantial compute at the ready.
You can make music with us https://riffusion.com and reach me at { seth at riffusion dot com }.