Show HN: Demon – open-source real-time music diffusion engine, 25Hz local GPU(daydreamlive.github.io)
daydreamlive.github.io
Show HN: Demon – open-source real-time music diffusion engine, 25Hz local GPU
https://daydreamlive.github.io/DEMON/
15 comments
Wow this is so cool! Do we have VST support?
can you explain performance differences vs Magenta / Lyria? that's the most similar thing I can think of
Good question. First and foremost, Magenta is autoregressive, and this is not. Practical differences:
THIS:
- has more controls; output can be highly custom
- has very fast response time for updating controls; this approaches real time
- runs on consumer cards
- is optimized for loops, and full songs, not continuous infinite music
MAGENTA: - can generate music forever, it plainly wins here - has fewer controls - has high parameter update latency (chunks of 2 seconds) - I did get magenta to fit on a 5090, but just barely
MAGENTA: - can generate music forever, it plainly wins here - has fewer controls - has high parameter update latency (chunks of 2 seconds) - I did get magenta to fit on a 5090, but just barely
Mind blown! This is absolutely sick.
[deleted]
this is sick and why not bro!!
what was the biggest unexpected unlock you got while building this? like was there something that just worked out unexpectedly easy compared to the rest?
also, another question, how much time it took you to build this thing all together?
what was the biggest unexpected unlock you got while building this? like was there something that just worked out unexpectedly easy compared to the rest?
also, another question, how much time it took you to build this thing all together?
biggest unexpected unlock i would have to say is the shared mutable curves that bypass the ringbuffer drain. i wouldnt have thought of that without the throughput that preceded it
very cool!
oh and as for time - this is several months of work, but is the culmination of several years of audio adjacent genai work
nice to see it all coming together. curious about the VST soon too!
[deleted]
I’m Ryan, lead author. I’ve been contributing open source generative audio stuff for a while now, audio reactive Comfy nodes, extended ACEstep support in Comfy, etc.. I just opened-sourced a new audio project that I've been working on for a few months and I want to tell y'all about it.
WHAT IS IS DEMON: Diffusion Engine for Musical Orchestrated Noise
This is StreamDiffusion but with audio instead of images, and ACEStep 1.5 instead of Stable Diffusion. It’s responsive enough that you can play it like an instrument, and remix in near real-time.
I also distilled the ACEStep VAE: it’s faster at the expense of some quality.
I also trained something like 200 lora/dora for ACEStep 1.5 and 1.5XL: I will release these in batches of 5 or 10 or something
WHY IT IS Two reasons: 1) Making music is an inherently real-time activity 2) Why not bro
SOME RUNTIME CAPABILITIES -Real-time remixing of songs -Denoise, structure, timbre strength adjustment -Reference track swapping -Prompt blending, parameter scheduling with curves -LoRA hotswapping, runtime strength adjustment -Latent channel (research preview) -Feedback -Vocal stem cutting/pasting with melformer (s/o u/BuffMcBigHuge) -XL support (its less stable, working out VRAM pressure issues and whatnot) -Lyrics/vocals SOON -Spectral quality SOON -Other stuff
SOME LIMITATIONS -ACEStep (correctly) ‘begins’ and ‘ends’ the song. This system is optimized for remixing either an entire song, or continuously remixing a loop. The loop works fine, but this is not pure, continuous music. Autogression wins here. -Many others, for a more exhaustive list, please see the full writeup via the project page -Please let us know if you find any, I would love to try and address them if possible
LINKS My YouTube (DEMON tutorial): https://youtu.be/FBv1b5gmjcE Github: https://github.com/daydreamlive/DEMON Project page: https://daydreamlive.github.io/DEMON LoRA: https://civitai.com/models/2416425/acestep-loras DreamVAE: https://huggingface.co/daydreamlive/DreamVAE Try it w/o installing: https://music.daydream.live