Show HN: Web Audio Spring-Mass Synthesis(blog.cochlea.xyz)
blog.cochlea.xyz
Show HN: Web Audio Spring-Mass Synthesis
https://blog.cochlea.xyz/string.html
19 comments
Great demo.
I used to do some web audio and tonejs works, but later switched to rust and glicol for sound synthesis.
For example, this handwritten dattorro reverb:
https://glicol.org/demo#handmadedattorroreverb
This karplus-stress-tester may also be interesting to you.
https://jackschaedler.github.io/karplus-stress-tester/
In short, I think to study more powerful physics synthesis, you need to consider the technology stack of
- rust -> wasm - audioworklet - sharedarraybuffer
Visual can rely on wgpu. Of course, webgl is enough in this case imho.
If it is purely desktop, you can consider using the physics library in bevy.
I used to do some web audio and tonejs works, but later switched to rust and glicol for sound synthesis.
For example, this handwritten dattorro reverb:
https://glicol.org/demo#handmadedattorroreverb
This karplus-stress-tester may also be interesting to you.
https://jackschaedler.github.io/karplus-stress-tester/
In short, I think to study more powerful physics synthesis, you need to consider the technology stack of
- rust -> wasm - audioworklet - sharedarraybuffer
Visual can rely on wgpu. Of course, webgl is enough in this case imho.
If it is purely desktop, you can consider using the physics library in bevy.
You're the author of Glicol, right? I've definitely had my eye on trying it out for a while. The karplus-stress-tester is great; I'm currently using message ports, because they seemed most accessible at first, but I'm happy to know there are other, and better options. I have done quite a bit of hand-optimizing of the code here, and while I think there's probably juice left to squeeze, it has become apparent to me that wasm is probably my next stop.
I've written one other AudioWorklet at this point, which just runs "inference" on a single-layer RNN given a pre-trained set of weights: https://blog.cochlea.xyz/rnn.html. It has similarly mediocre performance.
Thanks for all the great tips, and for your work on Glicol!
I've written one other AudioWorklet at this point, which just runs "inference" on a single-layer RNN given a pre-trained set of weights: https://blog.cochlea.xyz/rnn.html. It has similarly mediocre performance.
Thanks for all the great tips, and for your work on Glicol!
If you like this you might also like Anukari's 3D Physics Synthesizer.
It is also a spring-mass synth but with Midi and audio and much more options.
https://anukari.com/
It is also a spring-mass synth but with Midi and audio and much more options.
https://anukari.com/
This is cool - there’s some similar work here https://arxiv.org/pdf/2402.01571 which uses spiking neural networks (essentially Dirac pulses). I think the next step for this would be to learn a tonal embedding of the source alongside the event embedding so that you don’t have to rely on physically modelled priors. There’s some interesting work on guitar amp tone modelling that’s doing this already https://zenodo.org/records/14877373
How funny, I actually corresponded with one of the authors of the "Spiking Music..." paper when it first showed up on arxiv. I'll definitely give the amp-modeling paper a read, looks to be right up my alley!
Now that I understand the basics of how this works, I'd like to use a (much) more efficient version of the simulation as an infinite-dataset generator and try to learn a neural operator, or NERF like model that, given a spring mesh configuration, a sparse control signal, and a time, can produce an approximation of the simulation in a parallel and sample-rate-independent manner. This also (maybe) opens the door to spatial audio, such that you could approximate sound-pressure levels at a particular point in time _and_ space. At this point, I'm just dreaming out-loud a bit.
Now that I understand the basics of how this works, I'd like to use a (much) more efficient version of the simulation as an infinite-dataset generator and try to learn a neural operator, or NERF like model that, given a spring mesh configuration, a sparse control signal, and a time, can produce an approximation of the simulation in a parallel and sample-rate-independent manner. This also (maybe) opens the door to spatial audio, such that you could approximate sound-pressure levels at a particular point in time _and_ space. At this point, I'm just dreaming out-loud a bit.
This is possible but very very hard! Actually getting the model to converge on something that sounds reasonable will make you pull your hair out. It’s definitely a fun and worthwhile project though. I attempted something similar a few years ago. Good luck!
First of all - very cool, played with this for a few minutes before discovering the following bug:
In Safari (iOS 18.3.1) if you set the Mass slider to 0 and increase Tension, not only does the app crash, but a repeated clicking noise starts, and persists, even after the tab is closed - even after Safari itself is closed! Seems to be a Safari bug. I have reproduced it 3x.
If you reproduce this and want the noise to go away you have to start another app that tries to play sound.
In Safari (iOS 18.3.1) if you set the Mass slider to 0 and increase Tension, not only does the app crash, but a repeated clicking noise starts, and persists, even after the tab is closed - even after Safari itself is closed! Seems to be a Safari bug. I have reproduced it 3x.
If you reproduce this and want the noise to go away you have to start another app that tries to play sound.
Very cool! I've often wondered whether one could procedurally generate sounds of objects interacting in a physics engine? This approach seems like a good place to start.
You (or anyone interested in sound/physics/code) may find this video / series interesting
https://youtu.be/RKT-sKtR970?si=qvTXK6YSMsPaE-GE
https://youtu.be/RKT-sKtR970?si=qvTXK6YSMsPaE-GE
Same here! Not a physics engine per se, but I've been eyeing Taichi Lang (https://github.com/taichi-dev/taichi) as a potential next stop for running this on a much larger scale.
My assumption has been that any physics engine that does soft-body physics would work in this regard, just at a much higher sampling rate than one would normally use in a gaming scenario. This simulation is actually only running at 22050hz, rather than today's standard 44100hz sampling rate.
My assumption has been that any physics engine that does soft-body physics would work in this regard, just at a much higher sampling rate than one would normally use in a gaming scenario. This simulation is actually only running at 22050hz, rather than today's standard 44100hz sampling rate.
Interesting, I'm into physical modeling and learning DSP.
I did notice glitching in latest Firefox on a Mac, like I'd get running a DAW with too small a buffer...While the tab was open I got similar crackles and slightly delayed audio stream playing YouTube vids in other tabs.
I did notice glitching in latest Firefox on a Mac, like I'd get running a DAW with too small a buffer...While the tab was open I got similar crackles and slightly delayed audio stream playing YouTube vids in other tabs.
Thanks for the bug report, I'll look into this and see if I can make it better! There's definitely more optimization juice to be squeezed, I think, and it'd probably be smart to allow the number of nodes in the simulation to be adjusted.
Working with AudioWorklets (https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle...) has been really cool, and I've been surprised at what's possible, but I _haven't_ yet figured out how to get good feedback about when the custom processor node is "falling behind" in terms of not delivering the next buffer quickly enough.
Working with AudioWorklets (https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle...) has been really cool, and I've been surprised at what's possible, but I _haven't_ yet figured out how to get good feedback about when the custom processor node is "falling behind" in terms of not delivering the next buffer quickly enough.
This is fun, but if you reduce the tension and click a lot on it, it gets stuck with almost all the points at the top of the screen except one. Also weirdly in Firefox android it clicks a lot, not on chrome.
Thanks for the bug report, I definitely do see now that things fall apart at certain extreme values. I'll look into it and see if I can fix. My assumption is that it's some sort of numeric overflow or underflow resulting in NaN values, and that I'll probably just need to set the slider boundaries more carefully. Thanks again!
Here's a simple, but high quality model of piano and violin: https://mattdailis.github.io/simulating-strings/resources/ph...
amazing! there are a couple settings here where it sounded like how i wish the String engine on my old OP-1 sounded. the random setting is really fun
Please plonk a { color: lightblue; } on the css.
[deleted]
My current area of research is in sparse, event-based encodings of musical audio (https://blog.cochlea.xyz/sparse-interpretable-audio-codec-pa...). I'm very interested in decomposing audio signals into a description of the "system" (e.g., room, instrument, vocal tract, etc.) and a sparse "control signal" which describes how and when energy is injected into that system. This toy was a great way to start learning about physical modeling synthesis, which seems to be the next stop in my research journey. I was also pleasantly surprised at what's possible these days writing custom Audio Worklets!