Despite enjoying your idea, I probably won't do that, but certainly anyone who actually has this same problem should reach out to me and I'll put them in touch with the right folks at Apple, who can share the info.
Hey everyone, it worked, I had a super productive conversation with exactly the right person on the Metal team! Thanks for helping me get Apple's attention. I didn't at all expect this amount of support.
I attempted to preempt your question in the section of my blog post, "Why don’t you just pipeline the GPU code so that it saturates the GPU?" It's one of the less-detailed sections though so maybe you have further questions? I think the main thing is that since Anukari processes input like MIDI and audio data in real-time, it can't work ahead of the CPU, because those inputs are not available yet.
Possibly what you describe is a bit more like double-buffering, which I also explored. The problem here is latency: any form of N-buffering introduces additional latency. This is one reason why some gamers don't like triple-buffering for graphics, because it introduces further latency between their mouse inputs and the visual change.
But furthermore, when the GPU clock rate is too low, double-buffering or pipelining don't help anyway, because fundamentally Anukari has to keep up with real time, and every block it processes is dependent on the previous one. With a fully-lowered GPU clock, the issue does actually become one of throughput and not just latency.
It's a real-time audio app, so if it falls behind real time, no audio. You get cracks, pops, and the whole thing becomes unusable. If the user is doing audio at 48 kHz, the required latency is 1/48,000 seconds per sample, or realistically somewhat less than that to account for variance and overhead.
1. Anukari runs up to 16 entire copies of the physics model for polyphony, so 16 * 1024 * 48K (I should update the blog post)
2. Users can arbitrarily connect objects to one another, so each object has to read connections and do processing for N other entities
3. Using the full CPU requires synchronization across cores at each physics step, which is slow
4. Processing per object is relatively large, lots of transcendentals (approx OK) but also just a lot of features, every parameter can be modulated, needs to be NaN-proof, so on
5. Users want to run multiple copies of Anukari in parallel for multiple tracks, effects, etc
Another way to look at it is: 4 GHz / (16 voice * 1024 obj * 4 connections * 48,000 sample) = 1.3 cycles per thing
The GPU eats this workload alive, it's absolutely perfect for it. All 16 voice * 1024 obj can be done fully in parallel, with trivial synchronization at each step and user-managed L1 cache.
Thanks for the thought, unfortunately when running as a plugin Anukari is subject to whatever plist.txt the host application uses. I think that I did try that with the standalone binary at one point, but unfortunately I did not appear to take notes! That probably means I did not have success.
For anyone seeing this post a bit late: I need a bit of help from someone inside Apple who works on Metal. If you know someone, it would be great if you could connect me to them:
In that thread, the topic of macOS performance came up there. Basically Anukari works great for most people on Apple silicon, including base-model M1 hardware. I've done all my testing on a base M1 and it works wonderfully. The hardware is incredible.
But to make it work, I had to implement an unholy abomination of a workaround to get macOS to increase the GPU clock rate for the audio processing to be fast enough. The normal heuristics that macOS uses for the GPU performance state don't understand the weird Anukari workload.
Anyway, I finally had time to write down the full situation, in terrible detail, so that I could ask for help getting in touch with the right person at Apple, probably someone who works on the Metal API.
My RTX2080Ti (admittedly not a cheap card) supports 768 physics objects (masses) per each of 16 voices. And I think that beefier cards can do 1024. There are other limitations for performance depending on how the system is constructed, and I certainly don't want to claim that I know it can simulate a piano, but...
I really like doing deep optimization work, so probably the GPU stuff was more fun from an engineering geek perspective. But the physics was also super fun, experimenting with how to model each object, and so on. The physics is a little more playful, since changing a tiny bit of simple math often does fun/weird things.
RE strings/air, I have thought about it, but only a little! Down the road I really want to explore more physics objects. It's very fun to think about how I'd integrate strings into the world, especially w.r.t. how they'd interact and connect with the masses. It seems like there could possibly be some very cool ways to do it.
Glad to hear it. I have fun writing them, often I find it's a great way to clarify my thoughts even just for me. But also I enjoy reading other people's devlogs so am glad to contribute. :)
This is a long story, which is still ongoing. The GPU code is very, very heavily optimized (though I do still have some ideas on how to go further). The main problem we're having on Mac hardware is that the OS heuristics for when to turn the clock rate up on the GPU work really poorly for the audio use case. If you want gory details, I've written about it:
The animations have some subtlety, but the basic idea is that they all to from t=0 to t=1, and then Anukari drives that t parameter. So for example, the mallet would go from resting at t=0 to fully extended at t=1, and Anukari will animate it at the right speed from 0, up to some value based on the velocity, back to zero.
Some of the objects are cyclic, like the spinning oscillators, and those also use the t=0 to t=1 convention, but require that the animation is seamless (so it looks identical at t=0 and t=1, but does whatever you want in between).
I'm not aware of anyone fully customizing the 3D stuff yet, so the instructions may be a bit rough. If you try it, feel free to contact me at [email protected] or on discord and I'd be happy to answer questions.
> What if you make a circular string and attach 1 or more oscillators at random points?
If your computer meets the system requirements, you could always install the free demo and build this sound ring instrument to find out! Building these kinds of weird ideas and seeing what happens is my favorite thing to do with it.
Honestly I just haven't given wind models much thought yet! Long-term once it's stable and fast everywhere, I look forward to adding more physics features, and this is the kind of thing I'd love to look into.
I plan to write a devlog post about this at some point, but at one time I wondered the same thing. It turns out that starting in 2D, when there is more than 1 spring, things get more complex than you expect: it turns non-linear. If the mass is moved exactly along the axis of the 2 springs in the demo you linked, then it's just a simple harmonic oscillator. But if it's moved OFF axis, the force from the springs is constantly changing direction. The force orthogonal to the spring's resting axis increases with the sine of the angle that the spring has been pulled off to. So along the off-axis it is not a harmonic oscillator.
If this doesn't make sense it's entirely my fault for explaining it poorly -- I really want to write this up properly with some diagrams because it's a surprising thing and quite interesting, at least to me.
(BTW that demo has other things going on like mic isotropy and analog oscillators attached to vibrate two of the masses, so my spring tangent is not the only thing producing harmonics.)
Short answer: it has been a big pain in the butt. The GPU hardware is mostly really great, but the drivers/APIs were not designed for such a low-latency use case. There's (for audio) a large overhead latency in kernel execution scheduling. I've had to do a lot of fun optimization in terms of just reducing the runtime of the kernel itself, and a lot of less-fun evil dark magic optimization to e.g. trick macOS into raising the GPU clock speed.
Long answer: I've written a fair bit about this on my devlog. You might check out these tags:
No wind-type model so far, though the bow model can do some rather flute-y sounds (in fact for the current bow model, I might even argue that it is better for pan flute stuff than sounding like a realistic bow).