Audio libraries considered challenging (2022)(tesselode.github.io)
tesselode.github.io
Audio libraries considered challenging (2022)
https://tesselode.github.io/articles/audio-libraries-considered-challenging/
9 comments
The difference between audio and video is that the sample rates are 44kHz vs 24-60fps. Holding the last frame would be like holding the last sample which is DC (i.e. silent). Replaying the last buffer is what underrun typically does sounding like a fast repeated stutter more akin to replaying the last N frames.
Especially given that the buffer is often short, i.e. far shorter than what the sampling rate implies, in order to reduce latency. The buffer size of 32--1024 samples is common and, assuming 44.1 kHz, translates to 0.7--23 ms of audio to repeat in such cases. The limited buffer size would act as a crude high-pass filter and almost always result in a high-pitched noise.
It is possible to have a larger buffer that can be opportunistically filled for such cases, say, you may fill 1024 future samples while only the first 256 will be processed soon and you don't have to touch the remainder unless something else happened, but AFAIK this is not how most audio APIs work because such opportunistic buffering can be done outside of the API anyway.
It is possible to have a larger buffer that can be opportunistically filled for such cases, say, you may fill 1024 future samples while only the first 256 will be processed soon and you don't have to touch the remainder unless something else happened, but AFAIK this is not how most audio APIs work because such opportunistic buffering can be done outside of the API anyway.
- Some systems (notably, hardware ring buffers) do this, the difference is mainly the kind of noise you hear - instead of crackling, it stutters. Still not an acceptable fallback for game audio.
- Old consoles had dedicated sound processing in them. In fact, they usually had both a dedicated sound mixer and a separate CPU core[0] to put the audio thread on. Hung audio threads in this case would mean musical slowdown rather than stuttering or crackling, which is noticeable in yet another different way.
- Source Engine does this, sort of. It loads just the first half a second of each sound it expects to play in a given level, which is used to hide I/O latency reading from disk. The particular failure mode of this solution, typically caused by a bunch of sounds triggering right after a level load, is that it takes longer than half a second to read all those files, and you hear that half-second sample stuttering instead of the audio.
[0] In several cases the audio CPU core would also serve as the primary CPU in backwards compatibility modes. For example, the Sega Genesis/Mega Drive has a Z80 in it that can run either the audio thread of a Genesis game or the primary thread of a Master System one. The PlayStation 2 does the same thing with having an "I/O processor" that's the PS1 CPU in a trenchcoat. The DS ARM7 is a holdover from the Game Boy Advance and, AFAIK, could also be used as an audio thread CPU if you wanted.
- Old consoles had dedicated sound processing in them. In fact, they usually had both a dedicated sound mixer and a separate CPU core[0] to put the audio thread on. Hung audio threads in this case would mean musical slowdown rather than stuttering or crackling, which is noticeable in yet another different way.
- Source Engine does this, sort of. It loads just the first half a second of each sound it expects to play in a given level, which is used to hide I/O latency reading from disk. The particular failure mode of this solution, typically caused by a bunch of sounds triggering right after a level load, is that it takes longer than half a second to read all those files, and you hear that half-second sample stuttering instead of the audio.
[0] In several cases the audio CPU core would also serve as the primary CPU in backwards compatibility modes. For example, the Sega Genesis/Mega Drive has a Z80 in it that can run either the audio thread of a Genesis game or the primary thread of a Master System one. The PlayStation 2 does the same thing with having an "I/O processor" that's the PS1 CPU in a trenchcoat. The DS ARM7 is a holdover from the Game Boy Advance and, AFAIK, could also be used as an audio thread CPU if you wanted.
> on old consoles, sound used to be _generated_ on the fly…
My recollection as a game producer who worked with musicians around that time is that this was all MIDI-like synthesis (although short, low-quality sampled "special effects" were possible) handled by dedicated hardware — Yamaha YM2612 on Genesis, Ricoh 2A03 on NES, etc. — and so lots more forgiving timing-wise.
My recollection as a game producer who worked with musicians around that time is that this was all MIDI-like synthesis (although short, low-quality sampled "special effects" were possible) handled by dedicated hardware — Yamaha YM2612 on Genesis, Ricoh 2A03 on NES, etc. — and so lots more forgiving timing-wise.
Preventing buffer underruns is one thing, but it's also possible to mitigate their effect.
Voice communication codecs seem to manage well with covering up gaps in the stream - they do various tricks to "continue making the same sound" if there's a short gap, and can even "catch up" by replaying quickly if there's a longer gap. Quite effective.
EDIT: but, of course, a human voice over a phone with a heavy low-pass filter is a much, much easier thing to replicate and "fill the gaps" than full-spectrum, general-purpose audio. Perhaps therein lies the difficulty.
Voice communication codecs seem to manage well with covering up gaps in the stream - they do various tricks to "continue making the same sound" if there's a short gap, and can even "catch up" by replaying quickly if there's a longer gap. Quite effective.
EDIT: but, of course, a human voice over a phone with a heavy low-pass filter is a much, much easier thing to replicate and "fill the gaps" than full-spectrum, general-purpose audio. Perhaps therein lies the difficulty.
Voice comms benefit from already generally working in the frequency domain due to the heavy use of codecs; repeating the current spread of frequencies is much less intrusive than repeating entire samples: chirping or warbling instead of stuttering or clipping.
Pretty sure the source engine used to have single-threaded audio for a long time. They worked around the buffer underrun problem by inserting calls to their sound update function at multiple points during the frame, like during rendering. Completely avoids the locking problem
See also this classic article: http://www.rossbencina.com/code/real-time-audio-programming-...
Some random thoughts:
- instead of having the system run into an “underrun”, perhaps it could use a fallback rule to just loop over the previously known buffer? That would be the equivalent of the last frame being displayed until a new one arrives.
- on old consoles, sound used to be _generated_ on the fly, usually in lockstep/in sync with the game thread. I think this used to work well because then the game would be the only thing running on the system. Not sure how this worked when consoles got dedicated sound cards later to offload sound processing.
- I guess one could load all of the game’s sound at startup, and let the audio thread just switch what to play when, with a default rule of what to play if no command is received. Not great memory-wise, though.