Getting Audio Visualizations Working with Web Audio API(dwayne.xyz)
dwayne.xyz
Getting Audio Visualizations Working with Web Audio API
https://dwayne.xyz/post/audio-visualizations-web-audio-api
15 comments
I see the topic of audio visualization is getting more attention. Coincidentally, I've been working on another kind of sound visualization, based on FFT, but with a few tricks that create interesting visuals. Check out "github soundshader". I'll make a post on HN once my account becomes less green.
That’s such a cool project! I really like the imagery you were able to produce.
One of the next few things I want to work on is more interesting visualization options.
Just so you know, I got an error when trying to use the mic on mobile Safari when using your live demo. this.audioCtx was null when calling .createMediaStreamSource.
One of the next few things I want to work on is more interesting visualization options.
Just so you know, I got an error when trying to use the mic on mobile Safari when using your live demo. this.audioCtx was null when calling .createMediaStreamSource.
Ha, thanks! And those visuals are still quite simplistic as I haven't implemented yet all the ideas to visualize tiny wavelets present in any realistic sound.
A few possible next steps I had in mind:
(1) Turning those ACFs into actual shapes, not just colored rings assembled together, so say an accord on piano would create a sophisticated 2d or 3d shape with elaborate structure. Perhaps, just mixing the flat ACFs with an exp decay would achieve the result.
(2) Visualizing the ACF heights map as a water surface with a raytracer. This sounds nonsense until you see a few real (physical) examples: "cymaglyphs". Those are reflections and refractions on the rippled water surface that's assuming shapes of the so-called Faraday waves (a parametrically driven wave equation, see "Numerical simulation of Faraday waves" by L. Tuckerman on arxiv). However this requires solving a complex diff equation + an extremely high quality raytracer, as otherwise you'd get blurry pixelated images. None of the existing raytracers would work here.
As for compatibility with other browsers, it works on desktop Chrome and Firefox, but not on their mobile versions. I haven't figured out yet why. Must be something to do with GPU differences. Safari is a whole another story. The visualizer uses WebAudio + WebGLv2, and Safari isn't known for supporting the bleeding edge standards. While you could do everything without GPU, it would crawl at 0.03 fps and you wouldn't be hear and see sound in real time.
A few possible next steps I had in mind:
(1) Turning those ACFs into actual shapes, not just colored rings assembled together, so say an accord on piano would create a sophisticated 2d or 3d shape with elaborate structure. Perhaps, just mixing the flat ACFs with an exp decay would achieve the result.
(2) Visualizing the ACF heights map as a water surface with a raytracer. This sounds nonsense until you see a few real (physical) examples: "cymaglyphs". Those are reflections and refractions on the rippled water surface that's assuming shapes of the so-called Faraday waves (a parametrically driven wave equation, see "Numerical simulation of Faraday waves" by L. Tuckerman on arxiv). However this requires solving a complex diff equation + an extremely high quality raytracer, as otherwise you'd get blurry pixelated images. None of the existing raytracers would work here.
As for compatibility with other browsers, it works on desktop Chrome and Firefox, but not on their mobile versions. I haven't figured out yet why. Must be something to do with GPU differences. Safari is a whole another story. The visualizer uses WebAudio + WebGLv2, and Safari isn't known for supporting the bleeding edge standards. While you could do everything without GPU, it would crawl at 0.03 fps and you wouldn't be hear and see sound in real time.
Cool, I built one too in Observable using webgl and canvas exporting lifting audio from SoundCloud https://observablehq.com/@tomlarkworthy/sound-cloud-reactive... it's a good place to start if you want to hack your own as you can fork the notebook and code right in your browser
MDN has a good coverage on Web Audio API, including an example of how to set up animation frame-based canvas-drawn visualization. The browser does all the heavy lifting, including the FFT, making the whole process really quite trivial especially when compared to doing the same with a native desktop program.
https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_A...
https://github.com/mdn/webaudio-examples/
https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_A...
https://github.com/mdn/webaudio-examples/
Another here
https://stackoverflow.com/a/41443535
https://stackoverflow.com/a/41443535
Regarding the Safari issue and the need for a AudioContext.resume() call, this is due to Safari suspending the AudioContext on creation, this is probably a good behaviour but it should be resumed automatically, as explained in the thread here : https://stackoverflow.com/questions/56768576/safari-audiocon...
Not working for me on Safari Version 14.0 (15610.1.28.1.9, 15610) Catalina 10.15.7 (19H2)
Working on Chrome Version 86.0.4240.80 (Official Build) (x86_64) same machine
Working on Chrome Version 86.0.4240.80 (Official Build) (x86_64) same machine
Not working after the resume() call? Are you getting an error message or just missing data?
I'm clicking on your embedded demo that you introduce with Here's what I came up with: and caption "The local audio visualisation. I cycle through bar graph in light mode, bar graph in dark mode, sine wave in dark mode, then sine wave in light mode."
On Chrome, the mute button defaults to the muted state and never responds to a click. Pause/play always works to toggle the animated display with every click.
On Safari, the pause button is unresponsive. The mute button toggles the animated display with every click, and changes state between muted and unmuted. When muted, animation is paused. When unmuted, animation is active. When muted and animation paused, a click on the play button does not change the paused animation state, but does result in the triangle symbol displaying smaller during button down, then restore to larger when button up. Sometimes I can get the play button to work (toggle animation on/off) after clicking in the progress bar a number of times and toggling the animation state that way, after which (sometimes) the play button works, but not always.
Also, the display of controls is different on Chrome than Safari.
Chrome, progress bar is a full width bar underneath the other controls.
Safari, progress bar is in line with the other controls, which include -15 and +15 either side of Play. Those skip controls are not present on Chrome.
On Chrome, the mute button defaults to the muted state and never responds to a click. Pause/play always works to toggle the animated display with every click.
On Safari, the pause button is unresponsive. The mute button toggles the animated display with every click, and changes state between muted and unmuted. When muted, animation is paused. When unmuted, animation is active. When muted and animation paused, a click on the play button does not change the paused animation state, but does result in the triangle symbol displaying smaller during button down, then restore to larger when button up. Sometimes I can get the play button to work (toggle animation on/off) after clicking in the progress bar a number of times and toggling the animation state that way, after which (sometimes) the play button works, but not always.
Also, the display of controls is different on Chrome than Safari.
Chrome, progress bar is a full width bar underneath the other controls.
Safari, progress bar is in line with the other controls, which include -15 and +15 either side of Play. Those skip controls are not present on Chrome.
Ah. I should have made that part a little clearer. That's just a video file of a screen recording showing me cycling through display modes.
It sounds like you're describing some cross browser issues playing that mp4 file.
Thanks for the feedback though. I'll try to reword that section so it's more clear.
It sounds like you're describing some cross browser issues playing that mp4 file.
Thanks for the feedback though. I'll try to reword that section so it's more clear.
Yes! Sorry for the confusion. Thanks!
I made https://simplespectrogram.app/ which y’all may also enjoy. With the web audio API I was able to do it in less than 100 LoC - take a look at source
Where is the demo? Why isnt the demo at top?
Good point. I’m writing a bigger article on the whole WebRTC video chat project (which the audio visualizations are a part of). I’ll have some kind of demo ready for that one, although it’s a little awkward as everything was designed for one on one chats.