JackTrip WebRTC: high quality, uncompressed, low-delay audio streaming(github.com)
github.com
JackTrip WebRTC: high quality, uncompressed, low-delay audio streaming
https://github.com/JackTrip-webrtc/JackTrip-webrtc
37 comments
As a musician, I am thrilled that JackTrip and Jamulus are available to make real-time collaboration possible but they're not simple to install and configure for everyone. I spent numerous sessions trying to get a non-techie friend's Ubuntu laptop setup with Jamulus but we both got so frustrated we gave up. I hope the next generation brings with it easier setup, there's just so much potential.
+1
I get the problem with JackTrip (though JamTrip[1] looks a good start).
Out of interest, what were the problems with Jamulus? I've found it easier than expected (once I'd got Jack set up and happy / understood). On Windows / MacOS it's actually easier as there are are clear, well-trodden install guides (and no Jack / low-latency kernels to explain).
[1] https://github.com/vicwomg/jamtrip
I get the problem with JackTrip (though JamTrip[1] looks a good start).
Out of interest, what were the problems with Jamulus? I've found it easier than expected (once I'd got Jack set up and happy / understood). On Windows / MacOS it's actually easier as there are are clear, well-trodden install guides (and no Jack / low-latency kernels to explain).
[1] https://github.com/vicwomg/jamtrip
The problem was he's not a techie and only has a Linux laptop because of cost. Trying to get him to understand how to work the shell cost me a year of life.
Last time I've checked, Jamulus is not peer-to-peer but rather server-client and therefore has more latency.
I've just published an article that goes into depth about how to get a JackTrip Virtual Studio installed. This uses an inexpensive Raspberry Pi instead of requiring installing finicky software on a general purpose laptop or other computer. I wrote this while helping a choir get online, and it worked very well. I hope this helps!
https://wmleler1.medium.com/setting-up-your-virtual-studio-6...
https://wmleler1.medium.com/setting-up-your-virtual-studio-6...
This says ultra low delay, but how low? It seems to me like it's still going to be limited by the browser, and then you're going to have network latency on top of that?
Testing in Chrome on Mac OS 11.1, the lowest latency I can get is 23ms while in Firefox it's 14ms: https://www.jefftk.com/test/latency-demo-5
Testing in Chrome on Mac OS 11.1, the lowest latency I can get is 23ms while in Firefox it's 14ms: https://www.jefftk.com/test/latency-demo-5
30ms here in Chrome and 23ms in Firefox here. But yeah, same questions -- is this over the network (or no, that's the point of a p2p connection)? What is this measuring..?
My tool is local: it measures the time from when JS says "send this to the speaker" until JS receives that audio back from the mic. It's the client-side round-trip latency.
(As far as I know, there is no way to get the browser to tell you this latency, and you have to measure it.)
(As far as I know, there is no way to get the browser to tell you this latency, and you have to measure it.)
I'd love for this to make a difference for simple video calls with relatives -- in my imagination, I set it up, and we find out we've been in the audio equivalent of the uncanny valley this whole time. But I'm afraid in reality, the default audio codecs used in WebRTC are probably good enough to make poor microphones and pure connection latency the bigger issue. Am I wrong?
Obviously, even if I'm right, this has a target audience -- musicians, interviewers and other people who talk for a living -- for which it might be fantastic.
Obviously, even if I'm right, this has a target audience -- musicians, interviewers and other people who talk for a living -- for which it might be fantastic.
Truly is it uncompressed PCM WAV audio? In that case 44kHz 16-bit 1-channel is 705 kbps.
Are there not any good nonproprietary lossless low latency audio codecs? A brief look at Wikipedia shows a couple that are 5ms or lower, but either they are proprietary or they are low bitrate.
Are there not any good nonproprietary lossless low latency audio codecs? A brief look at Wikipedia shows a couple that are 5ms or lower, but either they are proprietary or they are low bitrate.
Opus has frame sizes down to 2.5ms. See https://en.m.wikipedia.org/wiki/Opus_(audio_format)#cite_not...
You pay a price for the super-small frame sizes; much worse compression ratios. Frame sizes smaller than 10ms disable the LPC and hybrid coding modes, which are quite advantageous. [0]
In my experiments with transmitting realtime audio signals using Opus, 10ms frame sizes are acceptable for 1-way synchronicity (e.g. if you want the user to perform an action and hear the result as simultaneous with the action) but it's definitely the upper bound. From what I remember my signal processing professor say, the threshold to try and hit is 7ms total latency for truly undetectable processing, but I can't find a reference for that, so take it with a grain of salt.
[0] https://www.opus-codec.org/docs/html_api/group__opusencoder....
In my experiments with transmitting realtime audio signals using Opus, 10ms frame sizes are acceptable for 1-way synchronicity (e.g. if you want the user to perform an action and hear the result as simultaneous with the action) but it's definitely the upper bound. From what I remember my signal processing professor say, the threshold to try and hit is 7ms total latency for truly undetectable processing, but I can't find a reference for that, so take it with a grain of salt.
[0] https://www.opus-codec.org/docs/html_api/group__opusencoder....
See for example:
Robert H. Jack, Tony Stockman, and Andrew McPherson. 2016. Effect of latency on performer interaction and subjective quality assessment of a digital musical instrument. In Proceedings of the Audio Mostly 2016 (AM '16). Association for Computing Machinery, New York, NY, USA, 116–123. DOI: https://doi.org/10.1145/2986416.2986428
https://scholar.google.com/scholar?cluster=13472622195552899...
Robert H. Jack, Tony Stockman, and Andrew McPherson. 2016. Effect of latency on performer interaction and subjective quality assessment of a digital musical instrument. In Proceedings of the Audio Mostly 2016 (AM '16). Association for Computing Machinery, New York, NY, USA, 116–123. DOI: https://doi.org/10.1145/2986416.2986428
https://scholar.google.com/scholar?cluster=13472622195552899...
Jamulus uses Opus for this reason (balance with quality) AIUI
As do SoundJack, SonoBus and probably many other similar apps.
The JackTrip Virtual Studio uses an inexpensive Raspberry Pi, and connects a low latency hardware codec directly into the Pi's bus, to achieve a latency of 1mS. At that level, the only other latency you have to worry about is the speed of light (through the internet). I've just published an article about the Virtual Studio at https://wmleler1.medium.com/setting-up-your-virtual-studio-6...
if you're in this space, consider trying out sonobus also. not the author (though he is a friend of mine), haven't tried it yet myself, but seems to have all the right stuff.
Yes, SonoBus looks really promising! I was surprised and also a bit proud to learn that it uses a fork of my AOO library (https://git.iem.at/cm/aoo/). I have been in touch with Jesse for the last couple of days and I'm (passively) following the development of SonoBus.
AOO is a C/C++ library for flexible on-the-fly multi-channel audio streaming and OSC messaging over local networks and the internet. The repo also includes Pure Data externals and Supercollider extensions.
WARNING: AOO is still alpha and there are breaking changes between the pre-releases, but I hope to publish a final release soon!
AOO is a C/C++ library for flexible on-the-fly multi-channel audio streaming and OSC messaging over local networks and the internet. The repo also includes Pure Data externals and Supercollider extensions.
WARNING: AOO is still alpha and there are breaking changes between the pre-releases, but I hope to publish a final release soon!
Thanks Paul.... happy to try it with you sometime :)
I am wondering whether using AudioWorklets actually has any benefits in this case.
AFAIU the benefit in general is that they run in the audio thread so the data does not need to be handed over to a different thread for processing. In this case however, the data is handed over to the main thread anyway to be send over the data channel.
Has anyone measured the impact of using AudioWorklets this way?
AFAIU the benefit in general is that they run in the audio thread so the data does not need to be handed over to a different thread for processing. In this case however, the data is handed over to the main thread anyway to be send over the data channel.
Has anyone measured the impact of using AudioWorklets this way?
What's the bitrate for these streams?
If you're on mobile, or corporate net, you might not be able to get "punched through" by the other party for actual p2p connection and will need a TURN server.
I'm guessing that an uncompressed data channel with audio in it might send a lot of traffic through it.
If you're on mobile, or corporate net, you might not be able to get "punched through" by the other party for actual p2p connection and will need a TURN server.
I'm guessing that an uncompressed data channel with audio in it might send a lot of traffic through it.
Is it possible to connect this to an RTMPS stream as well or does that defeat the purpose? Can only people Joined in the room get the stream?
"low-delay", "WebRTC" ... choose one.
I really don't get why everything needs to be in a web browser nowadays. WebRTC is simply the wrong choice for low-delay audio streaming. Audinate's DANTE protocol has been around for 14 years, is affordable, hardware-accelerated, and has practically no avoidable latency when sending audio over the network.
Why add 20+ms of latency just to use WebRTC?
I really don't get why everything needs to be in a web browser nowadays. WebRTC is simply the wrong choice for low-delay audio streaming. Audinate's DANTE protocol has been around for 14 years, is affordable, hardware-accelerated, and has practically no avoidable latency when sending audio over the network.
Why add 20+ms of latency just to use WebRTC?
Completely agree, while proprietary what Audinate has done with Dante is nothing short or a miracle. With their PCI-E FPA I have less then a 2ms round trip latency with my Hilo interface at 96Khz resolution. Additionally the virtual sound card can do 4ms latency in one direction with most modern CPUs.
[deleted]
Dante is proprietary for one thing.
Here's a GPL implementation
https://github.com/jsharkey/wycliffe
Thanks! Have you had any luck running it? It doesn't seem to have any documentation.
Good question; JackTrip normally uses UDP and I think most browsers do not support sending UDP packets directly without an extension.
If you could send UDP directly then WebRTC could presumably be avoided.
If you could send UDP directly then WebRTC could presumably be avoided.
Doesn’t webrtc use UDP?
In some cases on some web browsers WebRTC can be configured to use UDP.
But that's exactly the problem. Usually, nobody knows what's actually happening in that 20000+ lines of code WebRTC stack.
But that's exactly the problem. Usually, nobody knows what's actually happening in that 20000+ lines of code WebRTC stack.
This is great - should make it a lot easier to use JackTrip.
Er, what is this project exactly?
The README says “Multi-machine network music performance over the Internet is achieved” but then later describes how to create multi-person video chat sessions. Comments mention music.
What does this have to do with music? Is it just that remote live music collaboration and multi-machine audio playback are use cases for low-latency codec implementations?
Because it seems like this is an audio codec implemented in the browser that comes with a video chat demo. Am I missing something?
The README says “Multi-machine network music performance over the Internet is achieved” but then later describes how to create multi-person video chat sessions. Comments mention music.
What does this have to do with music? Is it just that remote live music collaboration and multi-machine audio playback are use cases for low-latency codec implementations?
Because it seems like this is an audio codec implemented in the browser that comes with a video chat demo. Am I missing something?
It's a browser implementation of JackTrip, which is low-latency audio conferencing software for musicians, because video conferencing (Zoom etc.) has terrible latency.
The way to have rehearsals over the internet is usually to use JackTrip for the audio and (Zoom etc.) for the video (which can be out of sync; obviously conductors have to tap in an audio metronome channel since video won't be in sync.)
The way to have rehearsals over the internet is usually to use JackTrip for the audio and (Zoom etc.) for the video (which can be out of sync; obviously conductors have to tap in an audio metronome channel since video won't be in sync.)
This video might have some more context in how it relates to musicians:
https://www.youtube.com/watch?v=Sj6Ij1Oxe1s
https://www.youtube.com/watch?v=Sj6Ij1Oxe1s
If you've ever been in a recording studio with headphone monitoring / studio talkback, it's like that. Really intimate.
This project uses RTCDataChannel for audio, which is very neat, but it's a shame that once again audio on the web has to be hacked to perform well.