Ultra Low Latency WebRTC Streaming – Open-Source Media Server(antmedia.io)
antmedia.io
Ultra Low Latency WebRTC Streaming – Open-Source Media Server
https://antmedia.io/
74 comments
The sad part about selective forwarding units (SFU) is you loose end-to-end-encryption. So you only have encryption between you and the sfu and sfu to your participant. For hosted solutions this means a server could record all your conferences.
Therefore i am waiting for the PERC standard. It will allow to send the audio/video to server only once and the server will retransmit to all peers - everything still end-to-end encrpted.
Therefore i am waiting for the PERC standard. It will allow to send the audio/video to server only once and the server will retransmit to all peers - everything still end-to-end encrpted.
How does the key distribution work for that?
How's the performance and features of mediasoup compared to this? I have been looking for an easy to use DFU.
Haven't benchmarked ant media server, but mediaSoup performance is very good.
"Easy to use" is in the eye of the beholder. Mediasoup isn't a packaged solution; you need to write some javascript to get a server up and running. From this perspective it's a framework that sits at the same level as, say, Express. But the API is flexible and powerful. It's exactly what you want if you're building an SFU cluster for a specific use case and want to decide on the architecture yourself. Here's excellent "1:N broadcast" sample code: [0]
With that sample code, you can broadcast a single WebRTC stream to ~500 clients on one vCPU of an AWS c5.large instance. To scale to more clients you need to use more than one vCPU. The new mediasoup v3 APIs make this pretty easy. There are now "pipe transports" and "plain rtp transports" for moving the low-level rtp bits around between processes and between machines.[1]
[0]-https://github.com/michaelfig/mediasoup-broadcast-example [1]-https://mediasoup.org/documentation/v3/differences-between-v...
"Easy to use" is in the eye of the beholder. Mediasoup isn't a packaged solution; you need to write some javascript to get a server up and running. From this perspective it's a framework that sits at the same level as, say, Express. But the API is flexible and powerful. It's exactly what you want if you're building an SFU cluster for a specific use case and want to decide on the architecture yourself. Here's excellent "1:N broadcast" sample code: [0]
With that sample code, you can broadcast a single WebRTC stream to ~500 clients on one vCPU of an AWS c5.large instance. To scale to more clients you need to use more than one vCPU. The new mediasoup v3 APIs make this pretty easy. There are now "pipe transports" and "plain rtp transports" for moving the low-level rtp bits around between processes and between machines.[1]
[0]-https://github.com/michaelfig/mediasoup-broadcast-example [1]-https://mediasoup.org/documentation/v3/differences-between-v...
500ms is low latency? What's usual?
The best Vimeo/Livestream can do is about 30 seconds from the time the frame hits their server, as a reference.
As far as I'm concerned, 500ms is normal latency (once setting aside network delay -- clearly a 200ms one way trip from Europe to NZ will add to that)
I'd say ultra low latency would be sub-frame, so <40ms. Low latency on the 1 frame to half a second, normal upto say 1.5s, and high latency beyond that.
But that's for point to point or multicast circuits, and ISPs don't like multicast as they can't charge for it.
I'd say ultra low latency would be sub-frame, so <40ms. Low latency on the 1 frame to half a second, normal upto say 1.5s, and high latency beyond that.
But that's for point to point or multicast circuits, and ISPs don't like multicast as they can't charge for it.
[deleted]
Twitch random 1080@60 6K viewers stream
>Latency To Broadcaster: 7.71 sec.
>Latency Mode: Low Latency
>Latency To Broadcaster: 7.71 sec.
>Latency Mode: Low Latency
Multiple seconds.
There's also possible latency savings from having the video encoder be aware of network conditions on a per-frame basis: http://ex.camera but I don't think it's found its way to the "mainstream".
There's also this which tried to do something similar: https://snr.stanford.edu/salsify/
Edit: It's the same author
Edit: It's the same author
And here I was blissfully unaware that I'm experiencing reality 30 seconds delayed. Had no idea, other than obviously there's some # frames delay needed to do motion compression, and some transit latency.
[deleted]
I've used Wowza (video streaming server) for years and this is a direct competitor. The pricing is a little higher for Wowza, but Wowza is a mature product with tons of options for web streaming. The weakness of Wowza has been its support for WebRTC. I am using it, but it's not easy to stream from RTSP/RTMP to WebRTC. It also doesn't scale out for WebRTC
Test AMS Live Demo in https://antmedia.io/livedemo/
AMS Github Pages: https://github.com/ant-media/Ant-Media-Server
Thanks.
It will help apps which require interaction from users, like google stadia's game streaming. for other use cases, I think one does not care even if the feed is delayed by few seconds.
Another interesting use case someone mentioned here before is the world cup - you don't want a penalty shoot-out ruined by your neighbours cheering a few seconds before you see the goal.
A few seconds would also such for live streaming where the streamer interacts in chat.
A few seconds would also such for live streaming where the streamer interacts in chat.
How does one-to-many streaming work? If 256 clients connect to one streamer, how does the streamer not need to send his/her stream 256 times?
[deleted]
That's what the server is for.
The problem is that webrtc is a very intensive CPU ( each connection have its own encription ).
How many users can this solution handle per server?
It looks like they support RTMP and a few other streaming methods. Maybe it's just WebRTC for ingress and then it converts to other formats. Given the advertised 500ms of latency I imagine it's not WebRTC->WebRTC.
How is encryption expense different from normal https streaming from a cdn?
packet size of webrtc(rtp) is VERY VERY VERY small, and since they are UDP, each individual packet must be encrypted. ( i can't just encode a big chunk of data and send to the network like tls ).
in fact, for scaling a solution that uses "near realtime broadcast", you probably don't want encryption at all. ( afaik you can't disable webrtc encryption. )
in my tests i can get 3x more performance just using rtmp/rtsp. ( ~8000 video streaming connections on a aws c5.xlarge at 800kbit/s ).
in fact, for scaling a solution that uses "near realtime broadcast", you probably don't want encryption at all. ( afaik you can't disable webrtc encryption. )
in my tests i can get 3x more performance just using rtmp/rtsp. ( ~8000 video streaming connections on a aws c5.xlarge at 800kbit/s ).
most https connections are ephemeral and limited to connections per domain.
WebRTC can spawn large number of p2p connections.
WebRTC can spawn large number of p2p connections.
I've heard enough of this anti-anonymity, anti-privacy protocol.
What makes this approach lower latency than, say, a centralized TURN server in one of those clouds?
From my understanding, the current challenge with ultra-low-latency is scale. Will it scale to 100, 1000, 100,000, concurrent users?
At the moment, everyone in the industry is working to figure out how to provide a viewer experience that is similar, or better than the current expectations of live streaming.
Live streaming is hard. Ultra-low-latency is just as hard.
However, I'm still not sure why we need ultra-low-latency. Will it improve the live experience? Maybe. Aside from live-action sports or significant events, do we as content consumers really need ultra-low-latency?
At the moment, everyone in the industry is working to figure out how to provide a viewer experience that is similar, or better than the current expectations of live streaming.
Live streaming is hard. Ultra-low-latency is just as hard.
However, I'm still not sure why we need ultra-low-latency. Will it improve the live experience? Maybe. Aside from live-action sports or significant events, do we as content consumers really need ultra-low-latency?
Ultra low latency absolutely improves the streaming experience for community streams like on Twitch. When everyone is +-20 seconds of each other the chat doesn't line up well.
Implying there's value to Twitch chat?
Yeah. Like millions if not billions of dollars.
As a former Twitch employee, I lol’d at this.
By that, do you mean there's billions of $$ in Twitch chat, or is the concept too absurd (and hilarious as a result)?
How so?
Feelsbadman
Twitch chat is fun.
Twitch chat is fun.
> However, I'm still not sure why we need ultra-low-latency.
Simply for passive viewing probably not. For live streaming in which the viewer is also using actuators, e.g., gamepads, wheel, etc., for controlling the stream capture device, then yes.
Simply for passive viewing probably not. For live streaming in which the viewer is also using actuators, e.g., gamepads, wheel, etc., for controlling the stream capture device, then yes.
You've just done a stellar job of making "gaming" sound like a tedious day job. I'm impressed.
I did coding livestreams and spoke with the audience. The lower latency the better. Others have mentioned control of things, similar idea. HLS over Cloudflare surprisingly beat YouTube and matched Twitch in my side-by-side tests—I built an Open Source self-hosted live streaming system. Currently live sports have about 40-85 seconds of latency and BBC is shooting for 1.5 for the next olympics with ip multicast techniques [1].
1. http://www.wired.co.uk/article/england-vs-croatia-live-strea...
1. http://www.wired.co.uk/article/england-vs-croatia-live-strea...
How low is ultra-low-latency that it's relevant for sports? Does anyone really care if their broadcast is 30 seconds delayed from live?
I subscribe to both Formula 1 and MotoGP online streaming services and couldn't care less if the broadcasts were even multiple minutes delayed. They may very well be already and I wouldn't know. As far as I can tell they both just use standard CDNs delivering DASH or similar (basically just short video files played in sequence).
That seems more than enough for sports. Multiple minutes may result in getting real time information from elsewhere (e.g., Twitter, local radio) but anywhere from 10 to 30 seconds seems fine.
I subscribe to both Formula 1 and MotoGP online streaming services and couldn't care less if the broadcasts were even multiple minutes delayed. They may very well be already and I wouldn't know. As far as I can tell they both just use standard CDNs delivering DASH or similar (basically just short video files played in sequence).
That seems more than enough for sports. Multiple minutes may result in getting real time information from elsewhere (e.g., Twitter, local radio) but anywhere from 10 to 30 seconds seems fine.
I work for a video streaming company (primarily sports). There is a huge problem with latency in sports.
We call it the "Twitter effect". Essentially people either see things on social media or app notifications before it happens. Kinda ruins it for some people.
We are working on sub-30 second latency in 4K w/ server side ad insertion, but it's hard work. Especially with encoding with DRM baked in.
We call it the "Twitter effect". Essentially people either see things on social media or app notifications before it happens. Kinda ruins it for some people.
We are working on sub-30 second latency in 4K w/ server side ad insertion, but it's hard work. Especially with encoding with DRM baked in.
I had a normal latency stream (about 1.5 seconds) of the world cup on one England game. I saw the goal going in about 5-10 seconds before the cheers from the neighbours watching on DTT.
For the FA Cup final I had two feeds, one the 4K BBC version, and one on SDI from Wembley. The goal went in, and I'd completely forgotten it by the time the 4K goal went in.
The trouble is, things like twitter, whatsapp etc, will work to the lowest latency, and for a football game that's about 50-600 nanoseconds to the earliest viewer. Twitter can push the message within a few seconds (or the VAR result or whatever)
Latency needs to be in the sub-10 second goal-glass range.
For the FA Cup final I had two feeds, one the 4K BBC version, and one on SDI from Wembley. The goal went in, and I'd completely forgotten it by the time the 4K goal went in.
The trouble is, things like twitter, whatsapp etc, will work to the lowest latency, and for a football game that's about 50-600 nanoseconds to the earliest viewer. Twitter can push the message within a few seconds (or the VAR result or whatever)
Latency needs to be in the sub-10 second goal-glass range.
Surely, in that case what would be important is relative-latency, not absolute.
If you delay all your friends by 30s +-2s then that would probably be fine...
That changes the problem somewhat.
If you delay all your friends by 30s +-2s then that would probably be fine...
That changes the problem somewhat.
good luck having a conversation with them with that type of latency
At the top of this thread I was wondering "how many miliseconds is low latency", I hadn't realised it was usually worse than 30 seconds!
Fubo? Liked the service during trial but the 4K World Cup stream was flaky.
Thanks! Do you know what the equivalent latency number is for cable TV? The Twitter effect only applies for the difference between the two numbers.
Broadcast TV latency is usually in the 4-10 second range, with typical latency being around 6s. Not sure how it breaks down between cable vs satellite vs over-the-air.
I'm actually not sure. I'm sure it's different for every station and mode of delivery. Our latency is measured directly from the broadcaster signal.
If your friends are watching the same events in other places; you basically need your video to be less delayed than your friends' viewing plus the latency of them texting you (friend observes something, types something, sends, then whatever network and push delay on the way to you).
If you're not watching with someone else, it doesn't make that much difference; but 30 seconds is plenty for someone to cheer for a goal and the notification to distract you from seeing it.
If you're not watching with someone else, it doesn't make that much difference; but 30 seconds is plenty for someone to cheer for a goal and the notification to distract you from seeing it.
That's already an issue with radio vs TV which is where I'm getting the ~10 to 30 second value. Normal cable TV already has a bunch of delay and someone listening on the radio is often 15+ seconds ahead of you. This can be disruptive. If that person is at a bar or other gathering they will disrupt the viewing because they will cheer (or not cheer) well ahead of the event so the suspense is gone. I've seen this happen.
So given that what you are competing against (cable TV) already has a significant delay I doubt a normal web stream can't reach the same or better latency with just standard HTTP download of video files.
So given that what you are competing against (cable TV) already has a significant delay I doubt a normal web stream can't reach the same or better latency with just standard HTTP download of video files.
Yes. 30 seconds is too much.
In Santiago, during national soccer matches everyone would watch, goals would reach us through the sound of neighbors rejoicing sooner than they would from actual television coverage because the satellite dish had a latency of say 3 seconds over cable.
In Santiago, during national soccer matches everyone would watch, goals would reach us through the sound of neighbors rejoicing sooner than they would from actual television coverage because the satellite dish had a latency of say 3 seconds over cable.
You're giving an example that validates my point. Existing technologies already have widely variable delays (radio vs broadcast TV vs cable TV vs sattelite TV) and yet they're all in wide use. And you can't compare directly the 30s I suggested (end-to-end latency) with the 3 seconds of latency differential between technologies. It's probably less than that but it's possible your neighbors are in a 30s delay and you are in a 33s delay.
And if the problem was actually that serious it would be easier to fix it by just adding additional delay to the faster technologies instead of having to reduce latency in internet streaming at huge cost.
And if the problem was actually that serious it would be easier to fix it by just adding additional delay to the faster technologies instead of having to reduce latency in internet streaming at huge cost.
Then you would have the people at stadium tweeting about the results and ruining the experience for the rest. We can and we should strive to get the best possible result if it is feasible.
[deleted]
Just experienced this delay being an issue for the 3rd period of this vegas-san jose playoff game. When goals pop off as fast as they do and it completely changes the game, that delay matters. Especially when my friends are either at a bar or streaming the game, it's hard to communicate and share the moment without spoiling. I basically have to keep a mental tab on who's stream is delayed and when I can get excited with who.
Ultra-low-latency is absolutely vital in sports betting industry where the odds are constantly changing based on what actions are taking place in real time.
Results in sports can switch from one side to the other end within split second, so whoever has the lowest latency will gain massive advantages. It's very similar to stock trading.
Results in sports can switch from one side to the other end within split second, so whoever has the lowest latency will gain massive advantages. It's very similar to stock trading.
Yes, absolutely. Huge deal for live sports.
It’s the entire reason that I won’t cut the cord.
It’s the entire reason that I won’t cut the cord.
The chord you're not cutting already has a large delay from live. If you're in the US even larger because besides the delay from the technology there's also intentional broadcast delay for censorship reasons.
Google Stadia(and other online game streaming) and other systems where actuators would be used remotely like drone control systems and others would greatly benefit from ultra low latency.
[deleted]
industries serving the sports industry / creating products related to sports will be very interested... not your average consumer though...
I'm interested in low latency WebRTC datachannel. Does this apply to that as well?
[deleted]
hqtrivia
I think even with turn there's lots of chatter back and forth. Connected clients can NACK for frames and ask for a bandwidth reduction. I think the expected optimal implementation is that the group of WebRTC participants normalize towards an optimal bitrate.
For 1->n streaming you just want very high quality broadcast->server and then presumably you want the downstream clients to not have the option to send packets back to the broadcaster. Transcoding is hard though, so consumers might be stuck with just the source video quality. I assume this lib just passes the frames on from the source (twitch did just that through almost all of its pre-acquisition growth).
Some of my details might be a little off, but I did spend a few months trying to build something very similar to this with gstreamer. WebRTC was hard to grok.
For 1->n streaming you just want very high quality broadcast->server and then presumably you want the downstream clients to not have the option to send packets back to the broadcaster. Transcoding is hard though, so consumers might be stuck with just the source video quality. I assume this lib just passes the frames on from the source (twitch did just that through almost all of its pre-acquisition growth).
Some of my details might be a little off, but I did spend a few months trying to build something very similar to this with gstreamer. WebRTC was hard to grok.
Actually it is not lower latency than centralized TURN server. The difference from TURN is that Ant Media Server behaves like peer to each connected clients. Not relaying the video/audio as TURN. Additionally, it can create adaptive bitrates on the server side. It can record streams and control other things on server side.
In answer to the questions about TURN, this approach won't be lower latency than TURN but will scale better. TURN servers are just (nearly) passive relays, so the sending client needs to set up as many outbound streams as there are receiving clients in the session.
The advantage to the TURN approach is that you can do end-to-end encryption.
The server this post is about "forwards" streams. It's in a class of infrastructure traditionally called a "Selective Forwarding Unit." So the sending client can send one stream, and the SFU copies the stream and sends it to the receiving clients. The server needs more CPU and outgoing bandwidth as the receiver count grows, but the sending client doesn't need much of either.
Once you have streams piping through a server, you can do lots of other things, of course. This server can transcode to several formats, including HLS and RTMP, and can copy streams internally across a cluster to scale more than a single machine can.
Media servers are really fun to work on: lots of small(-ish) hard problems that touch low-level network protocols, memory and cpu optimization, architecture (because eventually you want a clean plugins interface, etc.).
If you're interested in this stuff, check out Mediasoup, an open source WebRTC SFU with a very nice design in which all the low-level stuff is c++ and all the high-level interfaces are exposed as nodejs objects.[0]
[0]-https://mediasoup.org/