Rack makes Pion SCTP 71% faster with 27% less latency(pion.ly)
pion.ly
Rack makes Pion SCTP 71% faster with 27% less latency
https://pion.ly/blog/sctp-and-rack/
4 comments
SCTP isn't just a UDP pipe, It's a message oriented, congestion-controlled, reliability protocol, with bunch of other semantics.
We measured:
1. Association state + Per PATH CC/RTO, timers, RTT tracking, cwnd etc.
2. Selective ACKs and re-transmit logic.
3. chunk framing + tsn sequences.
4. ordered vs unordered delivery, and fragmentation/reassembly.
much more ...
Also our vnet-based implementation isn't just dumb buffer, we have packet on wire validation, SCTP parsing, CRC32c validations. deterministic network conditions emulator. With real time conditions.
Sure you can get 100 GB/Core second if you bypass all of that and just do huge batching
The blog post claim is just under the same SCTP semantics and the same test harness, enabling RACK has a huge win. not the absolute ceilings of in-process "virtual network" sockets :)
We measured:
1. Association state + Per PATH CC/RTO, timers, RTT tracking, cwnd etc.
2. Selective ACKs and re-transmit logic.
3. chunk framing + tsn sequences.
4. ordered vs unordered delivery, and fragmentation/reassembly.
much more ...
Also our vnet-based implementation isn't just dumb buffer, we have packet on wire validation, SCTP parsing, CRC32c validations. deterministic network conditions emulator. With real time conditions.
Sure you can get 100 GB/Core second if you bypass all of that and just do huge batching
The blog post claim is just under the same SCTP semantics and the same test harness, enabling RACK has a huge win. not the absolute ceilings of in-process "virtual network" sockets :)
Yes, I meant all of that when I explicitly said feature parity at 100 Gb/core-second. Reliable delivery of multiple independent bytestreams (which is actually more than SCTP gives since SCTP still suffers from head-of-line blocking due to SCTP SACKs being by TSN instead of a per-stream identifier) with dynamic stream count (again, more than SCTP gives) over a unreliable network that may reorder or lose packets.
Okay, I see your point, but our test harness isn't meant to be an absolute "max throughput" benchmark. every packet is parsed, corrected (if needed), and validated in real time (CRC32c, on-the-wire checks, deterministic network emulation, etc.).
If we ever want a true ceiling number, we could add a separate fast path (e.g, a dump-writer / sink that skips most validation) or validate after run, but that's not in scope right now. our scope was: (1) validate Pion/SCTP PRs and (2) compare performance against other branches and version. so for relative benchmark under identical conditions.
on head-of-line blocking: we have a pending RFC 8260 message interleaving (I-DATA) implementation, and we've tested with it; it helps reduce HoL on the sender side (especially around fragmentation). our benchmark tool has a flag to run with interleaving, and we tested it quit a bit. We plan to release it in Jan.
If we ever want a true ceiling number, we could add a separate fast path (e.g, a dump-writer / sink that skips most validation) or validate after run, but that's not in scope right now. our scope was: (1) validate Pion/SCTP PRs and (2) compare performance against other branches and version. so for relative benchmark under identical conditions.
on head-of-line blocking: we have a pending RFC 8260 message interleaving (I-DATA) implementation, and we've tested with it; it helps reduce HoL on the sender side (especially around fragmentation). our benchmark tool has a flag to run with interleaving, and we tested it quit a bit. We plan to release it in Jan.
Under those same conditions, you can quite readily do ~100 Gb/core-second (ignoring encryption, encryption will bottleneck you to 30-50 Gb/core-second on modern chips with AES acceleration instructions) in software with feature parity with proper protocol design and implementation.