A P2P Vision for QUIC (2024)(seemann.io)
seemann.io
A P2P Vision for QUIC (2024)
https://seemann.io/posts/2024-10-26---p2p-quic/
48 comments
UDP-based protocols are well suited for P2P, since hole punching is straightforward if you have predictable port mapping, you cannot disallow it. In that spirit, we are currently exploring this with:
https://github.com/tbocek/qotp and https://github.com/qh-project/qh
The main idea is to have a simple encryption (ed25519/chacha20+poly1305) for encryption in the transport layer, on top of that then qh, where certs are use for signing content.
With out of band key exchange, you can establish a connection after you successfully punched a hole.
However, its not QUIC compatible in any way (https://xkcd.com/927)
https://github.com/tbocek/qotp and https://github.com/qh-project/qh
The main idea is to have a simple encryption (ed25519/chacha20+poly1305) for encryption in the transport layer, on top of that then qh, where certs are use for signing content.
With out of band key exchange, you can establish a connection after you successfully punched a hole.
However, its not QUIC compatible in any way (https://xkcd.com/927)
> UDP-based protocols are well suited for P2P, since hole punching is straightforward if you have predictable port mapping, you cannot disallow it.
"Cannot" is a strong word:
> UDP hole punching will not work with symmetric NAT devices (also known as bi-directional NAT) which tend to be found in large corporate networks. In symmetric NAT, the NAT's mapping associated with the connection to the known STUN server is restricted to receiving data from the known server, and therefore the NAT mapping the known server sees is not useful information to the endpoint.
* https://en.wikipedia.org/wiki/UDP_hole_punching#Overview
I've also heard lots of people complain about how they're stuck behind CG-NAT and various P2P things do not work.
"Cannot" is a strong word:
> UDP hole punching will not work with symmetric NAT devices (also known as bi-directional NAT) which tend to be found in large corporate networks. In symmetric NAT, the NAT's mapping associated with the connection to the known STUN server is restricted to receiving data from the known server, and therefore the NAT mapping the known server sees is not useful information to the endpoint.
* https://en.wikipedia.org/wiki/UDP_hole_punching#Overview
I've also heard lots of people complain about how they're stuck behind CG-NAT and various P2P things do not work.
> and https://github.com/qh-project/qh
This link is 404.
QOTP looks really cool. Like what QUIC would be if DJB were in charge of it.
This link is 404.
QOTP looks really cool. Like what QUIC would be if DJB were in charge of it.
Thx, I cannot change my previous message, the QH is set to private, as me and my students are working on it. It will be made public next month. The qh:// protocol, we have just started, and we only focused on simplifying the header as an alternative to HPACK / QPACK. Currently we are collecting headers for identifying what should go into our static table for lookup, and for benchmarking. For simplicity, we decided against dynamic tables or huffman-style encoding/decoding. Just a table with often used header/values, the rest is binary encoded with no further compression.
the https://github.com/qh-project/qh link doesn't work for what its worth.
> UDP-based protocols are well suited for P2P, since hole punching is straightforward if you have predictable port mapping, you cannot disallow it.
... how does that work when the network disallows UDP altogether?
... how does that work when the network disallows UDP altogether?
That is not as common as you think, and will get less common as QUIC grows in popularity.
If you're really really desperate you can send UDP packets with fake TCP headers (i.e. you aren't actually doing any congestion control or retransmission) but you have to control both ends of the connection for that.
And there's ICMP.
If you're really really desperate you can send UDP packets with fake TCP headers (i.e. you aren't actually doing any congestion control or retransmission) but you have to control both ends of the connection for that.
And there's ICMP.
Blocking UDP to Outside and non-ping udp to Outside is common. It will remain common as long as servers fall back. And exceptions will be made for problem domains.
Isn’t a concept of TURN server from RFC 5766 a solution for this problem?
TURN means it's no longer P2P:
> Traversal Using Relays around NAT (TURN): Relay Extensions to Session Traversal Utilities for NAT (STUN)
> Abstract
> If a host is located behind a NAT, then in certain situations it can be impossible for that host to communicate directly with other hosts (peers). In these situations, it is necessary for the host to use the services of an intermediate node that acts as a communication relay. This specification defines a protocol, called TURN (Traversal Using Relays around NAT), that allows the host to control the operation of the relay and to exchange packets with its peers using the relay.
> Traversal Using Relays around NAT (TURN): Relay Extensions to Session Traversal Utilities for NAT (STUN)
> Abstract
> If a host is located behind a NAT, then in certain situations it can be impossible for that host to communicate directly with other hosts (peers). In these situations, it is necessary for the host to use the services of an intermediate node that acts as a communication relay. This specification defines a protocol, called TURN (Traversal Using Relays around NAT), that allows the host to control the operation of the relay and to exchange packets with its peers using the relay.
It really depends on the implementation? From my understanding, WebRTC uses it to broker the connection and nothing more. Then it's p2p
The "The Traditional Way" section of the article covers the shortcomings of the WebRTC approach (i.e. ICE) and why it does not always result in a P2P session forming/the TURN server leaving the picture.
The tldr is having a TURN server enables more hole-punching methods to be attempted, but that's not the same thing as a guarantee the session will succeed in switching to P2P.
The tldr is having a TURN server enables more hole-punching methods to be attempted, but that's not the same thing as a guarantee the session will succeed in switching to P2P.
You can't disallow hole punching.
Why not? Doesn't it depend on the type of NAT used?
As I understand it, most consumer devices will set up a port mapping which is completely independent of the destination's IP and port. It's just "incoming packet for $wanip:567 goes to $internal:123, outgoing packet from $internal:123 get rewritten to appear from $wanip:567". This allows any packet towards $wanip:567 to reach the internal host - both the original server the client initiated the connection to, and any other random host on the internet. Do this on two clients, have the server tell them each other's mappings, and they can do P2P comms: basic hole punching. I believe this is usually called "Full Cone NAT".
However, nothing is stopping you from setting up destination-dependent mapping, where it becomes "incoming packet from $server:443 to $wanip:456 goes to $internal:123, outgoing packet from $internal:123 to $server:443 gets rewritten to appear from $wanip:567". This would still work totally fine for regular client-to-server communication, but that mapping would only work for that specific server. A packet heading towards $wanip:456 would get dropped because the source isn't $server:443 - or it could even get forwarded to another host on the NATed network. This would block traditional hole punching. I believe this is called "Address Restricted Cone NAT" if it filters only on source IP, or "Port Restricted Cone NAT" if it filters on both source IP and source port.
As I understand it, most consumer devices will set up a port mapping which is completely independent of the destination's IP and port. It's just "incoming packet for $wanip:567 goes to $internal:123, outgoing packet from $internal:123 get rewritten to appear from $wanip:567". This allows any packet towards $wanip:567 to reach the internal host - both the original server the client initiated the connection to, and any other random host on the internet. Do this on two clients, have the server tell them each other's mappings, and they can do P2P comms: basic hole punching. I believe this is usually called "Full Cone NAT".
However, nothing is stopping you from setting up destination-dependent mapping, where it becomes "incoming packet from $server:443 to $wanip:456 goes to $internal:123, outgoing packet from $internal:123 to $server:443 gets rewritten to appear from $wanip:567". This would still work totally fine for regular client-to-server communication, but that mapping would only work for that specific server. A packet heading towards $wanip:456 would get dropped because the source isn't $server:443 - or it could even get forwarded to another host on the NATed network. This would block traditional hole punching. I believe this is called "Address Restricted Cone NAT" if it filters only on source IP, or "Port Restricted Cone NAT" if it filters on both source IP and source port.
If your NAT allows arbitrary connections out, and you're patient enough, you can probably finagle a peer to peer connection, eventually. Here's a discussion about that [1]. But that math is based on each NAT having a single external address; if your NAT spreads you over multiple addresses, the math gets much worse.
And there's a lot of other considerations; chances are your NAT won't be happy if you send all those probe packets at once, and your user may not be either. It's probably only worth it to do exhaustive probing if the connection is long lived, and proxying is expensive (in dollars because of bandwidth or in latency)
[1] https://github.com/danderson/nat-birthday-paradox/tree/maste...
And there's a lot of other considerations; chances are your NAT won't be happy if you send all those probe packets at once, and your user may not be either. It's probably only worth it to do exhaustive probing if the connection is long lived, and proxying is expensive (in dollars because of bandwidth or in latency)
[1] https://github.com/danderson/nat-birthday-paradox/tree/maste...
The feasibility of this assumes one peer is always behind an endpoint dependent mapping. That's great if you only care about peers working with you and you control your style of NAT, but it's still pretty broken for the case you want this to work for any 2 peers. In practical terms, the success rate goes from something like the 64% with 256 probes down to something less than 0.01%.
If you can manage to bump it up to 65536 probes without getting blocked, hitting a NAT limit, or causing the user to fall asleep waiting, then it should hit the same success rate :D. I'm not sure many would like to use that P2P service though, at that point just pay for the TURN server.
If you can manage to bump it up to 65536 probes without getting blocked, hitting a NAT limit, or causing the user to fall asleep waiting, then it should hit the same success rate :D. I'm not sure many would like to use that P2P service though, at that point just pay for the TURN server.
64k probes is a lot, but it might be reasonable if you're trying to get something like wireguard connected between locations that are behind CGNAT, send 10 probes a second for a couple hours and then remain connected for a long time. Of course, CGNAT might split your traffic over multiple IPs and then the math is terrible.
If you need to send 64k probes to get p2p and you want to make a 15 minute call, it probably doesn't make sense, but it's probably worth trying a bit in case you catch an easy case. Not that p2p is always better than going through a relay, but it's often less expensive.
If you need to send 64k probes to get p2p and you want to make a 15 minute call, it probably doesn't make sense, but it's probably worth trying a bit in case you catch an easy case. Not that p2p is always better than going through a relay, but it's often less expensive.
The math doesn't quite work that conveniently in that at least one side needs to actually initiate (and keepalive) 65k sessions through their NAT while the other tests 10 of those ports at a time. If you just do 10 at a time both sides until you've done 65k total you end up with even worse odds than having just done 256 at once, due to the Birthday Paradox nature of the problem.
For wireguard that might be fine because you likely control the head end and opening ~65k NAT sessions is something you can opt to do if you tune things accordingly. Of course, in that case, you can also just opt to use the more lenient form of NAT at your head end and just use attempt with 256 ports instead.
For wireguard that might be fine because you likely control the head end and opening ~65k NAT sessions is something you can opt to do if you tune things accordingly. Of course, in that case, you can also just opt to use the more lenient form of NAT at your head end and just use attempt with 256 ports instead.
Fair enough, I didn't go through the math. I don't think many NATs are realistically likely to let a single client run 64k sessions.
ISPs are increasingly putting customers behind CGNAT, so wireguard at home doesn't imply control over NAT policies. Especially new entrants and fixed wireless ISPs don't tend to have the resources to get an IP (v4) for every customer, and some of them don't offer v6 either, so having some form of hope would be nice.
ISPs are increasingly putting customers behind CGNAT, so wireguard at home doesn't imply control over NAT policies. Especially new entrants and fixed wireless ISPs don't tend to have the resources to get an IP (v4) for every customer, and some of them don't offer v6 either, so having some form of hope would be nice.
> You can't disallow hole punching.
Try doing it over a network that only allows connections through a SOCKS/Squid proxy, or on a network that uses CG-NAT (i.e., double-NAT).
See also:
> UDP hole punching will not work with symmetric NAT devices (also known as bi-directional NAT) which tend to be found in large corporate networks. In symmetric NAT, the NAT's mapping associated with the connection to the known STUN server is restricted to receiving data from the known server, and therefore the NAT mapping the known server sees is not useful information to the endpoint.
* https://en.wikipedia.org/wiki/UDP_hole_punching#Overview
Try doing it over a network that only allows connections through a SOCKS/Squid proxy, or on a network that uses CG-NAT (i.e., double-NAT).
See also:
> UDP hole punching will not work with symmetric NAT devices (also known as bi-directional NAT) which tend to be found in large corporate networks. In symmetric NAT, the NAT's mapping associated with the connection to the known STUN server is restricted to receiving data from the known server, and therefore the NAT mapping the known server sees is not useful information to the endpoint.
* https://en.wikipedia.org/wiki/UDP_hole_punching#Overview
TCP Simultaneous Open. If two clients happen to connect to each other's ephemeral ports at exactly the same moment, they connect to each other with no server involved. It should work the same as UDP hole punching but with a much smaller time window.
From TFA:
"Unfortunately, no matter how hard you try, there is a certain percentage of nodes for whom hole punching will never work. This is because their NAT behaves in an unpredictable way. While most NATs are well-behaved, some aren’t. This is one of the sad facts of life that network engineers have to deal with."
In this scenario, the article goes on to describe a convention relay-based approach.
I would guess that most consumer routers are very cooperative as far as hole punching because it's pretty critical functionality for bittorrent and many online games. Corporate firewalls wouldn't be as motivated to care about those use-cases or may want to actively block them.
"Unfortunately, no matter how hard you try, there is a certain percentage of nodes for whom hole punching will never work. This is because their NAT behaves in an unpredictable way. While most NATs are well-behaved, some aren’t. This is one of the sad facts of life that network engineers have to deal with."
In this scenario, the article goes on to describe a convention relay-based approach.
I would guess that most consumer routers are very cooperative as far as hole punching because it's pretty critical functionality for bittorrent and many online games. Corporate firewalls wouldn't be as motivated to care about those use-cases or may want to actively block them.
> Corporate firewalls wouldn't be as motivated
I think parents point is a bit like "you can't disallow lock picking"; the term "hole punching" being used to describe techniques that are intentionally trying to bypass whatever thing others (particularly corporations) try to put in the way, sometimes for good reasons and sometimes for kind of shit reasons.
I think parents point is a bit like "you can't disallow lock picking"; the term "hole punching" being used to describe techniques that are intentionally trying to bypass whatever thing others (particularly corporations) try to put in the way, sometimes for good reasons and sometimes for kind of shit reasons.
I hope some day the browser's webtransport also gets p2p support.
It seemed like there was such a good exciting start, but the spec has been dormant for years. https://github.com/w3c/p2p-webtransport
It seemed like there was such a good exciting start, but the spec has been dormant for years. https://github.com/w3c/p2p-webtransport
It is halfway there arguably, and libp2p does make use of it - https://docs.libp2p.io/concepts/transports/webtransport/
Unlike websockets you can supply "cert hash" which makes it possible for the browser to establish a TLS connection with a client that doesn't have a certificate signed by a traditional PKI provider or even have a domain name. This property is immensely useful because it makes it possible for browsers to establish connections to any known non-browser node on the internet, including from secure contexts (i.e. from an https page where e.g. you can't establish a ws:// connection, only wss:// is allowed but you need a 'real' tls cert for that)
Unlike websockets you can supply "cert hash" which makes it possible for the browser to establish a TLS connection with a client that doesn't have a certificate signed by a traditional PKI provider or even have a domain name. This property is immensely useful because it makes it possible for browsers to establish connections to any known non-browser node on the internet, including from secure contexts (i.e. from an https page where e.g. you can't establish a ws:// connection, only wss:// is allowed but you need a 'real' tls cert for that)
My understanding is that wide adoption of WebTransport is currently blocked by WebKit:
https://caniuse.com/webtransport
However, there have been some recent pull requests indicating gradual progress:
https://github.com/WebKit/WebKit/pulls?q=is%3Apr+is%3Aclosed...
https://caniuse.com/webtransport
However, there have been some recent pull requests indicating gradual progress:
https://github.com/WebKit/WebKit/pulls?q=is%3Apr+is%3Aclosed...
Someone correct me if I'm wrong, but I think p2p-webtransport was superseded by "webtransport" (https://github.com/w3c/webtransport). Supposedly, the webtransport design should be flexible enough to support p2p even though focus is the traditional server<>client.
The story here is a bit complicated. WebTransport is, in some sense, an evolution of RTCQuicTransport API, which was originally meant to solve the issues people had with SCTP/DTLS stack used by RTCDataChannel. At some point, the focus switched to client-server use cases, with an agreement that we can come back to the P2P scenario after we solve the client-server one.
Superceded? No. Webtransport already was well on its way to approval when p2p-webtransport was created.
Webtransport as a protocol certainly could be used for p2p, but the browser APIs aren't there: hence p2p-webtransport was created, to allow its use beyond traditional server<->client.
Webtransport as a protocol certainly could be used for p2p, but the browser APIs aren't there: hence p2p-webtransport was created, to allow its use beyond traditional server<->client.
Any UDP protocol can be made P2P if it can be bidirectionally authenticated.
For TCP based protocols it's very hard since there is no reliable way to hole punch NATs and stateful firewalls with TCP.
For TCP based protocols it's very hard since there is no reliable way to hole punch NATs and stateful firewalls with TCP.
Maybe success rates are higher with UDP – I don’t know. But it certainly works to hole punch with TCP as well. If you’re lucky you can even run into a rare condition called ”TCP simultaneous open”, where both sides believe they are the dialer.
> where both sides believe they are the dialer.
First time I've heard about this, and went looking for more. Came across https://news.ycombinator.com/item?id=5969030 (95 points - July 1, 2013 - 49 comments) that had bunch of background info + useful discussions.
First time I've heard about this, and went looking for more. Came across https://news.ycombinator.com/item?id=5969030 (95 points - July 1, 2013 - 49 comments) that had bunch of background info + useful discussions.
It can be done, but it's less reliable and also requires the ability to forge packets that is not allowed on all platforms. So it's hard to use in any production application if you want it to run in user space, on Windows, or on mobile.
No I don’t think it does. But I am using it in user space[1] successfully with no packet forging. I believe the outbound attempt (SYN) creates a mapping, and even if that conn is blocked on the other end, the inbound conn (SYN but in the other direction) is allowed.
Again, maybe packet forging is needed for some routers/middleboxes/firewalls, since careful inspection would show that the conns are technically independent. If you have any details about this, please let me know! (Networking is difficult to test.)
1: https://payload.app/
Again, maybe packet forging is needed for some routers/middleboxes/firewalls, since careful inspection would show that the conns are technically independent. If you have any details about this, please let me know! (Networking is difficult to test.)
1: https://payload.app/
Wait? How does that work? QUIC REQUIRES CA TLS for all endpoints. So you can do the discovery/router workarounds but then the person trying to connect to you with QUIC won't be able to unless you have a signed corporate CA TLS cert. I guess you could integrate some Lets Encrypt ACME2 periodic updater scheme into your P2P program but that's getting pretty complex and fragile. And it also provides a centralized way for anyone who doesn't like your P2P tool to legally/socially pressure it to shut it down.
> QUIC REQUIRES CA TLS for all endpoints
No. QUIC require TLS. TLS just provides a way to move certificates, but doesn't care what a "certificate" actually is. JPEG of your 10m swimming certificate from school? Sure, that's fine.
The endpoints get to decide which certificates to accept and in practice in a web browser and many other modern programs that'll be some sort of X.509 certificate more or less following PKIX and on the public Internet usually the Web PKI which is a PKI operated on behalf of the Relying Parties (literally everybody) by the Trust Stores (in practice the OS vendors plus Mozilla for the Free Unix systems) but none of that is defined by QUIC.
No. QUIC require TLS. TLS just provides a way to move certificates, but doesn't care what a "certificate" actually is. JPEG of your 10m swimming certificate from school? Sure, that's fine.
The endpoints get to decide which certificates to accept and in practice in a web browser and many other modern programs that'll be some sort of X.509 certificate more or less following PKIX and on the public Internet usually the Web PKI which is a PKI operated on behalf of the Relying Parties (literally everybody) by the Trust Stores (in practice the OS vendors plus Mozilla for the Free Unix systems) but none of that is defined by QUIC.
A good time to mention that the P2P Yggdrasil network uses QUIC/TLS selfsigned certs but then runs its own encryption over that. You can add as many peers as desired, and the network will automatically choose the best path (latency). So no multi-pathing but gets around the issue of changing IP addresses/network locations. Plus, it's able to do multicast to find peers in your LAN without having a centralized control server. I'm actually getting better speeds than WireGuard over my LAN - but this is a stable link. Once you start sending the yggdrasil packets over long unstable links you may start to get into funky issues like TCP in TCP head of line blocking, but they try to mitigate this by having huge MTU sizes and packet dropping algorithms. (https://yggdrasil-network.github.io/2018/08/19/congestion-co...)
https://yggdrasil-network.github.io/documentation.html
I'm currently working on creating a managed Yggdrasil relay node service. A feature I hope they implement is QUIC multistream support.
https://yggdrasil-network.github.io/documentation.html
I'm currently working on creating a managed Yggdrasil relay node service. A feature I hope they implement is QUIC multistream support.
Ok so you need to trust each other's certs. What's the big deal? Presumably you already have some other channel to share addresses so you can also share temporary self signed certs for this purpose.
I'm working with QUIC in a personal project, while you can roll your own QUIC library the spec is large enough that it's quite a bit of work to implement it yourself. Most libraries allow you to pass in your own certificates. Realistically you could just bake in certs to your program and call it a day. Otherwise yes, you can implement your own cert logic that completely ignores certs altogether. s2n-quic for example specifically allows for both, though the former is much easier to do.
What prevents you from just using certificates not signed by a CA and verifying them based on the public key fingerprint?
I guess most if not all QUIC endpoints you come across the internet will have encryption, as the specification requires as such. But if you control both ends, say you're building a P2P application that happens to use QUIC, I don't think there is anything stopping you from using an implementation of QUIC that doesn't require that, or use something else than TLS, even if the specification would require you to have it.
Just as long as you statically build and ship your application. Because I guarantee the QUIC libs in $distro are not going to be compiled with the experimental flags to make this possible. You're going to be fighting QUIC all the way to get this to work. It's the wrong choice for the job. Google did not design QUIC for human use cases and the protocol design reflects this.
Judging (guessing) by the author's GitHub profile (https://github.com/marten-seemann), seems they've built their own "pure Go" QUIC implementation, maybe precisely for those purposes :)
The existing WebTransport API implemented in all browsers actually supports you providing the fingerprint of a certificate that can be self-signed.
RFC 7250: Using Raw Public Keys in Transport Layer Security (TLS).
If I recollect, sha256 public or private signatures that are used on a thin-client for P2P server architecture.
Carrier peering using the UDP hashes for encrypting network traffic from a WAN to serve a Tier 1 network.
Carrier peering using the UDP hashes for encrypting network traffic from a WAN to serve a Tier 1 network.
"Unfortunately, no matter how hard you try, there is a certain percentage of nodes for whom hole punching will never work."
It might be helpful to cite the percentage
It's relatively small
A default policy that relays traffic through a third party is asinine
For the small percentage, the third parties will always be there if they need them. The internet has an enormous supply of middlemen, like Google
For everyone else, the third parties, i.e. the middlemen, can be avoided
It might be helpful to cite the percentage
It's relatively small
A default policy that relays traffic through a third party is asinine
For the small percentage, the third parties will always be there if they need them. The internet has an enormous supply of middlemen, like Google
For everyone else, the third parties, i.e. the middlemen, can be avoided
Or they are centrally/corporate-controlled and do not allow hole punching.