Multipath TCP for Linux (2022)(mptcp.dev)
mptcp.dev
Multipath TCP for Linux (2022)
https://www.mptcp.dev/
103 comments
I suspect that a lot of innovation energy moved to QUIC, because with TCP your nice new variant can be randomly nobbled by middleboxes. For example, see https://blog.apnic.net/2021/12/08/efficient-multipath-transp...
QUIC is a step backwards here; it has no multipath support: https://lwn.net/Articles/964377/
Multipath: There are several areas where TCP still has an advantage over QUIC. One of those is multipath support. Multipath TCP connections can send data on different network paths simultaneously — for example, sending via both WiFi and cellular data — to provide better throughput than either path permits individually.
Server connection migration is explicitly forbidden by QUIC:
https://github.com/quicwg/base-drafts/pull/2031
Multipath: There are several areas where TCP still has an advantage over QUIC. One of those is multipath support. Multipath TCP connections can send data on different network paths simultaneously — for example, sending via both WiFi and cellular data — to provide better throughput than either path permits individually.
Server connection migration is explicitly forbidden by QUIC:
https://github.com/quicwg/base-drafts/pull/2031
The draft multipath extension is here: https://datatracker.ietf.org/doc/draft-ietf-quic-multipath/
It's on the standards track, rather than experimental, so likely to be supported once finished. There seem to be some implementations, including Apple:
https://github.com/quicwg/multipath/wiki/QUIC-Implementation...
It's on the standards track, rather than experimental, so likely to be supported once finished. There seem to be some implementations, including Apple:
https://github.com/quicwg/multipath/wiki/QUIC-Implementation...
Nobody cares about those middleboxes, those are only relevant in corporate networks.
Such middleboxes can also be seen in cellular networks. (And firewall in free access points / guest networks)
>Such middleboxes can also be seen in cellular networks.
Complain to your ISP if they mingle with a layer they are not supposed to mingle with.
>(And firewall in free access points / guest networks
I consider those as "corporate".
Complain to your ISP if they mingle with a layer they are not supposed to mingle with.
>(And firewall in free access points / guest networks
I consider those as "corporate".
I wanted to like it, and Apple included it in iOS, but supporting it on real servers was going to be too hard...
When I was deployed on FreeBSD with no load balancers, there weren't recent patches. And even if there were, I'd need to do some serious work to avoid advertising the private network ips as alternates...
When I was on Linux behind a load balancer, it's too complex to get the streams to the right place. And the load balancer doesn't want to do it anyway.
Processing two streams together involves a lot of complexity in a high throughput code path. It's a lot of risk, and you've got to reboot for changes.
And then you do all that work and it only benefits iOS users, who tend to be on better networks anyway.
When I was deployed on FreeBSD with no load balancers, there weren't recent patches. And even if there were, I'd need to do some serious work to avoid advertising the private network ips as alternates...
When I was on Linux behind a load balancer, it's too complex to get the streams to the right place. And the load balancer doesn't want to do it anyway.
Processing two streams together involves a lot of complexity in a high throughput code path. It's a lot of risk, and you've got to reboot for changes.
And then you do all that work and it only benefits iOS users, who tend to be on better networks anyway.
Apple also contributed[1] MPTCP support to Envoy Proxy.
[1]https://github.com/envoyproxy/envoy/pull/18780
[1]https://github.com/envoyproxy/envoy/pull/18780
> iOS users, who tend to be on better networks anyway.
I don't think there is any basis to claim that.
I don't think there is any basis to claim that.
> A U.S. analysis of Wi-Fi and mobile Internet usage across unique smartphones on the iOS and Android platforms reveals that 71 percent of all unique iPhones used both mobile and Wi-Fi networks to connect to the Internet, while only 32 percent of unique Android mobile phones used both types of connections. A further analysis of this pattern of behavior in the U.K. shows consistent results, as 87 percent of unique iPhones used both mobile and Wi-Fi networks for web access compared to a lower 57 percent of Android phones.
https://www.comscore.com/lat/Prensa-y-Eventos/Infographics/i...
https://www.comscore.com/lat/Prensa-y-Eventos/Infographics/i...
Since wi-fi networks tend to be higher quality than cell networks, what you provided works against the point I responded to.
Someone paying for a premium phone is probably also inclined to pay for a premium mobile network.
Not all iPhones are 'premium' phones, and there are not really 'premium' cell phone networks in the US. Or anywhere.
Lol, have you ever been to Europe? iPhones are definitely considered premium and there definitely are networks that are more expensive but offer better reception. In Germany, that would be Telekom, in Switzerland, it's Swisscom.
Yes used to live in Germany. I was talking mainly about the US though.
iPhone isn't always 'premium', since they have their version of cheap phones as well. Point is cell network service quality is independent from phone quality.
iPhone isn't always 'premium', since they have their version of cheap phones as well. Point is cell network service quality is independent from phone quality.
A lot of this is bought in instalments isn’t it?
It sounds like this would have taken off if it were added to various managed cloud load balancers based on what you're saying.
The only question I have is if it opens up a different can of worms even if you've got a magic box terminating layer 7 for you or not. Never dug deep enough into mptcp myself to know.
The only question I have is if it opens up a different can of worms even if you've got a magic box terminating layer 7 for you or not. Never dug deep enough into mptcp myself to know.
I think it's a no brainer if it's no effort or small effort (set a socket option on the client, somehow)... but it's a big effort to support it in a large load balancing situation.
If you balance your load balancers with ECMP, I don't know if you can get two client streams to the same mptcp terminating place.
If you've optimized the heck out of your tcp flows, this throws a wrench in there, because the second stream is likely to get hashed into a different nic queue, and then you have communication between cpus to move forward on the logical stream.
It would have been really handy though, and solve real issues with real users.
Edit to add: it could also solve some issues on private networking / interserver networking I saw... although the contention would be a much bigger problem on higher bandwidth streams. On networks with link aggregation, while there are many paths from one host to another, usually path selection is by hashing the connection 5-tuple {src ip, dst ip, protocol, src port, dst port} so a long running tcp connection remains on the same path for the duration, if a path segment has high loss/corruption or is congested, MPTCP could help if you had an extra connection that hit a different path. Otherwise, you need to find the segment and get network operations to fix it; it's not easy to figure that out (i had to write a tool to sample and find port combinations with trouble and then a patch for mtr to run a trace with fixed ports) and then you still need to reconnect your affected tcp sockets unless you can get a quick response from net ops (sometimes they can check error stats once the right devices are pointed out to them, and then replacing a cable/fiber often helps, or disconnecting it during investigation can help the traffic flow across the redundant links)
If you balance your load balancers with ECMP, I don't know if you can get two client streams to the same mptcp terminating place.
If you've optimized the heck out of your tcp flows, this throws a wrench in there, because the second stream is likely to get hashed into a different nic queue, and then you have communication between cpus to move forward on the logical stream.
It would have been really handy though, and solve real issues with real users.
Edit to add: it could also solve some issues on private networking / interserver networking I saw... although the contention would be a much bigger problem on higher bandwidth streams. On networks with link aggregation, while there are many paths from one host to another, usually path selection is by hashing the connection 5-tuple {src ip, dst ip, protocol, src port, dst port} so a long running tcp connection remains on the same path for the duration, if a path segment has high loss/corruption or is congested, MPTCP could help if you had an extra connection that hit a different path. Otherwise, you need to find the segment and get network operations to fix it; it's not easy to figure that out (i had to write a tool to sample and find port combinations with trouble and then a patch for mtr to run a trace with fixed ports) and then you still need to reconnect your affected tcp sockets unless you can get a quick response from net ops (sometimes they can check error stats once the right devices are pointed out to them, and then replacing a cable/fiber often helps, or disconnecting it during investigation can help the traffic flow across the redundant links)
> If you balance your load balancers with ECMP, I don't know if you can get two client streams to the same mptcp terminating place.
At Google, we do something similar with QUIC and connection migration. Our mechanism for ensuring these hit the same backend is Maglev [0], where we use the QUIC connection ID for hashing purposes in software. (Our routers still mostly use ECMP based on the 5-tuple, so being able to consistently hash to the same backend across multiple LB instances is crucial.)
> if a path segment has high loss/corruption or is congested, MPTCP could help if you had an extra connection that hit a different path.
Incidentally, we also have a family of internal mechanisms that do this, although we don't rely on MPTCP. (We instead twiddle some other bits in the packet that we make sure our routers use for hashing, at least for RPCs between prod machines.) This inspired some of the connection migration work in our QUIC implementation [1], wherein we can migrate to a different ephemeral port if we detect issues with the current path. This works shockingly often for routing around network problems.
[0] https://research.google/pubs/maglev-a-fast-and-reliable-soft...
[1] https://github.com/google/quiche/blob/main/quiche/quic/core/...
At Google, we do something similar with QUIC and connection migration. Our mechanism for ensuring these hit the same backend is Maglev [0], where we use the QUIC connection ID for hashing purposes in software. (Our routers still mostly use ECMP based on the 5-tuple, so being able to consistently hash to the same backend across multiple LB instances is crucial.)
> if a path segment has high loss/corruption or is congested, MPTCP could help if you had an extra connection that hit a different path.
Incidentally, we also have a family of internal mechanisms that do this, although we don't rely on MPTCP. (We instead twiddle some other bits in the packet that we make sure our routers use for hashing, at least for RPCs between prod machines.) This inspired some of the connection migration work in our QUIC implementation [1], wherein we can migrate to a different ephemeral port if we detect issues with the current path. This works shockingly often for routing around network problems.
[0] https://research.google/pubs/maglev-a-fast-and-reliable-soft...
[1] https://github.com/google/quiche/blob/main/quiche/quic/core/...
> I've heard about MPTCP back in 2013.
> I assumed it's going to get adopted in no time due to how much of a ux improvement it would have been back in the day.
You might also be interested in SCTP[1] from the year 2000, which also hasn't gotten any traction so far.
[1]: https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...
> I assumed it's going to get adopted in no time due to how much of a ux improvement it would have been back in the day.
You might also be interested in SCTP[1] from the year 2000, which also hasn't gotten any traction so far.
[1]: https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...
> You might also be interested in SCTP[1] from the year 2000, which also hasn't gotten any traction so far.
Probably partly because middleware boxes (e.g., firewalls) either didn't/don't support it and/or rules were written to only support "TCP" (as opposed to 'stream') or "UDP" (as opposed to 'dgram'; see also "DCCP").
Probably partly because middleware boxes (e.g., firewalls) either didn't/don't support it and/or rules were written to only support "TCP" (as opposed to 'stream') or "UDP" (as opposed to 'dgram'; see also "DCCP").
Certainly that's a part, but it didn't help that SCTP has some fundamental low-level flaws.
Given that TCP also has at least one unfixable flaw, the only recommendation I can make is to use something UDP-based - which, to make sure you don't stomp on everybody else's traffic, means use the only popular one: QUIC (the layer beneath HTTP/3).
Given that TCP also has at least one unfixable flaw, the only recommendation I can make is to use something UDP-based - which, to make sure you don't stomp on everybody else's traffic, means use the only popular one: QUIC (the layer beneath HTTP/3).
The protocol is specified by a byte in the IP packet; how many middleware boxes block everything except for ICMP, TCP, and UDP? What is the probability that a packet with that byte set to something unexpected actually gets from source to destination?
The “funny” thing is that http3 really really looks like a transport protocol encapsulated into… uso. Exactly because many middle boxes block anything that’s not a very well known protocol
The internet is just broken and only works because of lot of hacked bandaids.
> The protocol is specified by a byte in the IP packet; how many middleware boxes block everything except for ICMP, TCP, and UDP?
Most firewalls are default deny out of the box and you have to allow things through. How many folks bother opening up SCTP/DCCP/etc?
Most firewalls are default deny out of the box and you have to allow things through. How many folks bother opening up SCTP/DCCP/etc?
How does sctp work with NAT that your typical home box uses?
SCTP can run over UDP. It's part of the spec.
Now we have HTTP3 which runs over UDP - where there is a will, there is a way.
Perhaps SCTP was ahead of its time.
Now we have HTTP3 which runs over UDP - where there is a will, there is a way.
Perhaps SCTP was ahead of its time.
> SCTP can run over UDP. It's part of the spec.
SCTP over UDP came out in 2013:
* https://datatracker.ietf.org/doc/html/rfc6951
SCTP came out in 2000:
* https://datatracker.ietf.org/doc/html/rfc2960
Over a decade is quite a while in Internet-time.
SCTP over UDP came out in 2013:
* https://datatracker.ietf.org/doc/html/rfc6951
SCTP came out in 2000:
* https://datatracker.ietf.org/doc/html/rfc2960
Over a decade is quite a while in Internet-time.
SCTP is used a lot inside telco networks for carrying switching control metadata for voice connections. https://en.wikipedia.org/wiki/SIGTRAN
WebRTC data channels use SCTP, which ain't nothing!
https://datatracker.ietf.org/doc/html/rfc8831
(SCTP over DTLS, that is...)
(SCTP over DTLS, that is...)
I was excited about it because we were working on delivery robots and I wanted a good solution for instant failover given 2 cellular modems.
We ended up going with PepLink's SpeedFusion to save engineering time. But the license was costly. I really hope for a free solution in the future for 2 cellular networks and <50ms failover.
Multipath UDP + OpenVPN would also probably be a viable solution.
We ended up going with PepLink's SpeedFusion to save engineering time. But the license was costly. I really hope for a free solution in the future for 2 cellular networks and <50ms failover.
Multipath UDP + OpenVPN would also probably be a viable solution.
I created something like what you're describing with the addition of P2P communication using NAT traversal (https://www.hyperpath.ie)
It will connect your devices in a P2P Mesh VPN and allow them to send and receive data using multiple links (e.g. multiple 5G or 5G + Satellite).
It is significantly cheaper than Peplink's license, less latency and no bandwidth / data limits.
You need to bring your own hardware though. Like a Raspberry Pi with 3 USB 4G/5G dongles.
It will connect your devices in a P2P Mesh VPN and allow them to send and receive data using multiple links (e.g. multiple 5G or 5G + Satellite).
It is significantly cheaper than Peplink's license, less latency and no bandwidth / data limits.
You need to bring your own hardware though. Like a Raspberry Pi with 3 USB 4G/5G dongles.
what about something like this? two minipcie slots which i suppose you could put two cellular modems into. not sure what OS it runs though but presumably some flavor of linux.
https://mikrotik.com/product/rbm33g#fndtn-specifications
maybe someone could make one that uses an RPi compute module instead.
https://mikrotik.com/product/rbm33g#fndtn-specifications
maybe someone could make one that uses an RPi compute module instead.
It looks like it runs "RouterOS" which has a Linux kernel, so it should be possible to run it there.
I found this board on AliExpress (https://www.aliexpress.com/item/1005003540616473.html?spm=a2...) based on the CM4 and with 3 cellular modems. That could also be a good candidate.
Also found this one (https://www.gateworks.com/products/industrial-single-board-c...) US-made and has 3 minipcie slots (other options available with 2 and 4)
I found this board on AliExpress (https://www.aliexpress.com/item/1005003540616473.html?spm=a2...) based on the CM4 and with 3 cellular modems. That could also be a good candidate.
Also found this one (https://www.gateworks.com/products/industrial-single-board-c...) US-made and has 3 minipcie slots (other options available with 2 and 4)
I see it as depressing that this is gaining traction it doesn't deserve. TCP doesn't need one hack at a time and then to make us choose combinations that sort of work in half the use cases in the modern world, it needs to be replaced with SCTP.
ZeroMQ yeah!
I don't know which makes me sadder-- IPv4 only having a 32-bit address space or TCP using the source and destination IP addresses in the connection tuple. That's one of those "if I had a time machine" of things-- I'd go back and have Cert and Kahn change both of those items.
I think they'd point out that they gave you source routing which is half of what you want and is correctly specified as an option.
How would you change TCP?
Do you mean how we have to use 4 fields to track a connection - IP address and port for both ends?
Do you mean how we have to use 4 fields to track a connection - IP address and port for both ends?
If TCP had a protocol specific identifier for connections (a couple of 32-bit values, for example-- a client nonce and server nonce) rather than using the source/destination IP addresses multi-homed hosts and seamless transition between different networks would become native features of the protocol. A client could roam between two different IP networks and TCP connections would "survive", for example. (I'm oversimplifying nearly to the point of hyperbole, to be sure...)
(Another fun future would have been one where SCTP got widespread adoption.)
(Another fun future would have been one where SCTP got widespread adoption.)
a client nonce and server nonce) rather than using the source/destination IP addresses multi-homed hosts and seamless transition between different networks would become native features of the protocol. A client could roam between two different IP networks and TCP connections would "survive", for example.
This is mostly how Mosh [1] works and allows for IP roaming, changing IP's, etc... without losing ones SSH session. The connection can even be interrupted for a prolonged period of time and restore on its own on a new IP seamlessly.
[1] - https://mosh.org/
This is mostly how Mosh [1] works and allows for IP roaming, changing IP's, etc... without losing ones SSH session. The connection can even be interrupted for a prolonged period of time and restore on its own on a new IP seamlessly.
[1] - https://mosh.org/
How would routing be done without source/destination? When the device changes networks, how does the origin and all routers along the way know that this device is on a new network?
> How would routing be done without source/destination?
There is still a source/destination address. Routing still works. But those addresses are allowed to change without disrupting the connection because the connection isn't based on the values of these addresses.
> When the device changes networks, how does the origin and all routers along the way know
The routers don't need to "know" these things.
MPQUIC does this. To the network it's just UDP packets moving around. Connection state is dealt with at higher levels and doesn't rely on IP addresses.
There is still a source/destination address. Routing still works. But those addresses are allowed to change without disrupting the connection because the connection isn't based on the values of these addresses.
> When the device changes networks, how does the origin and all routers along the way know
The routers don't need to "know" these things.
MPQUIC does this. To the network it's just UDP packets moving around. Connection state is dealt with at higher levels and doesn't rely on IP addresses.
> how does the origin and all routers along the way
It's just the origin that needs to know what address(es) it should be using as the destination at layer 3.
The big problems with this is that it depends upon things that weren't really feasible in the early 80's -- bigger packet headers, a bit more state on each side of the connection, potential need for cryptographic authentication.
It's just the origin that needs to know what address(es) it should be using as the destination at layer 3.
The big problems with this is that it depends upon things that weren't really feasible in the early 80's -- bigger packet headers, a bit more state on each side of the connection, potential need for cryptographic authentication.
Yeah it's a shortsighted plan. How would i set up rules on firewall if I don't know how to distingush the connections?
There's two separate ideas here:
* Where to send a frame to get to the other side of the connection
* Whose connection this is.
TCP combined the two, because we didn't have mobile clients or a lot of multihomed systems that would benefit from distinguishing them. Also, every octet in the header counted.
In practice, this means we have to keep building a lot of infrastructure on top of TCP (or parallel to it, in datagram protocols) to handle retries and splitting flows well. In turn, these things are completely opaque to the network and it's difficult to write rules about them.
Whereas if we had different packet fields for "where am I sending this packet right now" and "whose flow does this belong to"? we could write better firewall rules, have less infrastructure built on top of TCP, and have better typical application performance.
* Where to send a frame to get to the other side of the connection
* Whose connection this is.
TCP combined the two, because we didn't have mobile clients or a lot of multihomed systems that would benefit from distinguishing them. Also, every octet in the header counted.
In practice, this means we have to keep building a lot of infrastructure on top of TCP (or parallel to it, in datagram protocols) to handle retries and splitting flows well. In turn, these things are completely opaque to the network and it's difficult to write rules about them.
Whereas if we had different packet fields for "where am I sending this packet right now" and "whose flow does this belong to"? we could write better firewall rules, have less infrastructure built on top of TCP, and have better typical application performance.
But the stuff that carries TCP is IP. That's why TCP can work seamlessly, because it uses identification from a previous layer. Consider I bind a server to an ID, and not IP:port, the operating system running it must know how to communicate that via IP, so there will be a corellation map somewhere and that map needs to be synchronized between all peers that wish to host the roaming server.
Otherwise you're just switching port (16-bit) value to arbitrary 32-bit identifier.
Otherwise you're just switching port (16-bit) value to arbitrary 32-bit identifier.
If TCP didn't use L3 source and destination addresses to distinguish connections, it could be more easily taught to deal with:
* Clients roaming between L3 addresses
* Clients/servers with multiple L3 addresses
* Clients roaming between L3 addresses
* Clients/servers with multiple L3 addresses
But... it doesn't? TCP has no notion of IP address in the protocol, only the port.
TCP with changing IPs can work e.g. on top of an ip-ip tunnel with applications not being aware at all.
> TCP has no notion of IP address in the protocol,
RFC793:
> TCP with changing IPs can work e.g. on top of an ip-ip tunnel with applications not being aware at all.
That's just because the IPs have not changed from its point of view: it receives the same frame with the same destination/source IP addresses the entire time.
Part of the reason why we need things like IP-IP tunnels is because L4 connections can't "move" with TCP. In scenarios where we're using tunneling for this, we're accepting worse performance than if we could just directly send TCP to its true destination and have it processed.
RFC793:
To allow for many processes within a single Host to use TCP
communication facilities simultaneously, the TCP provides a set of
addresses or ports within each host. **Concatenated with the network
and host addresses from the internet communication layer,** this forms
a socket. A pair of sockets uniquely identifies each connection.
That is, a socket may be simultaneously used in multiple
connections.
> TCP with changing IPs can work e.g. on top of an ip-ip tunnel with applications not being aware at all.
That's just because the IPs have not changed from its point of view: it receives the same frame with the same destination/source IP addresses the entire time.
Part of the reason why we need things like IP-IP tunnels is because L4 connections can't "move" with TCP. In scenarios where we're using tunneling for this, we're accepting worse performance than if we could just directly send TCP to its true destination and have it processed.
So you want to implement persistent connections on L4 without implementing persistent addresses on L3 first?
This doesn't make much sense to me. The hardest problem here is not assigning uuids to pipes, it's the routing/mapping of the "true destination".
- If you manage to solve it on L3, ip-ip tunnels or not — you have it, TCP works unmodified and so does UDP and everything else including quic and http/3.
- If you didn't solve it, then support for persistent connections in TCP is useless.
In another words I don't see what a "transmission control protocol" has to do with it. It's very reasonable to assume that addreses are already figured out when designing transmission control and that's exactly what TCP did.
This doesn't make much sense to me. The hardest problem here is not assigning uuids to pipes, it's the routing/mapping of the "true destination".
- If you manage to solve it on L3, ip-ip tunnels or not — you have it, TCP works unmodified and so does UDP and everything else including quic and http/3.
- If you didn't solve it, then support for persistent connections in TCP is useless.
In another words I don't see what a "transmission control protocol" has to do with it. It's very reasonable to assume that addreses are already figured out when designing transmission control and that's exactly what TCP did.
> - If you didn't solve it, then support for persistent connections in TCP is useless.
SCTP and multipath TCP (which is what we're talking about) already do pretty much this. Assuming that endpoints to a stream connection have single, unchanging network addresses isn't a reasonable assumption anymore. But we're stuck with the assumption that hosts won't move in one of our most common protocols.
https://en.wikipedia.org/wiki/Multipath_TCP#/media/File:Diff...
https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...
In the OSI model, you got similar functionality up at layer 5, but TCP only handles the connection/disconnection aspect of the session layer. In the internet world, we have a bunch of haphazard sets of retries, session balancing, multihoming and reconnecting behavior that are protocol specific (and completely missing from many well-used protocols) kludged on top. (Actually arguably MP-TCP is a session layer on top of TCP).
The only way you solve this on layer 3 is to build some kind of messy overlay network, because addresses have no real relation to where things are anymore. And we know that overlay networks are suboptimal and inefficient. Solving it at layer 4 doesn't have to be (but it's too late for that now).
SCTP and multipath TCP (which is what we're talking about) already do pretty much this. Assuming that endpoints to a stream connection have single, unchanging network addresses isn't a reasonable assumption anymore. But we're stuck with the assumption that hosts won't move in one of our most common protocols.
https://en.wikipedia.org/wiki/Multipath_TCP#/media/File:Diff...
https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...
In the OSI model, you got similar functionality up at layer 5, but TCP only handles the connection/disconnection aspect of the session layer. In the internet world, we have a bunch of haphazard sets of retries, session balancing, multihoming and reconnecting behavior that are protocol specific (and completely missing from many well-used protocols) kludged on top. (Actually arguably MP-TCP is a session layer on top of TCP).
The only way you solve this on layer 3 is to build some kind of messy overlay network, because addresses have no real relation to where things are anymore. And we know that overlay networks are suboptimal and inefficient. Solving it at layer 4 doesn't have to be (but it's too late for that now).
The protocol would have to handle binding the network to the transport. MPTCP and SCTP both handle that via registering and un-register network layer endpoints. This parallel universe TCP would be the same in that regard.
(I did say I was oversimplifying...
(I did say I was oversimplifying...
How do you even firewall such a protocol?
Like the IPv6 flow label?
The problem is that the TCP/IP model stops at level 4, and if we consider TCP a protocol of transport, it shouldn't do that.
In the OSI model what you talk about is level 5, that is session, but in TCP/IP there is no such level, thus it must be handled by the application (e.g. trough a session cookie, in HTTP).
In the OSI model what you talk about is level 5, that is session, but in TCP/IP there is no such level, thus it must be handled by the application (e.g. trough a session cookie, in HTTP).
Slavish adherence to theoretical models is a recipe for failure. Even worse, the OSI model was developed in the 1970s before successful internetworks existed so it's not informed by experience; it's mostly made up.
I disagree. There’s no reason TCP couldn’t work like that. See QUIC for example
Too bad there is no link to projects using mptcp, such as Openwrt derivatives.
PS: I mentored a student for 2 years at GSOC to patch Openwrt with mptcp:
https://blog.freifunk.net/2017/05/29/gsoc-2017-add-mptcp-sup...
PS: I mentored a student for 2 years at GSOC to patch Openwrt with mptcp:
https://blog.freifunk.net/2017/05/29/gsoc-2017-add-mptcp-sup...
I've been looking at this project for a while which may be interesting to you: https://github.com/Ysurac/openmptcprouter.
I recently bought a property where I cannot get a full fibre connection, but I can get 150-400 Mbps using 5G. I've been thinking about using dual 5G connections and tunneling my traffic via mptcp to a VPS to aggregate the connections.
I recently bought a property where I cannot get a full fibre connection, but I can get 150-400 Mbps using 5G. I've been thinking about using dual 5G connections and tunneling my traffic via mptcp to a VPS to aggregate the connections.
It was recently enabled in the Home Assistant ‘HAOS’ kernel.
https://github.com/home-assistant/operating-system/pull/3248
I hope some apps will start using it :)
one example
on openwrt
http://www.openmptcprouter.com/
Good point, the last version (v0.60) is using the upstream kernel by default. I just added it in the list: https://www.mptcp.dev/apps.html#misc
Whats the benefit of an openwrt router supporting mptcp?
I'd think support on web servers and mobile devices would be most important...
I'd think support on web servers and mobile devices would be most important...
Link aggregation and redundancy
I got fiber run to my neighborhood, and for a while, had a 1gb coax connection and a 1gb fiber connection. I used openmptcprouter to aggregate my connections through a droplet and I effectively had a 2 gigabit internet connection. I would have stuck with it, but having a datacenter IP for your home network really doesn’t work.
Except TCP is just a bad protocol to start with for tunnelling, because packetized data has to be delivered in-order, and head of line blocking messes up congestion control algorithms in the tunnelled data.
Why does this require explicit opt in by applications if there’s transparent fallback? Wouldn’t it make most sense for the kernel to do it transparently for every TCP connection so that it can make more global decisions about path aggregation / link preference?
Using this implies that there are multiple IPs per endpoint associated with a single TCP connection. That is going to need explicit support/awarness by the application in many cases.
My understanding is that it was basically a condition enforced by the maintainers of the Linux TCP / networking subsystems. If you look at the initial upstreaming discussions[1], this was setup as a ground rule.
If you look at the older multipath TCP implementation, prior to the upstreaming, it was intended to be fully transparent to the application, which I think makes more sense for the intent of the protocol. Sure, in many cases MPTCP may be better with application-guided logic, but having a standard system approach (e.g. establish sub-flows on an LTE connection for automatic failover, but don't send any data along those sub-flows) would have worked for 95% of cases.
[1] https://lore.kernel.org/all/alpine.OSX.2.21.1707181728570.11...
If you look at the older multipath TCP implementation, prior to the upstreaming, it was intended to be fully transparent to the application, which I think makes more sense for the intent of the protocol. Sure, in many cases MPTCP may be better with application-guided logic, but having a standard system approach (e.g. establish sub-flows on an LTE connection for automatic failover, but don't send any data along those sub-flows) would have worked for 95% of cases.
[1] https://lore.kernel.org/all/alpine.OSX.2.21.1707181728570.11...
Yes, that's correct. In the Linux kernel, it would not be possible to switch to MPTCP by default.
But apps can use it by default. For the server case, it really makes sense: https://www.mptcp.dev/faq.html#why--when-should-mptcp-be-ena...
GNU/Linux distributions could even switch MPTCP on by default (via eBPF).
But apps can use it by default. For the server case, it really makes sense: https://www.mptcp.dev/faq.html#why--when-should-mptcp-be-ena...
GNU/Linux distributions could even switch MPTCP on by default (via eBPF).
I can imagine new security holes being opened up by allowing multiple IP's to talk over the same TCP connection...
Imagine you have an application which checks the clients IP (eg. against a whitelist) at the time of connection and then assumes it doesn't change...
Imagine you have an application which checks the clients IP (eg. against a whitelist) at the time of connection and then assumes it doesn't change...
The only practical use of MPTCP for me is to use mobile and Wi-Fi network together to boost the speed. iOS and WeChat both support this. However, I always turn them off because my mobile network is metered. So in the end, MPTCP is useless for me *personally*.
I worked on this. We called it the parking lot bug. WiFi still shows signal but no proper connection. With MPTCP, it will failover to cell.
You can even learn to predict these handovers to make it really smooth [0].
[0] https://arxiv.org/pdf/1907.10493.pdf
[0] https://arxiv.org/pdf/1907.10493.pdf
I work supporting, debugging, fixing the Linux network stack and drivers. I am amazed how little adoption this has seen.
Like everything which came along and tried to supplant regular TCP, such as SCTP, it seems MPTCP has also been confined to a niche of application developers who will use it forever while the rest of the world forgets about it.
Like everything which came along and tried to supplant regular TCP, such as SCTP, it seems MPTCP has also been confined to a niche of application developers who will use it forever while the rest of the world forgets about it.
Other folks have commented - Apple Siri uses mptcp so not exactly niche considering how many devices that is
Similarly, I promise you everything you've ever done over a 4G or 5G phone has used SCTP inside the phone provider. That doesn't mean the general developer population know about it. I bet most developers have never even heard of it.
I worked on IMS before it was widely rolled out and at least back then its was just SIP over TCP and UDP (RTP/RTCP) for media. SCTP was widely used for eNodeB - MME comms tho iirc
I found [1] which describes the architectural difference between MPTCP and QUIC, and also introduces the authors' proposed MPQUIC protocol:
> QUIC multiplexes application streams on a single UDP flow, whereas MPTCP splits a single stream on multiple TCP subflows. MPQUIC combines both features by multiplex- ing application streams on multiple UDP subflows.
[1]: "Multipath QUIC: A Deployable Multipath Transport Protocol" https://www.researchgate.net/publication/327122884_Multipath...
Now I'm curious about how these protocols compare in production operation. Anybody have experience with both?
> QUIC multiplexes application streams on a single UDP flow, whereas MPTCP splits a single stream on multiple TCP subflows. MPQUIC combines both features by multiplex- ing application streams on multiple UDP subflows.
[1]: "Multipath QUIC: A Deployable Multipath Transport Protocol" https://www.researchgate.net/publication/327122884_Multipath...
Now I'm curious about how these protocols compare in production operation. Anybody have experience with both?
Note that MPQUIC is still being discussed at the IETF. At the last IETF meeting, more changes have been discussed. Unfortunately, that slows down its adoption. https://lwn.net/Articles/964377/
But both tries to achieve the same goal. Technically, you can have a very similar behaviour. MPTCP is implemented in the Linux kernel, while QUIC is on the userspace side.
But both tries to achieve the same goal. Technically, you can have a very similar behaviour. MPTCP is implemented in the Linux kernel, while QUIC is on the userspace side.
Apple supports it too and uses it for Siri https://developer.apple.com/documentation/foundation/urlsess...
It’s quite easy to use in other app - it’s there in the basics. We were quite amazed to find our voip app to be quite robust in 2011 :D
> If any middlebox in between does not support it, the returned SYN+ACK packet will not contain MPTCP options in the TCP option field.
That sounds .. quite restrictive. Is the only requirement on a middlebox to just forward the MPTCP options as-is?
That sounds .. quite restrictive. Is the only requirement on a middlebox to just forward the MPTCP options as-is?
Before we finalized the specs we did a lot of testing to make sure that enabling MPTCP wouldn't break connectivity. Either it would be passed correctly, or it would fall back to single-path TCP safely. Generally, so long as a middlebox passes through unknown options unmodified, and does not try to enforce that the TCP sequence space it sees is contiguous, MPTCP should work through that middlebox.
If you're interested, we wrote a couple of papers on this:
[1] https://www.usenix.org/conference/nsdi12/technical-sessions/...
[2] https://www.researchgate.net/publication/229002024_Is_it_sti...
If you're interested, we wrote a couple of papers on this:
[1] https://www.usenix.org/conference/nsdi12/technical-sessions/...
[2] https://www.researchgate.net/publication/229002024_Is_it_sti...
This can help in security/privacy setting.
for example Great Chinese firewall: if you can split your traffic across multiple uplink channels, the firewall will have a hard time to put them together for enforcement?
for example Great Chinese firewall: if you can split your traffic across multiple uplink channels, the firewall will have a hard time to put them together for enforcement?
They just block or rate limit it severely when they dont know what it is.
The examples given on the page seem to focus on multipath to get to a device over the internet, but I can see this being more likely to work properly without needing to fallback on home networks.
At home/lan we use LACP, VRRP... I mean link aggregation and HA needs are solved time ago.
With multiple ISPs, or on a complex enough LAN, we can use multiple routing tables + weights too.
Also, if the ISP at home can do 10Gbps, 1Gbps, 300 Mbps whatever... I want to be able to use them with a single path, so there is no gain using multiple paths. Eventually, when I have cable+wifi connected at the same time, I use to force one of both, cannot see a reason to prefer using both at the same time.
Maybe the latency thing? Never had that issue at home, but could understand that usage case "just use the network segment with less latency to reach $thing".
With multiple ISPs, or on a complex enough LAN, we can use multiple routing tables + weights too.
Also, if the ISP at home can do 10Gbps, 1Gbps, 300 Mbps whatever... I want to be able to use them with a single path, so there is no gain using multiple paths. Eventually, when I have cable+wifi connected at the same time, I use to force one of both, cannot see a reason to prefer using both at the same time.
Maybe the latency thing? Never had that issue at home, but could understand that usage case "just use the network segment with less latency to reach $thing".
> Also, if the ISP at home can do 10Gbps, 1Gbps, 300 Mbps whatever... I want to be able to use them with a single path, so there is no gain using multiple paths. Eventually, when I have cable+wifi connected at the same time, I use to force one of both, cannot see a reason to prefer using both at the same time.
>
I don't understand why you would want to be able to use them with a single path. the gain would be being able to aggregate them and have individual tcp streams faster than any one IP connection could handle.
Though personally I think the resilience is more appealing. Not having to have a hard cutover when wifi degrades as I walk away would be nice
I don't understand why you would want to be able to use them with a single path. the gain would be being able to aggregate them and have individual tcp streams faster than any one IP connection could handle.
Though personally I think the resilience is more appealing. Not having to have a hard cutover when wifi degrades as I walk away would be nice
If my ISP gives me 10 Gbps, I want my PC to have (at least) a 10Gbps single path to the router.
So, If I already have a 10 Gbps path to the router, I don't want to add a 300 Mbps failing air path added to my way to the router.
In the context of the parent (at home networks), I think most people has two paths... WiFi or RJ45-UTP. And with that multipath setup (WiFi + RJ45, I don't get why other comments are talking about cellular networks "at home") is not usual to walk away; right, you could walk, as far as long is the rj45 cable, but...
To keep HA on WiFi when walking around, there are other technologies more battle tested than MPTCP.
So, If I already have a 10 Gbps path to the router, I don't want to add a 300 Mbps failing air path added to my way to the router.
In the context of the parent (at home networks), I think most people has two paths... WiFi or RJ45-UTP. And with that multipath setup (WiFi + RJ45, I don't get why other comments are talking about cellular networks "at home") is not usual to walk away; right, you could walk, as far as long is the rj45 cable, but...
To keep HA on WiFi when walking around, there are other technologies more battle tested than MPTCP.
For a long time enterprise firewalls (and more recently SD-WAN) allowed load balancing between different links, but unlike MPTCP the traffic of a single TCP connection is not split up. This is in line with the established network admin wisdom saying that reordering packets of a TCP connection hurts performance.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Ho...
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Ho...
Some ISPs in Europe are using MPTCP for people being too far from the street cabinets. Typically, for people in the countryside, with < 50 Mbps. Thanks to a transparent proxy installed in the home gateway, and servers in the ISP's network, they can combine both the fixed and cellular networks, and use the fixed one in priority.
MPTCP can also be very interesting for mobility use-cases, even when one network is used at a time, e.g. switching from WiFi to cellular, or different cellular networks in the train, etc.
MPTCP can also be very interesting for mobility use-cases, even when one network is used at a time, e.g. switching from WiFi to cellular, or different cellular networks in the train, etc.
It made so much sense back then, when mobile apps were not that robust to networks changing, I assumed it's going to get adopted in no time due to how much of a ux improvement it would have been back in the day.
It's incredibly depressing that this gained barely any traction in the last 10 years, and kernel options are appearing just recently, after everyone has wrapped they http calls in multiple retry handlers, and mobile operating systems have abstracted network connectivity to the point where it feels more like you are using zeromq rather than tcp.