OpenMPTCProuter(openmptcprouter.com)
openmptcprouter.com
OpenMPTCProuter
https://www.openmptcprouter.com
33 comments
This project is truly an outstanding bit of open-source software. I'm using it to bond two VDSL2 connections together; both sync at 79mbps and via OpenMPTCPRouter and a £3/mo VPS I get 140mbps download speeds.
I'm glad other people are having luck with this, but I just couldn't get it to work with my setup. I'm wondering if it's because of the weird MTU and CGNAT on the connections I'd like to bond (300Mbps up and down municipal-ish WiFi).
I was able to get two OpenMPTCProuter installations talking to each other, but downstream traffic wouldn't balance across interfaces, and upstream was limited to 0.25-0.5Mbps (not a typo, and the links on their own have >300Mbps capability). I'll maybe try it again, but for now I have to add this to the pile of solutions that haven't worked for me (including VyOS WAN aggregation, ZeroTier multipathing, and Wireguard and ECMP).
I was able to get two OpenMPTCProuter installations talking to each other, but downstream traffic wouldn't balance across interfaces, and upstream was limited to 0.25-0.5Mbps (not a typo, and the links on their own have >300Mbps capability). I'll maybe try it again, but for now I have to add this to the pile of solutions that haven't worked for me (including VyOS WAN aggregation, ZeroTier multipathing, and Wireguard and ECMP).
Any idea what the most cost effective VPS for this purpose would be?
I use the Vultr $5/mo
I use and love vultr too, but their IP space is shit for VPN purposes. Constant captchas from google and eBay. Delta.com is blocked completely. Amazon refuses to play videos and says to disconnect your VPN.
Which latency do you get?
I have a cheap VPS for non latency-sensitive stuff, and I just noticed that a ping to returns 15ms more than 1.1.1.1 (for instance). I'm wondering if you can get a cheap VPS with a good latency.
I have a cheap VPS for non latency-sensitive stuff, and I just noticed that a ping to returns 15ms more than 1.1.1.1 (for instance). I'm wondering if you can get a cheap VPS with a good latency.
The latency depends on the geographical location (well, network location but geographical is a good proxy most of the time) of the VPS more than anything else. Buy closer if you want it lower.
Is there a solution that is better than a failover which will not break the TPC connection for the client. Say I have two independent ISP routers, an offshore VPS with a
perfect stable connection, and an old video call software which disconnect the call when the TCP stream breaks. Is there a custom protocol that can retry packets on the other 2nd ISP connection, or even say duplicate all packets like RAID 1 for hard drives, and the VPS+Client will determine which packets made it and which didn't, ensuring perfect connections.
https://zerotier.atlassian.net/wiki/spaces/SD/pages/56845926...
Zerotier has started supporting multipath, that's what I'd use.
Zerotier has started supporting multipath, that's what I'd use.
I've looked at Zerotier many times over the years. You are using this as your dedicated VPN to load balance multi WAN connections, is that what you are saying?
I'm using it for fallback connections on robots, but I think that's a usecase it would be good for as well.
There is a redundant mode available, but even in default mode when a connection break you should not have any problem on TCP stream that will just use another available connection.
Is MPTCP support required on both source and destination up to the app level?
No, the MPTCP part of the link is between the router and the VPS. The end application doesn't see this.
Correct. It's more like a VPN that bonds multiple commodity internet connections for better performance. OPENMPTCP tries to be as transparent as possible. There are a handful of servers on the internet that actually support end to end MPTCP as well so the above question is legitimate.
... isn't this exactly that? am i missing something from your comment?`to be clear using mptcp your packets will leave your gateway on redundant links simultaneously while keeping connection characteristics such as bandwidth latency reliability and availability separate afaict.
Yes, this is exactly that. I've been using that openwrt distribution to do just that for a year and it's been working out really great. The nice thing about the approach is that you get a boost in both bandwidth and reliability as there is always a link available for your packets.
The drawback is increased latency (about 60ms instead of 40 on each individual links) but openmptcprouter provides a nice tool to easily exclude specific services or devices from using the tunnel. My gaming packets and netflix are both setup to keep using the otherwise adsl link. (Netflix override is because their stupid geo check think my vps is a vpn endpoint used to mask my location).
The drawback is increased latency (about 60ms instead of 40 on each individual links) but openmptcprouter provides a nice tool to easily exclude specific services or devices from using the tunnel. My gaming packets and netflix are both setup to keep using the otherwise adsl link. (Netflix override is because their stupid geo check think my vps is a vpn endpoint used to mask my location).
is that latency coming from routing to your upstream network or the packet scheduling itself?
Most likely the packet scheduling, I suppose the mpctp kernel module has sometimes to wait to get all packets in the right order.
Also, I am aggregating 2 4G link which might not help since both links send large packets of data infrequently if I understand correctly. Maybe aggregating 2 wired link would have lower increased latency cost.
In the end, I'm getting up to 40Mbits of bandwidth instead of being stuck with my 3MBits adsl link and I'm pretty happy.
Also, I am aggregating 2 4G link which might not help since both links send large packets of data infrequently if I understand correctly. Maybe aggregating 2 wired link would have lower increased latency cost.
In the end, I'm getting up to 40Mbits of bandwidth instead of being stuck with my 3MBits adsl link and I'm pretty happy.
that is actually somewhat impressive performance in my books. i would guess 4G links are inherently unstable regarding prediction of connection characteristics for proper scheduling... thanks for sharing your experience with that :)
look into ECMP
Why would I use this over ECMP via for example two OSPF paths? Which is proven robust technology. Honest question.
For the "EC" of ECMP and for the "TCP" of MPTCP. I'm only half-joking:
MPTCP works just fine on different internet connections, while ECMP assumes everything has the same bandwidth. And the MPTCP gives the reliability of TCP: if one connection is dead, it'll detect it and transmit lost packets to the other one. It even supports break-before-make, so you can loose all your internet connections, then get a new one, and it'll resume.
MPTCP works just fine on different internet connections, while ECMP assumes everything has the same bandwidth. And the MPTCP gives the reliability of TCP: if one connection is dead, it'll detect it and transmit lost packets to the other one. It even supports break-before-make, so you can loose all your internet connections, then get a new one, and it'll resume.
ECMP typically doesn't split a single flow over both connections, and due to hashing might very well try to send multiple flows over one connection while underutilizing the other. Typical ECMP as far as I know also doesn't handle links of different speeds very well.
Indeed, ECMP uses some kind of hashing (dependant on the implementation) to do the distribution and doesn't take different line speeds into account at all.
Great improvement if this project does take this into account.
The C in ECMP stands for “cost”, which is usually calculated as the number of hops or intermediate nodes to the destination. If you were to use link speed as the “cost”, links of unequal speed wouldn’t be equal cost anymore. You can certainly use ECMP over links of different speeds, but as you said bandwidth doesn’t scale for a single connection, so any TCP connections that end up on the slower link perform at the speed of the slower link and it’s difficult to figure out why.
This is intentional: TCP treats packets received out-of-order as representative of an overloaded network. There’s generally a guarantee that packets sent in-order over a single link will be received in that order, but no such guarantee exists for packets sent through two completely independent links/networks. You need something on the receiving end (either at the destination or at the point where the separate paths converge) that will coalesce and re-order packets, such as an MPTCP or PPP implementation.
This is intentional: TCP treats packets received out-of-order as representative of an overloaded network. There’s generally a guarantee that packets sent in-order over a single link will be received in that order, but no such guarantee exists for packets sent through two completely independent links/networks. You need something on the receiving end (either at the destination or at the point where the separate paths converge) that will coalesce and re-order packets, such as an MPTCP or PPP implementation.
Obligatory "we did that back in the day with Multilink PPP" ... which sucked ROCKS, had no real standards, and only actually worked with personally applied patches and black magic in my experience.
Really glad to see this project advancing.
Really glad to see this project advancing.
I only got multilink PPP working on FreeBSD with mpd5 between two systems. A laptop with four USB<->RS232 adapters and the other an outdated piece of crap with a quad port RS232 card (running at 1Mbaud). It worked at close to the theoretical throughput and allowed the old Pentium 3 to join a conference wifi network that disabled legacy 802.11b to preserve airtime.
I had a patchset to make linux 2.0 / 2.2 talk PPP to several ports on livingston gear or itself. I got reports that it worked in some other situations; and there was an interesting bug report from an early version of Android that looked a lot like one of the failure modes i saw and never fixed in that code.
Best I ever did with that code was a bank of 8 ISDN modems to a PM3 that we did as a test at the ISP's office just to see how far it could stretch. 230kbps serial ports on an ISA bus.
Best I ever did with that code was a bank of 8 ISDN modems to a PM3 that we did as a test at the ISP's office just to see how far it could stretch. 230kbps serial ports on an ISA bus.
Yeah...we never had success with the OSS solutions here. We used a lot of Tiara Networks (later bought by Nortel) boxes for this sort of thing. Worked great, but expensive, closed source, yada-yada. If there were only a couple of parallel paths, routers worked reliably if not always optimally or cheaply; sticking to one vendors multilink PPP was generally reliable, and worst case there is OSPF as one mysteriously downvoted commenter upstream noted.
My partner and I have been preparing for a long-term #vanlife traveling session (think: year or two) and this project has made the connectivity part much easier.
That being said, the script to install the VPS works great but is absolutely horrible. When I can finally get some time I'd love to try to make it better
That being said, the script to install the VPS works great but is absolutely horrible. When I can finally get some time I'd love to try to make it better