Linux Networking Shallow Dive: WireGuard, Routing, TCP/IP and NAT(im.salty.fish)
im.salty.fish
Linux Networking Shallow Dive: WireGuard, Routing, TCP/IP and NAT
https://im.salty.fish/index.php/archives/linux-networking-shallow-dive.html
65 comments
I experienced this back when I configured my home Linux boxes as a router, VPN server, firewall, media server, etc. Since I had the time, compiled all of the info I found on random blogs and sites and added them to the Ubuntu Community wiki. That was the 12.x days, when Ubuntu was in its prime and the distro to use.
While these blogs were a great resource, I often found the commands outdated or applied to a different distro. A distro specific wiki solves both those issue. While I don't get the glory of a blog, I just checked an it's nice to see my notes still there for future Denvercoder9's.
While these blogs were a great resource, I often found the commands outdated or applied to a different distro. A distro specific wiki solves both those issue. While I don't get the glory of a blog, I just checked an it's nice to see my notes still there for future Denvercoder9's.
The Gentoo Wiki was a great resource for many networking questions, even when I wasn't using Gentoo.
A major hurdle is simply learning how to describe what you want to do "in the industry terms" - "I can't access my server from my computer but it works from the Internet" is a lot easier to resolve when you learn what "hairpin NAT" is.
A major hurdle is simply learning how to describe what you want to do "in the industry terms" - "I can't access my server from my computer but it works from the Internet" is a lot easier to resolve when you learn what "hairpin NAT" is.
Searching for things when you don't know the terminology seems like an ideal use for AIs.
Ahhh, it sounds like you've only done this once. I started off with ipfw, then ipchains, then iptables and now whatever firewalld supports. OK that's roughly 25 years so not too much firewalling churn! I stopped hand rolling my own rule sets with ipchains and switched to generators and there are loads of them.
For me some of the problems nowadays are caused by search engine manipulation. Up until around five or so years ago Linux concept searches would get you pointed at the usual big hitters - Arch/Gentoo/Ubuntu/etc wikis and useful and quite well known blogs. My modern block list for ublacklist is huge and barely scratches the surface.
Now I come to think of it, we now have ChatGPT and I bet it can roll a decent ruleset without hallucinating madly. No doubt someone will soon be Showing HN: their smart new firewall prompt generator language for <insert AI here>. It will make the LLM use Rust as an intermediary for extra safety.
For me some of the problems nowadays are caused by search engine manipulation. Up until around five or so years ago Linux concept searches would get you pointed at the usual big hitters - Arch/Gentoo/Ubuntu/etc wikis and useful and quite well known blogs. My modern block list for ublacklist is huge and barely scratches the surface.
Now I come to think of it, we now have ChatGPT and I bet it can roll a decent ruleset without hallucinating madly. No doubt someone will soon be Showing HN: their smart new firewall prompt generator language for <insert AI here>. It will make the LLM use Rust as an intermediary for extra safety.
I found WireGuard to actually be especially frustrating, because it doesn't really log anything. If you do a single configuration misstep the packets just won't flow. Then good luck figuring out whether it's the authentication that's wrong (or any of the cert checks in the chain), rounting, MTU, firewall or anything inbetween.
It's kinda horrible. Not that IPsec is any better. OpenVPN at least yells something at you.
It's kinda horrible. Not that IPsec is any better. OpenVPN at least yells something at you.
I'm wondering whether one can get these notifications through netlink. Not having any way to get the kinds of feedback you're mentioning from a stateful thing is horrible user experience. It doesn't need to cover the firewall part, mtu rejects, etc. Because if it's actually modular, you're supposed to be able use the usual tools (/proc/net and /sys/net for stats and netlink for firewall logs?), hopefully they're usable...
Ironically enterprise networking on Linux is incredibly simple
When you ask for help and the person responds with "It's all in the man pages."
We recently switched a bunch of stuff from OpenVPN to Wireguard. A number of the links were OpenVPN layer 2 tunnels to pass, of all things, Novell Netware running on IPX (the particular situation precludes switching to TCP/IP for those customers). Now, layer 2 tunneling is being performed using RFC 3378 EtherIP, and it's much more performant, not to mention easier to manage.
Old and new systems are running OpenBSD.
Old and new systems are running OpenBSD.
Good grief - NetWare with only IPX! Presumably you'll be installing their Y2K patches any day soon 8)
I'm pretty sure NetWare can natively tunnel IPX/SPX over TCP/IP, assuming you are not stuck on 3.12. I don't think you have to fork out for the multi protocol router thing. They will crash regularly until you get the magic combination spot on and then run forever.
A NW 6.5 box will run quite happily as a pretty tiny VM - you could scatter them around as routers to support whatever nightmare of an app you need to run over IPX. Tunnel them over IPSEC or whatever floats your boat.
I'm pretty sure NetWare can natively tunnel IPX/SPX over TCP/IP, assuming you are not stuck on 3.12. I don't think you have to fork out for the multi protocol router thing. They will crash regularly until you get the magic combination spot on and then run forever.
A NW 6.5 box will run quite happily as a pretty tiny VM - you could scatter them around as routers to support whatever nightmare of an app you need to run over IPX. Tunnel them over IPSEC or whatever floats your boat.
That particular client is stuck on NetWare 5.1 and due to how bad their maintenance has been I dare not touch the running instances. Virtualization didn't work out as there's some issue with the current patchlevel that dies on Intel CPUs greater than Pentium 4 (including hyperthreading P4s). It's...a stupid story.
On VMware you can hide CPU features - ie backrev the CPU and I'm sure HV can do the same, and no doubt KVM/QEMMU too.
I used to run a lot of NW back in the day. I remember deploying a NW 5 cluster of three Compaq boxes with six NICs each (for each VLAN) to do just DHCP/Dynamic DNS! I also ran up four HP boxes a year later with single ATM cards in them with a lot of VLANs to replace a load of 4.11 jobbies. The autoexec.ncf was a masterpiece! The cluster hosts had 6GB RAM each and despite being 32bit had quite a lot of cache which nss absolutely loved. I can't remember when NW 32 bit managed to devote >4Gb to cache but it was handy. As a file server it was absolutely unmatched. Apply an ACL and it simply worked - none of that marking subfolders and files thing that MS and Unix need. NDS/eDir was streets ahead of that weird LDAP n Kerberos thingie that MS "invented" and frankly still is.
I would suggest that you virty them as a matter of urgency. Presumably you have limited and dwindling hardware resources available and at some point something will go pop that can't be fixed. Once you have them as VMs then you can snapshot and all the other lovely things that virty brings to the game and you will never run out of hardware!
I used to run a lot of NW back in the day. I remember deploying a NW 5 cluster of three Compaq boxes with six NICs each (for each VLAN) to do just DHCP/Dynamic DNS! I also ran up four HP boxes a year later with single ATM cards in them with a lot of VLANs to replace a load of 4.11 jobbies. The autoexec.ncf was a masterpiece! The cluster hosts had 6GB RAM each and despite being 32bit had quite a lot of cache which nss absolutely loved. I can't remember when NW 32 bit managed to devote >4Gb to cache but it was handy. As a file server it was absolutely unmatched. Apply an ACL and it simply worked - none of that marking subfolders and files thing that MS and Unix need. NDS/eDir was streets ahead of that weird LDAP n Kerberos thingie that MS "invented" and frankly still is.
I would suggest that you virty them as a matter of urgency. Presumably you have limited and dwindling hardware resources available and at some point something will go pop that can't be fixed. Once you have them as VMs then you can snapshot and all the other lovely things that virty brings to the game and you will never run out of hardware!
Pretty cool to hear about big NW installs! Thanks for sharing!
We're actively working on moving that customer off Netware entirely. The main reason they can't get away from it is they're running a custom god program that manages the entire business, and it's tied in pretty tightly with their old configuration. The whole thing is in Delphi 7.
W.R.T. old hardware, they are actually some of the newer old boxes we support! My main line of business is keeping old industrial control systems online and reliable. On the PC side, the oldest stuff we have in 24/7 operation is 286-based. Pre-PC, we have a few customers running CNC stuff on PDP-11s. I don't know if we still have any PDP-8 customers, I think most of them closed up shop during the pandemic.
We're actively working on moving that customer off Netware entirely. The main reason they can't get away from it is they're running a custom god program that manages the entire business, and it's tied in pretty tightly with their old configuration. The whole thing is in Delphi 7.
W.R.T. old hardware, they are actually some of the newer old boxes we support! My main line of business is keeping old industrial control systems online and reliable. On the PC side, the oldest stuff we have in 24/7 operation is 286-based. Pre-PC, we have a few customers running CNC stuff on PDP-11s. I don't know if we still have any PDP-8 customers, I think most of them closed up shop during the pandemic.
Good skills.
You beat our DOS 5 booting control system into a cocked hat, that a customer is running. You've got to love cough technical debt. Why on earth is "Enterprise gone awry" considered the right way to go for your IT strategy over that boring old open source bollocks that has a nasty habit of still being supported or at least working decades later? To be fair, open source wasn't a thing in the 80s for most people (nor the 90s ... ).
PDP-11s are my uncle's era and I'm 53! My first real PC was a 80286 (don't forget the 80) I bought a '287 maths co pro for about £115 so I could run a dodgy copy of AutoCAD on it. I do still have my old Commodore 64 which now has a USB interface. I assembled a ZX80 or two ...
Delphi is "just" Pascal - port it!
You beat our DOS 5 booting control system into a cocked hat, that a customer is running. You've got to love cough technical debt. Why on earth is "Enterprise gone awry" considered the right way to go for your IT strategy over that boring old open source bollocks that has a nasty habit of still being supported or at least working decades later? To be fair, open source wasn't a thing in the 80s for most people (nor the 90s ... ).
PDP-11s are my uncle's era and I'm 53! My first real PC was a 80286 (don't forget the 80) I bought a '287 maths co pro for about £115 so I could run a dodgy copy of AutoCAD on it. I do still have my old Commodore 64 which now has a USB interface. I assembled a ZX80 or two ...
Delphi is "just" Pascal - port it!
The bad part is mostly the DB and the Netware integration, but mostly the DB. It's Borland Paradox.
If you have a write up of how you managed to get layer 2 working inside wireguard, I'd love to read it.
I basically just followed the OpenBSD documentation! One of the big advantages of OpenBSD is that pretty much everything you need to know is contained in the manpages.
As I'd said above, we ended up using RFC 3378 EtherIP to link the two layer 2 broadcast domains across the Wireguard tunnel. OpenBSD supports this with the etherip interface. You end up creating a bridge with the etherip interface and whatever physical Ethernet interfaces you want to bridge, on either side of the Wireguard tunnel.
I also tried VXLAN but did not have good results. I'm not entirely sure it wasn't a problem with my configuration. Traffic often went one-directional, where broadcast packets from Site A made it to Site B, but they did not come from Site B to Site A. EtherIP worked right off, so I didn't investigate further.
As I'd said above, we ended up using RFC 3378 EtherIP to link the two layer 2 broadcast domains across the Wireguard tunnel. OpenBSD supports this with the etherip interface. You end up creating a bridge with the etherip interface and whatever physical Ethernet interfaces you want to bridge, on either side of the Wireguard tunnel.
I also tried VXLAN but did not have good results. I'm not entirely sure it wasn't a problem with my configuration. Traffic often went one-directional, where broadcast packets from Site A made it to Site B, but they did not come from Site B to Site A. EtherIP worked right off, so I didn't investigate further.
Use a GRETAP interface; Red Hat's virtual interface documentation is phenomenal:
https://developers.redhat.com/blog/2019/05/17/an-introductio...
https://developers.redhat.com/blog/2019/05/17/an-introductio...
The docs are indeed great, but to me it seems like they are recommending GENEVE (RFC 8926):
> Generic Network Virtualization Encapsulation (GENEVE) supports all of the capabilities of VXLAN, NVGRE, and STT and was designed to overcome their perceived limitations. Many believe GENEVE could eventually replace these earlier formats entirely
I'm bit surprised that they didn't have section on vxlan there considering it is pretty popular afaik?
Anyways, I think tunneling GENEVE (or any other Ethernet-over-IP protocol) should work fine over WireGuard, same as using regular network interfaces.
> Generic Network Virtualization Encapsulation (GENEVE) supports all of the capabilities of VXLAN, NVGRE, and STT and was designed to overcome their perceived limitations. Many believe GENEVE could eventually replace these earlier formats entirely
I'm bit surprised that they didn't have section on vxlan there considering it is pretty popular afaik?
Anyways, I think tunneling GENEVE (or any other Ethernet-over-IP protocol) should work fine over WireGuard, same as using regular network interfaces.
https://developers.redhat.com/blog/2018/10/22/introduction-t... vxlan and many others are here
Yes! I definitely agree. I've used GRETAP for L2 over Wireguard in the past, but it was quite a while ago. GENEVE looks like the way to go these days
Since WireGuard is Layer 3, what would is everyone's use case of doing Layer 2 on it? Or, what can it improve over existing solutions? I have tried to do the same for a bit while still learning networking, but ran into Layer 3 limitations.
People who require layer 2 require either a protocol which is neither TCP nor UDP or they need devices in the same broadcast domain
Can one do multicast over wireguard?
Frustratingly enough, apparently not as I could never get it to work. It is pretty easy to set up a vxlan tunnel over wireguard if you absolutely need stuff like that though.
Oh hadn't thought about that, thanks. 'need' is a big word here but sometimes you can't change the client and server apps so, having support for the basic (although niche) features in the lower layers helps migrating smoothly.
Probably the most common use case is letting Avahi/Bonjour/etc. or DHCP work across a tunnel.
Bonjour is built on top of DNS. You don't need a layer 2 tunnel to make it work.
However, it normally does rely on multicast. Rather than trying to bridge broadcast domains (which is going to cause performance issues), a more efficient option is to setup an Avahi mDNS reflector on either end of the tunnel to rebroadcast mDNS packets.
See, for example: https://www.reddit.com/r/WireGuard/comments/g80bxf/comment/h...
Alternatively, there's also a Wide-Area Bonjour service that works over unicast and doesn't need any special packet forwarding, provided you run a Bonjour-aware DNS server:
http://www.dns-sd.org/serversetup.html
https://help.dyn.com/bonjour-and-dns-discovery/
(More generally, Layer 2 tunnels are best avoided unless you really need them for something arcane, like IPX or NetBIOS.)
However, it normally does rely on multicast. Rather than trying to bridge broadcast domains (which is going to cause performance issues), a more efficient option is to setup an Avahi mDNS reflector on either end of the tunnel to rebroadcast mDNS packets.
See, for example: https://www.reddit.com/r/WireGuard/comments/g80bxf/comment/h...
Alternatively, there's also a Wide-Area Bonjour service that works over unicast and doesn't need any special packet forwarding, provided you run a Bonjour-aware DNS server:
http://www.dns-sd.org/serversetup.html
https://help.dyn.com/bonjour-and-dns-discovery/
(More generally, Layer 2 tunnels are best avoided unless you really need them for something arcane, like IPX or NetBIOS.)
You are technically correct (best kind of correct) however, in reality, I see folks using L2 tunnels to solve for bonjour etc all the time. Usually those without networking knowledge to solve the forwarding.
Yeah, you can do it the right way...or you can just tunnel layer 2 and forget about it. I see it done a fair bit for both Bonjour/Avahi and DHCP (why?).
One example usecase would be to try to tunnel something like BOOTP/DHCP/PXE/TFTP stack, which iirc is bit tricky with only L3 tunneling.
Yes, this is common among those who don't understand how DHCP Relay works
Many don't, and I suspect searching for "how to do on VPN" yields "turn on Layer 2 tunneling."
The best way to perform something like this on Layer 2 is to use Shortest Path Bridging (SPB) based on IEEE 802.1Q-2018. However the Linux kernel does not yet fully supporting this feature natively although the standard has been out for quite sometime and already being supported by commercial network solutions and the popular Open vSwitch (OVS) [1].
[1] Ask HN: Project ideas for a Linux kernel module:
https://news.ycombinator.com/item?id=35785158
[1] Ask HN: Project ideas for a Linux kernel module:
https://news.ycombinator.com/item?id=35785158
SPB uses MAC-in-MAC encapsulation which won't help you run over Wireguard. You need some flavor of Ethernet over IP like GRE, VXLAN, or GENEVE.
gretap.
But why run Wireguard+gretap when you could just run tap-mode OpenVPN?
But why run Wireguard+gretap when you could just run tap-mode OpenVPN?
Me too.
[deleted]
Meta: Huh. I don’t use “shallow dive” enough. “Deep dive” of course, everyone loves a good deep dive. But what about a shallow dive, or even just “getting your feet wet”? These are useful concepts too. This headline alone revealed a blind spot for me.
Had the same thought. Could do with more shallow dives to be honest.
Maybe we can expand X in Y minutes beyond programming languages?
> We will be using the local IPv6 addresses fc00::/7 for routing inside our WireGuard network. For example, if we use the subnet fc00:0:0:160::/64, this can be the new Address config on Server:
> Address = 192.168.160.2, fc00:0:0:160::2
Please don't do this. fc00::/7 has been assigned for "Unique Local Unicast" and address ranges must be generated as specified in RFC 4193: https://www.google.com/search?hl=en&q=ipv6%20ula%20generator
> Address = 192.168.160.2, fc00:0:0:160::2
Please don't do this. fc00::/7 has been assigned for "Unique Local Unicast" and address ranges must be generated as specified in RFC 4193: https://www.google.com/search?hl=en&q=ipv6%20ula%20generator
Or instead you can have HTTP proxy over TLS in just four steps: https://github.com/Snawoot/dumbproxy/wiki/Quick-deployment
You don't even need a client for this, any modern browser can work with it right away: https://github.com/Snawoot/dumbproxy#using-http-over-tls-pro...
You don't even need a client for this, any modern browser can work with it right away: https://github.com/Snawoot/dumbproxy#using-http-over-tls-pro...
I just moved from OpenVPN to tailscale, which uses Wireguard, on my personal stuff. I have a similar situation as OP describes at first where my residential account has the ports blocked.
I am quite happy so far, just wish it was innately supported in my consumer grade router, which support vanilla wireguard.
I am quite happy so far, just wish it was innately supported in my consumer grade router, which support vanilla wireguard.
I thought tailscale works through all kinds of firewalls due to some connection setup magic (initiating connection from both sides at once).
EDIT: I think I misunderstood your comment, you probably are wishing for tailscale client support in the router?
EDIT: I think I misunderstood your comment, you probably are wishing for tailscale client support in the router?
>consumer grade router, which support vanilla wireguard.
See if your consumer grade router supports flashing OpenWRT. It supports Wireguard.
See if your consumer grade router supports flashing OpenWRT. It supports Wireguard.
It doesn't, really. It's an AX-82 and there's only a hackish version for it.
Dumb question but the frp example shows op forwarding to port 443 on local, how then did the spammers get access to port 25 or was there a separate forward rule setup for 25?
i agree it's very hard to learn from the internet.. (the bottomline in the post)
It was incredibly so much easier when popular search engines did search instead of "recommendation." The hardest thing is actually finding the thing that exists, in multiple forms, that you set out to find.
We've gone backwards in the past 15 years.
We've gone backwards in the past 15 years.
signal-to-noise ratio: off the charts.
Many thanks, OP.
Many thanks, OP.
nice
Link was probably supposed to be https://im.salty.fish/index.php/archives/linux-networking-sh....
tql
UPS and Downs a chinesse programmer should do.
Hosting is prohibited in CN
It would be interesting to make a Github repo titled: how to hack CN goverment !
It would be interesting to make a Github repo titled: how to hack CN goverment !
For people using wireguard, it was not designed to provide anonymity. Otherwise it is fine for use in Countries with decent protections for their citizens. If you need privacy, you should use OpenVPN.
Quote:
>WireGuard is highly secure, but it’s not designed with privacy in mind.
from
https://www.tomsguide.com/how-to/is-the-new-wireguard-protoc...
Quote:
>WireGuard is highly secure, but it’s not designed with privacy in mind.
from
https://www.tomsguide.com/how-to/is-the-new-wireguard-protoc...
> WireGuard is highly secure, but it’s not designed with privacy in mind.
I'm sorry, but I must inform you that the Toms guide contains affiliate links to OpenVPN services. However, it is important to note that neither OpenVPN nor WireGuard can guarantee your safety if you are being targeted by government agencies. The guide's attempt from TFA is to promote these VPN services as a solution for anonymity and censorship (deep packets inspection can block all VPN protocols) avoidance is misleading. VPNs are primarily useful for accessing corporate or home resources and viewing geo-blocked streaming content (say from your home network) on insecure networks like hotel or cafe WiFi.
I'm sorry, but I must inform you that the Toms guide contains affiliate links to OpenVPN services. However, it is important to note that neither OpenVPN nor WireGuard can guarantee your safety if you are being targeted by government agencies. The guide's attempt from TFA is to promote these VPN services as a solution for anonymity and censorship (deep packets inspection can block all VPN protocols) avoidance is misleading. VPNs are primarily useful for accessing corporate or home resources and viewing geo-blocked streaming content (say from your home network) on insecure networks like hotel or cafe WiFi.
Parts of this article are just downright wrong
At time of writing, the biggest privacy weakness that WireGuard has is how it assigns IP addresses. When you connect to a VPN service using OpenVPN or IKEv2, you’re assigned a different IP address each time. WireGuard instead gives you the same IP address each time. This is faster, but it means the VPN server must keep logs of your real IP address and connection timestamps.
The address assigned inside the tunnel has nothing to do with your real address, and definitely does not have anything to do with whether or not the VPN server is keeping logs of your real IP address and timestamps of your connection.
OpenVPN and charon keep far more logs of those things by default that wireguard and you have to trust your VPN provider turned them off.
At time of writing, the biggest privacy weakness that WireGuard has is how it assigns IP addresses. When you connect to a VPN service using OpenVPN or IKEv2, you’re assigned a different IP address each time. WireGuard instead gives you the same IP address each time. This is faster, but it means the VPN server must keep logs of your real IP address and connection timestamps.
The address assigned inside the tunnel has nothing to do with your real address, and definitely does not have anything to do with whether or not the VPN server is keeping logs of your real IP address and timestamps of your connection.
OpenVPN and charon keep far more logs of those things by default that wireguard and you have to trust your VPN provider turned them off.
Almost nothing was created with privacy in mind. Security and privacy are different things.
I hate that people think that a VPN is private as in anonymous. But then again, those providers had great marketing.. So now devs and sysops need to call VPNs "tunneled networks".
I hate that people think that a VPN is private as in anonymous. But then again, those providers had great marketing.. So now devs and sysops need to call VPNs "tunneled networks".
You're not wrong, but there are VPN services that add on privacy to their wireguard offerings, such as PIA (private internet access). They open sourced the connection code so you can see how they do it[1] using an API that initializes a temporary wireguard connection for you. I've been really pleased with PIA's wireguard setup, which even includes forwarding of an incoming port!
[1]: https://github.com/pia-foss/manual-connections/blob/master/c...
[1]: https://github.com/pia-foss/manual-connections/blob/master/c...
...
>It is extremely frustrating when somebody interested in setting up their own network infrastructure has to at some point get stuck at some convoluted networking concepts, intricate and abstract tools, mysterious errors here and there, or lack of systematic documentation. I wish everyone has some choices other than spending days and weeks trying to figure these out alone, so I decided to write down what I have done, what I have learned and what I have to share with the rest of the internet. I sincerely hope that some day IT operations would be more beginner-friendly, and hosting one's own network infrastructure no longer means headache and mess.
These are exactly my feelings. Stinky.fish makes more sense for a Linux blog. Everything about it stinks until it actually works. :)