OpenBSD Router Guide (2020)(openbsdrouterguide.net)
openbsdrouterguide.net
OpenBSD Router Guide (2020)
https://openbsdrouterguide.net/
40 comments
0. Accept that for NAT on home gateway you really don't need wirespeed at minimal MTU, as the majority of your data transfers will be over TCP at max MSS/MTU.
1. Get a Xeon D box (like a Supermicro X10SDV-TLN4F, TDP 45W).
2. Install any Linux distribution on it and just use iptables/nftables. It's good enough.
3. You now have a normal Linux box with normal userspace networking, feel free to also run some more services on it without having to deal with DPDK getting in your way.
DPDK is nice and all if you absolutely need to scream through tiny packets at multi-dozen-gigabit speeds, but you almost certainly don't need it at home for Netflix and seeding Ubuntu ISOs.
1. Get a Xeon D box (like a Supermicro X10SDV-TLN4F, TDP 45W).
2. Install any Linux distribution on it and just use iptables/nftables. It's good enough.
3. You now have a normal Linux box with normal userspace networking, feel free to also run some more services on it without having to deal with DPDK getting in your way.
DPDK is nice and all if you absolutely need to scream through tiny packets at multi-dozen-gigabit speeds, but you almost certainly don't need it at home for Netflix and seeding Ubuntu ISOs.
> 0. Accept that for NAT on home gateway you really don't need wirespeed at minimal MTU, as the majority of your data transfers will be over TCP at max MSS/MTU.
This can't be true for video gamers? I'd assume most video game protocols are tiny UDP packets.
But maybe video game protocols don't need "wirespeed", since they're actually quite efficient in terms of bytes transferred. But you want to minimize latency.
This can't be true for video gamers? I'd assume most video game protocols are tiny UDP packets.
But maybe video game protocols don't need "wirespeed", since they're actually quite efficient in terms of bytes transferred. But you want to minimize latency.
It's not needed for games, they use a few megabits at most. We use the wirespeed term in the context of networking devices when discussing their limitations. If a device is wirespeed it's capable of routing or switching on all of its interfaces using the smallest packet size, for a 10Gbit interface that is 14.8m packets per second in one direction.
I don't think any games rely on more than a few dozen megabits of Internet bandwidth at a few hundred packets per second. And that netfilter/iptables NAT on a Xeon is not gonna add more latency than the equivalent netfilter/iptables NAT that runs on a typical ARM/MIPS home gateway/router.
I've found a few open source NAT implementations on DPDK, but several appears to be reference/demo implementations. This project looks interesting: https://vigor-nf.github.io/
DPDK network HBAs have been out for several years now and 25/50/100 is replacing 10 in the datacenter. Maybe there's some deals available on ebay? Quick glance, an Intel X520 is available for ~90 USD.
DPDK network HBAs have been out for several years now and 25/50/100 is replacing 10 in the datacenter. Maybe there's some deals available on ebay? Quick glance, an Intel X520 is available for ~90 USD.
VPP (http://fd.io) NAT plugins are used in various production deployments.
Various NAT-related perf tests: https://docs.fd.io/csit/master/report/vpp_performance_tests/...
Disclaimer: I am involved with the project
Various NAT-related perf tests: https://docs.fd.io/csit/master/report/vpp_performance_tests/...
Disclaimer: I am involved with the project
[deleted]
Any old gaming PC with 10G NICs from ebay will do the trick. Install any Linux distro, do some basic config and you're done. Wire speed with the smallest packet size isn't required, you'll probably get at least a few million pps which should be enough.
Or ubiquity ER-8-XG.
Or ubiquity ER-8-XG.
Vanilla Linux/BSD isn't bad at 10G these days but if you want more performance out of less hardware, TNSR is a productized DPDK/VPP router: https://www.tnsr.com/
Neat!
I already have an OpenBSD firewall running on a PC Engines APU 2. Despite this, I still learned a few things from this guide (namely that urpf-failed already handles antispoof and that scrub shouldn't just be littered without any consideration).
I was actually impressed at how well things perform despite the BKL of OpenBSD. I have only a 350Mbps down 35Mbps up connection and OpenBSD is nowhere near being a bottleneck.
I already have an OpenBSD firewall running on a PC Engines APU 2. Despite this, I still learned a few things from this guide (namely that urpf-failed already handles antispoof and that scrub shouldn't just be littered without any consideration).
I was actually impressed at how well things perform despite the BKL of OpenBSD. I have only a 350Mbps down 35Mbps up connection and OpenBSD is nowhere near being a bottleneck.
Conversely, I tried running openbsd as a router on my ubiquiti edgerouter for my 950 megabit up/down connection and couldn't get even close to the performance of even the shitty ISP-provided router/ONT combo box.
As someone who had considered a similar setup, what model of edgerouter? Would you consider doing a writeup? What kind of speeds did you achieve?
I get line speed with the Ubiquiti software installed.
Here's a writeup I did last year but it doesn't touch on OpenBSD: https://0x85.org/ubiquiti.html
I'm mostly over BSD in general. I don't know why, given that Debian exists and performs much better, I'd reach for BSD unless it had support for some esoteric hardware that Linux doesn't support.
Here's a writeup I did last year but it doesn't touch on OpenBSD: https://0x85.org/ubiquiti.html
I'm mostly over BSD in general. I don't know why, given that Debian exists and performs much better, I'd reach for BSD unless it had support for some esoteric hardware that Linux doesn't support.
My favourite is currently NixOS for a router, entire configuration (system packages, version, firewall rules, interfaces, VLANs, etc) is declarative and can be deployed with automatic rollbacks in the event of misconfiguration.
This is what I’m working on right now too, built around an APU board. I’m also wrestling with hostapd to get WiFi working as well, just because. Once it’s working I’ll try and write it up.
I also love this. I ran into issues trying to get it working with extra very strict kernel security features enabled, but even then I eventually figured it out. Was really nice to use.
That sounds very interesting. Have you documented it somewhere?
Here’s a really simple config I started with (not mine, just inspiration):
https://github.com/jgillich/nixos/blob/master/roles/router.n...
I spent some time a while back using this guide, along with some ansible to setup a home router. I ended up putting it on the back burner for a few reasons.
1) DNS reflection was an issue for some internal services I host. 2) Connecting to my Virgin Media (uk) router and handling PPPoE was confusing and I struggled with the documentation. 3) Just in general, having an internet connection is really importaint to troubleshooting problems. Having my internet down while trying to fix things was just too inconvenient for me.
In the end, I went back to PfSense. I have a bash script that backs up the config daily to s3. Was good enough to recover from a different issue recently. It's a real shame, I'd love to do this all my self but I may never pick this project back up.
1) DNS reflection was an issue for some internal services I host. 2) Connecting to my Virgin Media (uk) router and handling PPPoE was confusing and I struggled with the documentation. 3) Just in general, having an internet connection is really importaint to troubleshooting problems. Having my internet down while trying to fix things was just too inconvenient for me.
In the end, I went back to PfSense. I have a bash script that backs up the config daily to s3. Was good enough to recover from a different issue recently. It's a real shame, I'd love to do this all my self but I may never pick this project back up.
What Virgin Media modem+router are you using?
I have a Superhub 3.0. In modem-only mode there's no PPPoE, it just rather transparently acts like you have an ethernet cable running to your ISP. Just putting dhcp in my hostname.em0 works exactly as intended and my router gets assigned the public IP address.
I have a Superhub 3.0. In modem-only mode there's no PPPoE, it just rather transparently acts like you have an ethernet cable running to your ISP. Just putting dhcp in my hostname.em0 works exactly as intended and my router gets assigned the public IP address.
I did exactly that last week with Debian 11. I got tired of IPFire (a linux distribution for routers, like pfSense). Since my only connectivity to the Internet is LTE, the most challenging part was bootstrapping the system without an Internet connection, after that it went very smoothly.
I also had some issues with PPPoE, for some reason it did not worked for me, then suddenly it started to work with the very same config. That was very weird, but it still works fine.
shark# cat /etc/hostname.pppoe0
inet 0.0.0.0 255.255.255.255 NONE \
pppoedev em0 authproto chap \
authname '<username>' authkey '<my_password>' up
dest 0.0.0.1
!/sbin/route delete default
!/sbin/route add default -ifp pppoe0 0.0.0.1
shark# cat /etc/hostname.em0
up
shark#PfSense is not a bad fall-back IMHO (^_^)
Or OPNSense.
moved from Pfsense to Opnsense, never looking back
What were the compelling reasons for you to switch? Was it something Opnsense did better, or did Netgate just piss you off?
I'm running pfsense now because I'm nothing if not lazy, but if opnsense makes things nicer in some way, I'd like to hear it!
I'm running pfsense now because I'm nothing if not lazy, but if opnsense makes things nicer in some way, I'd like to hear it!
OpenBSD on the APU2 has been the most reliable, solid, and lowest maintenance router I have ever owned. The thing never needs to be restarted, never hiccups, and just is really rock solid. Highly recommend, just wish there was a good wireless AP solution too.
Anyone know similar up-to-date comprehensive guide for Linux?
This book by Tony Mancill used to be an excellent guide for Linux routers but now after 20 years it is already obselete [1].
[1]https://www.amazon.com/Linux-Routers-Primer-Network-Administ...
This book by Tony Mancill used to be an excellent guide for Linux routers but now after 20 years it is already obselete [1].
[1]https://www.amazon.com/Linux-Routers-Primer-Network-Administ...
Is OpenBSD still largely single-threaded or have there been SMP improvements in the network stack over the years?
The feature set OpenBSD has is impressive, but is there a large gap in networking perf compared to Linux/FreeBSD?
From my $work computer:
$ uname -a
OpenBSD XXXXX 6.9 GENERIC.MP#3 amd64
$top
load averages: 0.83, 0.62, 0.44
94 processes: 93 idle, 1 on processor
CPU00 states: 0.5% user, 0.0% nice, 0.2% sys, 0.3% spin, 0.6% intr, 98.4% idle
CPU02 states: 6.5% user, 0.0% nice, 1.9% sys, 0.4% spin, 0.0% intr, 91.1% idle
CPU04 states: 4.9% user, 0.0% nice, 1.6% sys, 0.3% spin, 0.0% intr, 93.1% idle
CPU06 states: 4.0% user, 0.0% nice, 1.1% sys, 0.3% spin, 0.0% intr, 94.6% idle
CPU08 states: 1.4% user, 0.0% nice, 0.6% sys, 0.2% spin, 0.0% intr, 97.8% idle
CPU10 states: 1.4% user, 0.0% nice, 0.5% sys, 0.2% spin, 0.0% intr, 98.0% idle
CPU12 states: 0.6% user, 0.0% nice, 0.3% sys, 0.1% spin, 0.0% intr, 99.0% idle
CPU14 states: 0.7% user, 0.0% nice, 0.3% sys, 0.1% spin, 0.0% intr, 98.8% idle
Memory: Real: 3292M/5988M act/tot Free: 9109M Cache: 1692M Swap: 0K/15GThe person you're responding to was asking about network performance. I imagine specifically regarding pf (packet filter, the firewall component).
To the grandparent: PF is still single threaded. If you had performance issues with that before you may still have them, but CPU improvements over time may have negated that impact. It's worth trying it out again.
To the grandparent: PF is still single threaded. If you had performance issues with that before you may still have them, but CPU improvements over time may have negated that impact. It's worth trying it out again.
Building a OpenBSD router has always been one of those projects in the back of my head but just never had the time. I plan on upgrading my home network in the next year and am considering MikroTik / RouterOS or a Ubiquity router. Anyone have experience with any of those and a custom OpenBSD build? What would be the advantage of going OpenBSD and is it worth it for your setup?
Are there any open-source, "next-generation" firewalls, running on OpenBSD?
SD-WAN, DLP, Application-aware filtering, etc
SD-WAN, DLP, Application-aware filtering, etc
This is an interesting question.
One would think, since OpenBSD is BSD licensed, it would be more likely to have vendorized forks available on the marketplace. Companies being obliged to share their code is allegedly a disadvantage of GPL-licenced Linux. But with OpenBSD, there are no prominent commercial forks. The one "active" OpenBSD fork today is HyperbolaBSD, and it hasn't made a single release yet after 2 years. Previous attempts at OpenBSD forks have failed (Bitrig, ÆrieBSD). Somehow, OpenBSD appears to be less fork-able than other BSDs.
Why does the OpenBSD project subvert our expectations of a BSD-licenced OS?
I'm asking as someone who had a positive experience setting up an OpenBSD router and VPN, but not much beyond that.
One would think, since OpenBSD is BSD licensed, it would be more likely to have vendorized forks available on the marketplace. Companies being obliged to share their code is allegedly a disadvantage of GPL-licenced Linux. But with OpenBSD, there are no prominent commercial forks. The one "active" OpenBSD fork today is HyperbolaBSD, and it hasn't made a single release yet after 2 years. Previous attempts at OpenBSD forks have failed (Bitrig, ÆrieBSD). Somehow, OpenBSD appears to be less fork-able than other BSDs.
Why does the OpenBSD project subvert our expectations of a BSD-licenced OS?
I'm asking as someone who had a positive experience setting up an OpenBSD router and VPN, but not much beyond that.
Maybe because the founder and team is known for zero tolerance toward BS, gives a shit about percieved 'needs' of shitty 'bizniz', and drama.
What is a "vendorized fork" ?
There are companies that build commercial (virtual) firewall appliances based on OpenBSD. But they do not publish an open source, BSD-licensed, operating system.
There are companies that build commercial (virtual) firewall appliances based on OpenBSD. But they do not publish an open source, BSD-licensed, operating system.
FreeBSD underpins a fair number of commercial producers. Junos from Juniper perhaps being the most famous.
https://en.m.wikipedia.org/wiki/List_of_products_based_on_Fr...
https://en.m.wikipedia.org/wiki/List_of_products_based_on_Fr...
I don't know of any big ones that are still going. OPNSense runs on HardenedBSD (a FreeBSD fork) and is excellent. FreeBSD and/or NetBSD is/was used by Juniper, Sandvine, Dell (Force10).
I think the main blocker for OpenBSD in the commercial market is the inability to load binary blob drivers without maintaining a custom kernel.
I think the main blocker for OpenBSD in the commercial market is the inability to load binary blob drivers without maintaining a custom kernel.
Next release of OPNSense will be rebased on 'stock' FreeBSD.
The current release is the last to be based on HardenedBSD.
This is known since April https://forum.opnsense.org/index.php?topic=22761.0 and confirmed there https://forum.opnsense.org/index.php?topic=24112.0
This is known since April https://forum.opnsense.org/index.php?topic=22761.0 and confirmed there https://forum.opnsense.org/index.php?topic=24112.0
Those of us building our own routers (I've done so since about 2006 when I got 1Gbps fibre installed for the first time) usually have a quite large cross-section with people on bleeding edge internet connections, and with (pro-)consumer 10Gbps internet and networking becoming more and more common a router capable of such is also more relevant. I see very little information about what's the minimal/cost effective hardware requirements capable of routing with NAT at wirespeed for these applications. Most of the content I read is either geared towards slower speeds (e.g. 1Gbps) or using overly beefy machines which are both power hungry and most likely overkill for the purpose.
Something that would be interesting to see as well is high speed NAT routing performance comparison between Linux, NetBSD, OpenBSD and FreeBSD using Intel or other well supported 10Gbps nics.