VyOS From Scratch – Edition 1 (set up your own router)(blog.kroy.io)
blog.kroy.io
VyOS From Scratch – Edition 1 (set up your own router)
https://blog.kroy.io/2020/05/04/vyos-from-scratch-edition-1/
84 comments
It's a fork vyatta[1], the same system used by Ubiquiti for EdgeOS.
Yes, it's Debian with a declarative configuration system. It works more or less similarly to NixOS, if you know about that: basically it uses a bunch of perl scripts to install and set up software from a single unified configuration file, hiding all the implementation details.
[1]: https://en.wikipedia.org/wiki/Vyatta
[1]: https://en.wikipedia.org/wiki/Vyatta
And it works great because your router starts from a read-only image, then executes the scripts and applies the configuration. You can easily roll back and forward configuration and firmware versions.
I don't think they use Perl anymore. They have been moving to Python.
Had no idea vyatta is what Ubiquiti uses. Very cool.
Thats only on EdgeOS though. UbiquitiOS uses a different system, built around Podman and Debian
It's a Network Operating System for configuring routers & switches (implemented on top of Debian). This means:
* You can SSH in and configure it like you would a managed switch or router. There's a single object that models all of the device's configuration, when you commit it a bunch of scripts activate and actaully apply the configuration to the running system.
* Deployments are image based, you can roll back to older images etc.
* You don't need to look at any of the system's underlying configuration files or use any of the normal Linux commands to examine and manipulate the state of the system (the commands are still there for convenience of course). You don't even need to be aware that you're really using a bunch of custom bash functions to examine and manipulate the state of the system.
* You can SSH in and configure it like you would a managed switch or router. There's a single object that models all of the device's configuration, when you commit it a bunch of scripts activate and actaully apply the configuration to the running system.
* Deployments are image based, you can roll back to older images etc.
* You don't need to look at any of the system's underlying configuration files or use any of the normal Linux commands to examine and manipulate the state of the system (the commands are still there for convenience of course). You don't even need to be aware that you're really using a bunch of custom bash functions to examine and manipulate the state of the system.
Yes - it's a Debian Linux based router distro that is at least partially modelled after Juniper's configuration style.
edit: used to be a maintainer for a short while :)
edit: used to be a maintainer for a short while :)
It's a Debian 8 base with a lot of its own custom packages. The homepage has an FAQ that tries to reassure you that being based on Debian 8 is not a problem. Whether that convinces you or not is up to you, of course.
I personally use regular Debian 12 on my router without problems. It also has "declarative config" since all the configuration, firewall rules, etc are a bunch of config files that I can scp / ansible over any time.
I personally use regular Debian 12 on my router without problems. It also has "declarative config" since all the configuration, firewall rules, etc are a bunch of config files that I can scp / ansible over any time.
There's incorrect comments in this thread re: VyOS, Vyatta, EdgeOS.
Vyatta is the original OS, based on Debian, dating back to 2005. Its history is detailed here:
https://en.wikipedia.org/wiki/Vyatta
In 2011, Ubiquiti launched their EdgeMax products with EdgeOS which was a fork of Vyatta Core 6.3 ported from x86 to Cavium.
In 2012, Vyatta was acquired by Brocade.
IN 2013, Vyatta Core 6.6 was forked as VyOS.
That's the rough origin of these three OSes.
I used Vyatta Core on a PC at a startup from 2009-2013 as our office router. I haven't paid attention to it or VyOS since then.
I've been running various EdgeOS routers at my home since 2014 or so, first an EdgeRouter Lite and today an EdgeRouter 4.
EdgeOS has been updated quite a bit over the years from its Vyatta Core origins, but the original developers are no longer with Ubiquiti. EdgeOS hasn't seen updates in quite some time now.
Also, not all Ubiquiti run EdgeOS. Only the EdgeRouters do. The rest of their products run a completely different OS, generally either UbiquitiOS or UnifiOS.
Sources besides my own memory:
https://blog.vyos.io/versions-mystery-revealed
https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happe...
Vyatta is the original OS, based on Debian, dating back to 2005. Its history is detailed here:
https://en.wikipedia.org/wiki/Vyatta
In 2011, Ubiquiti launched their EdgeMax products with EdgeOS which was a fork of Vyatta Core 6.3 ported from x86 to Cavium.
In 2012, Vyatta was acquired by Brocade.
IN 2013, Vyatta Core 6.6 was forked as VyOS.
That's the rough origin of these three OSes.
I used Vyatta Core on a PC at a startup from 2009-2013 as our office router. I haven't paid attention to it or VyOS since then.
I've been running various EdgeOS routers at my home since 2014 or so, first an EdgeRouter Lite and today an EdgeRouter 4.
EdgeOS has been updated quite a bit over the years from its Vyatta Core origins, but the original developers are no longer with Ubiquiti. EdgeOS hasn't seen updates in quite some time now.
Also, not all Ubiquiti run EdgeOS. Only the EdgeRouters do. The rest of their products run a completely different OS, generally either UbiquitiOS or UnifiOS.
Sources besides my own memory:
https://blog.vyos.io/versions-mystery-revealed
https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happe...
My ER4 has been a very solid piece of kit.
I was pretty confused what it is too and then I loved it.
It's debian plus some shell trickery and CLI tools that let you configure debian and debian packages as a router from one large config tree using neat CLI tools (that support commit/rollback).
Normally you'd need iptables, a separate DNS package, DHCP server, etc etc to set up a router, with VyOS you just change VyOS config and it configures normal debian packages for you.
Plus everything is exhaustively tested and configs are reverse compatible, hiding all breaking changes underneath.
It's super neat and it works perfectly on a £100 fanless Celeron J4125 box from Aliexpress as a home router, routing and shaping 1gbit without breaking a sweat and with deeply sub-ms delay.
It's debian plus some shell trickery and CLI tools that let you configure debian and debian packages as a router from one large config tree using neat CLI tools (that support commit/rollback).
Normally you'd need iptables, a separate DNS package, DHCP server, etc etc to set up a router, with VyOS you just change VyOS config and it configures normal debian packages for you.
Plus everything is exhaustively tested and configs are reverse compatible, hiding all breaking changes underneath.
It's super neat and it works perfectly on a £100 fanless Celeron J4125 box from Aliexpress as a home router, routing and shaping 1gbit without breaking a sweat and with deeply sub-ms delay.
This is probably the best explanation I've seen.
Do you have an idea why the CLI tools aren't distributed independently? Why shouldn't I be able to run it on a Debian system I already have (and understand)?
Running an entire new distro just seems like overkill for what it actually does over a normal Linux system. It's just a configuration manager!
Do you have an idea why the CLI tools aren't distributed independently? Why shouldn't I be able to run it on a Debian system I already have (and understand)?
Running an entire new distro just seems like overkill for what it actually does over a normal Linux system. It's just a configuration manager!
I don't know, but I imagine it would make testing substantially more difficult (right now they have fully tested images available), and it would be impossible to implement their distro update thing (you can update the entire .iso backwards and forwards, the config will be reapplied).
VyOS is unfortunately completely useless for larger applications, since it's difficult to impossible to automate due to it's unique way of applying configurations. Don't get me wrong, for manual administration it's great, there's a lot of missed automation potential given that it's just Linux underneath.
As an example, the Ansible modules for VyOS are basicially just variations of an adapted ansible.builtin.shell, instead of offering to manage state in a more first class manner (via attributes and values):
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
As an example, the Ansible modules for VyOS are basicially just variations of an adapted ansible.builtin.shell, instead of offering to manage state in a more first class manner (via attributes and values):
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
From what I've seen of VyOS using a configuration file that is then used to generate the actual system configuration I'm not really sure its so hard to automate. Take your target state, generate a configuration file in the right format, then send it over and apply.
To be honest this feels more like a limitation in Ansible, which has always felt like a bit of a hacky config management system to me in that the way it functions is generally to run a bunch of commands that gradually mutate the system's state, rather than atomically applying the target state, but then I've been spoiled by NixOS on my personal infrastructure recently.
To be honest this feels more like a limitation in Ansible, which has always felt like a bit of a hacky config management system to me in that the way it functions is generally to run a bunch of commands that gradually mutate the system's state, rather than atomically applying the target state, but then I've been spoiled by NixOS on my personal infrastructure recently.
You can ssh into the router, copy a new config (state) and load that config.
It is not very elegant though.
Do you know of an open source router that does what you are looking for?
It is not very elegant though.
Do you know of an open source router that does what you are looking for?
That doesn’t sound that far off from what unifi does, it uses a daemon to do those things. (Receive configuration and then do a local apply)
unifi runs vyos underneath
UniFi doesn't use VyOS anywhere afaik. Some Ubiquiti products run EdgeOS which is a Vyatta fork (EdgeRouter, the original security gateways). More recent UniFi products (UDM, UDM-SE, UNVR, etc) run UniFi OS which is their own Debian-based OS that runs applications inside Podman containers.
There's higher level configuration for a number of major things like the interfaces, bgp, ospf and the firewall:
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
AT&T had bought Vyatta before selling it to it's current owner, but I know they used a REST API internally when deploying it for 5G Edge use cases. It looks like VyOS gained an API in 2019.
idk I've found VyOS fairly easy to automate. It doesn't have an HTTP API and everything needs to be configured through vbash afaik.
But what'd you do with Ansible on a router? Looks like VyOS has REST API and OpenFlow support, btw.
1.4 has config sync and a REST API coming in
The theory of what VyOS does is (per my understanding) really simple: Configure all the networking components of a Linux system from a single place.
Why isn't doing this much more popular? All the systems are already there, after all! Why aren't there (that I know of) dozens of projects to accomplish this relatively easy, but relatively useful task?
I think it's a pretty big deal to be able to configure that stuff from a single place. Commercial router manufacturers all do it. Why does (as far as I know) only VyOS do it on the open source side of things?
Why isn't doing this much more popular? All the systems are already there, after all! Why aren't there (that I know of) dozens of projects to accomplish this relatively easy, but relatively useful task?
I think it's a pretty big deal to be able to configure that stuff from a single place. Commercial router manufacturers all do it. Why does (as far as I know) only VyOS do it on the open source side of things?
IMO, Open Source routers are a niche thing, and open source declarative CLI routers even more of a niche thing.
Most enterprises prefer buying something with a support contract from a known name vendor (Cisco, Juniper, etc.). Most home users just use what their ISP provides them with, and of those that want something more, they either pick a SOHO vendor like Ubiquiti/Mikrotik, or if DIYing the hardware, choose pfSense / OPNSense / DD-WRT for the clickOps options, because networking really isn't trivial. For those for whom networking is trivial, Debian is fine router OS if you know your way around iptables and friends.
That leaves all those who want to use DIY hardware, and a enterprise-like declarative CLI. That's really not a whole lot of people in the end.
Most enterprises prefer buying something with a support contract from a known name vendor (Cisco, Juniper, etc.). Most home users just use what their ISP provides them with, and of those that want something more, they either pick a SOHO vendor like Ubiquiti/Mikrotik, or if DIYing the hardware, choose pfSense / OPNSense / DD-WRT for the clickOps options, because networking really isn't trivial. For those for whom networking is trivial, Debian is fine router OS if you know your way around iptables and friends.
That leaves all those who want to use DIY hardware, and a enterprise-like declarative CLI. That's really not a whole lot of people in the end.
OPNsense and pfSense exist, plus OpenWrt.
The question 'what is this thing' is probably best answered by the Github project page: https://github.com/vyos
It's a decent-ish option if you need advanced routing functionality; one thing to keep in mind, though, is that unless you're OK with running unstable 'nightly' code, you'll be spending USD 8K+ on an annual basis.
It's a decent-ish option if you need advanced routing functionality; one thing to keep in mind, though, is that unless you're OK with running unstable 'nightly' code, you'll be spending USD 8K+ on an annual basis.
> If you are an individual, you can get the generic ISO by donating on Open Collective. And if you are contributing to VyOS, whether you are writing code, improving the docs, or promoting VyOS publicly, we are happy to share pre-built images with you through contributor subscriptions. Finally, you can always build your own images — just follow these instructions.
Sounds fair to me. Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
Sounds fair to me. Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
> Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
OpenWRT comes to mind, I've been using it for decades on first dedicated hardware, the last 6 years running in a container on a ProxMox box (DL380 G7). It has no problems whatsoever routing at (gigabit) line speed using a few megabytes of RAM and a few cores. Configuration is mostly declarative using UCI although it also offers the freedom (which comes with responsibility) to use scripts. I use the latter to deal with edge cases which lie outside of the purview of normal routing operations, e.g. triggered actions related to the use of Timelimit [1] on my daughter's phone, IoShit things with special needs, etc.
[1] https://codeberg.org/timelimit/timelimit-server
OpenWRT comes to mind, I've been using it for decades on first dedicated hardware, the last 6 years running in a container on a ProxMox box (DL380 G7). It has no problems whatsoever routing at (gigabit) line speed using a few megabytes of RAM and a few cores. Configuration is mostly declarative using UCI although it also offers the freedom (which comes with responsibility) to use scripts. I use the latter to deal with edge cases which lie outside of the purview of normal routing operations, e.g. triggered actions related to the use of Timelimit [1] on my daughter's phone, IoShit things with special needs, etc.
[1] https://codeberg.org/timelimit/timelimit-server
If you want a web GUI, then pfSense or OPNSense are the general go tos.
However, if you're comfortable with CLI and modifying configs in /etc/ then just running a bare metal Alpine Linux box is perfectly doable on a tiny box. iptables/nftables for firewall/NAT, dnsmasq/bind9 for dns, dnsmasq/isc-dhcp for DHCP. I've got a handful of these boxes all interlinked via wireguard, sharing routes via BGP using bird.
Sure, you miss the config verification that VyOS provides, but does mean you learn the underlying tools themselves and that knowledge is portable to any other box running those systems.
Personally, I don't quite understand why VyOS is a standalone distro when it could just be a config generator/checker package. Could even support multiple different underlying tools so if you want to use dnsmasq over bind9, or vice versa, it can provide a unified config interface for them.
However, if you're comfortable with CLI and modifying configs in /etc/ then just running a bare metal Alpine Linux box is perfectly doable on a tiny box. iptables/nftables for firewall/NAT, dnsmasq/bind9 for dns, dnsmasq/isc-dhcp for DHCP. I've got a handful of these boxes all interlinked via wireguard, sharing routes via BGP using bird.
Sure, you miss the config verification that VyOS provides, but does mean you learn the underlying tools themselves and that knowledge is portable to any other box running those systems.
Personally, I don't quite understand why VyOS is a standalone distro when it could just be a config generator/checker package. Could even support multiple different underlying tools so if you want to use dnsmasq over bind9, or vice versa, it can provide a unified config interface for them.
isc-dhcp is EOL. I'd suggest using kea-dhcp from the same ISC. I believe there could be a script or some kind of migration path from isc-dhcp to Kea. I've been using Kea in production with no problems.
https://www.isc.org/blogs/isc-dhcp-eol/
https://www.isc.org/blogs/isc-dhcp-eol/
I really wish there was a better option for a Linux DHCP server. At my past job we were using isc-dhcp and it was absolutely horribly showing it's age (md5 api "keys", bespoke socket-based API "protocol", most things impossible to do via the "API", clustering that didn't really work, etc.). Kea is barely an improvement, with oddities such as being written in C++ and requiring recompiling for plugins, or an extremely weird API. It's obvious is still written by the same old folks who have no idea how software is supposed to work in the 2020s.
We need something modern - easy clustering, modern API, event stream, gRPC-based plugins, etc. (And yes, I have thought about developing it myself, it's on my pile of TODO)
We need something modern - easy clustering, modern API, event stream, gRPC-based plugins, etc. (And yes, I have thought about developing it myself, it's on my pile of TODO)
One remark.
Kea doesn't have some of functionalities provided by ISC-DHCP.
If you use a lot of dhcp-eval and make decisions based on different dhcp options content kea is still a no-go since there is no workarround. Usually at ISP level dhcp is one puzzle of much more complicated system.
I tried if few times and every time I stuck on something and messagefrom developers was: this isc-dhcp feature is not supported. This was huge national scale ISP and bypassing those limitations means a lot of $ to adapt surrounding systems providing input to isc-dhcp LDAP DB in its own config style.
I tried if few times and every time I stuck on something and messagefrom developers was: this isc-dhcp feature is not supported. This was huge national scale ISP and bypassing those limitations means a lot of $ to adapt surrounding systems providing input to isc-dhcp LDAP DB in its own config style.
How do you do zone based firewalls with alpine?
define zone based?
If wanting internal and external subnets as "zones", iptables/nftables lets you match against incoming and outgoing interfaces. It would be trivial to make match against an incoming interface and jump to a zone specific chain. This is how I manage private subnets. fw-mark is also useful for setting routing rules. Can change which routing table is used by matching rules in iptables.
If wanting to do more stateful things, I'm not aware of any default package, but setting a rule to send packets to an NFQUEUE and implementing some custom logic on that nfqueue would be rather trivial too. I'm sure eBPFs are useable in there somewhere too, but I've very little experience with them.
Obviously iptables/nftables has its own issues, as seen in recent (and not so recent) posts about it being bypassable with raw sockets, but that tends to be host only and not when used as a gateway.
If wanting internal and external subnets as "zones", iptables/nftables lets you match against incoming and outgoing interfaces. It would be trivial to make match against an incoming interface and jump to a zone specific chain. This is how I manage private subnets. fw-mark is also useful for setting routing rules. Can change which routing table is used by matching rules in iptables.
If wanting to do more stateful things, I'm not aware of any default package, but setting a rule to send packets to an NFQUEUE and implementing some custom logic on that nfqueue would be rather trivial too. I'm sure eBPFs are useable in there somewhere too, but I've very little experience with them.
Obviously iptables/nftables has its own issues, as seen in recent (and not so recent) posts about it being bypassable with raw sockets, but that tends to be host only and not when used as a gateway.
> define zone based?
https://support.vyos.io/en/support/solutions/articles/103000...
You create a _zone_. You name it and assign some interfaces to it. For my needs, I only assign 1 interface per zone. Then, you specify with which other zone that zone can receive traffic from. That also comes with the identification of a firewall rulesets to apply to that pair.
So, `'Zone WAN (iface eth0) <- Zone LAN (iface eth1)' => apply fw LAN-TO-WAN`
When you do that, the firewall rules become much simpler to write and maintain.
But, a best practice is to assign every zone to every other zone. This soon becomes a combinatorial nightmare. When you want to add a zone, you have to create 2xN new zone configurations and 2xN new firewall rulesets.
https://support.vyos.io/en/support/solutions/articles/103000...
You create a _zone_. You name it and assign some interfaces to it. For my needs, I only assign 1 interface per zone. Then, you specify with which other zone that zone can receive traffic from. That also comes with the identification of a firewall rulesets to apply to that pair.
So, `'Zone WAN (iface eth0) <- Zone LAN (iface eth1)' => apply fw LAN-TO-WAN`
When you do that, the firewall rules become much simpler to write and maintain.
But, a best practice is to assign every zone to every other zone. This soon becomes a combinatorial nightmare. When you want to add a zone, you have to create 2xN new zone configurations and 2xN new firewall rulesets.
So the equivalent of:
iptables -N eth0toeth1; iptables -P eth0toeth1 DROP; iptables -A FORWARD -i eth0 -o eth1 -j eth0toeth1; iptables -A eth0toeth1 -m tcp -p 80 -j ACCEPT; # add any more rules
Or, as you say to avoid exponential combinations, just make a chain for each zone (interface) and explicitly allow specific protocols/ports to target interfaces. Zones with multiple interfaces are just multiple rules to jump to the same zone chain.
iptables -N eth0toeth1; iptables -P eth0toeth1 DROP; iptables -A FORWARD -i eth0 -o eth1 -j eth0toeth1; iptables -A eth0toeth1 -m tcp -p 80 -j ACCEPT; # add any more rules
Or, as you say to avoid exponential combinations, just make a chain for each zone (interface) and explicitly allow specific protocols/ports to target interfaces. Zones with multiple interfaces are just multiple rules to jump to the same zone chain.
You can check awall: https://git.alpinelinux.org/awall/about/ for an abstraction over iptables
This looks interesting, but nowhere near VyOS abilities and documentation.
I can't even find anything related to policy-based routing (PBR).
I can't even find anything related to policy-based routing (PBR).
You have very narrow definition of what software router/service router is. Firewall and two interfaces and VPN server is not the best scenario :)
All those systems (pfSense etc) are for private/soho use.
Big networks need stuff that is not avaiable on mentioned platforms like i.e BFD (Bidirectional Forwarding Detecion), MPLS (MultiProtocol Label Switching), VXLAN (Virtual Extensible LAN), IS-IS routing protocol or Segment Routing.
If anyone knows other Opensource routing software that support all of this - let me know. To my knowledge vOS is the only one.
If anyone knows other Opensource routing software that support all of this - let me know. To my knowledge vOS is the only one.
DANOS [0] supports all of those I believe. It's essentially AT&Ts continued development of Vyatta (a subset of it at least) with a DPDK dataplane + multiple other enhancements.
[0]: https://www.danosproject.org/
[0]: https://www.danosproject.org/
openwry,ipfire,ipcop,all do the job well
Definitely they are not. I couldn't find any other open source routing software that has support for following very popular features on enterprise level:
- MPLS
- VXLAN
- IS-IS
This is must have to be considered by any ISP/enterprise where networking is their core business.
vOS has all three. More or less buggy but they are here.
This is must have to be considered by any ISP/enterprise where networking is their core business.
vOS has all three. More or less buggy but they are here.
How about OpenWRT? It appears it has support for MPLS[1], IS-IS[2], and VXLAN[3]. Where VXLAN appears to even have a LuCI component[4].
[1] https://openwrt.org/packages/pkgdata/kmod-mpls
[2] https://openwrt.org/packages/pkgdata/frr
[3] https://openwrt.org/packages/pkgdata/vxlan
[4] https://openwrt.org/packages/pkgdata/luci-proto-vxlan
[1] https://openwrt.org/packages/pkgdata/kmod-mpls
[2] https://openwrt.org/packages/pkgdata/frr
[3] https://openwrt.org/packages/pkgdata/vxlan
[4] https://openwrt.org/packages/pkgdata/luci-proto-vxlan
Enterprise level I assume some will buy commercial equipment with support etc.
The above mentioned I feel are good enough for home and SOHO(small businesses).
The above mentioned I feel are good enough for home and SOHO(small businesses).
Yeah, the cost of buying equipment and licenses supported by a team with a phone number is immaterial for most enterprise-level organizations.
https://support.vyos.io/en/support/solutions/103000152091
They have an LTS release, no?
They seem to follow the RedHat strategy though, only subscribers can download prebuilt images, but you can build the LTS ones yourself:
https://blog.vyos.io/vyos-1.3.2-lts-release
They have an LTS release, no?
They seem to follow the RedHat strategy though, only subscribers can download prebuilt images, but you can build the LTS ones yourself:
https://blog.vyos.io/vyos-1.3.2-lts-release
I have created some automation to build the LTS ISO every time a new commit is made on the 1.3 LTS branch.
A definite tangent: About three house moves ago I had OpenWRT on a cable router and knew what was going on. But with family and work and house moves I am now just staring at a flashing BT Home Hub and wondering how to tackle the inevitable "of course that router firmware wont allow that and your ISP wont give out its password and ..."
Is there a uptodate reliable guide (possibly including how to persuade your wife it's a good idea to drill holes in the living room ceiling to run cat6)
Is there a uptodate reliable guide (possibly including how to persuade your wife it's a good idea to drill holes in the living room ceiling to run cat6)
Same here. And AFAICT those "Hubs" have no bridge mode, so the best you can do is double-NAT yourself. Even if you can replace it with your own device, I've just received a letter informing me that they're migrating our landlines to be VOIP, delivered through the phone socket on the back of the router, so if you want to keep landline service then you may need to keep their hardware too
What I find easy(ier?) is to run (x)sense on a dedicated firewall and either a mesh with cheap openwrt routers or get something like Deco mesh and run it in AP mode if you don’t have cat6 at home. I think this combination can be under 300$ for a 3-pack of mesh Deco x20 + an intel card on a refurb dell optiplex.
IME (x)sense is quite problematic when you start doing anything other than the bare minimum. I spent multiple days trying to figure out why it silently stopped accepting IPv6 delegation, or why does it spike latency for no visible reason under load. The underlying reality is that FreeBSD's network stack is much more conservative and has less resources than Linux's, which shows up in articles like this one [1].
On the same Celeron J4125/i226 box VyOS was absolutely perfect, not a single issue, significantly low (and always low) latency with higher throughput.
On the hardware side, I think the /r/homelab hivemind doesn't get challenged enough. Dell optiplexes cost very similarly to Aliexpress Protectli alternatives (such as [2]), while being larger, having a fan, and being overall more hassle. TP Link/Ubiquiti WiFi APs seem to be overall inferior to Aruba Instant On, which is exactly the same hardware HP sells in their Aruba line, but for the same SOHO price.
[1] https://teklager.se/en/knowledge-base/opnsense-performance-o...
[2] https://www.aliexpress.com/item/1005004272231167.html
On the same Celeron J4125/i226 box VyOS was absolutely perfect, not a single issue, significantly low (and always low) latency with higher throughput.
On the hardware side, I think the /r/homelab hivemind doesn't get challenged enough. Dell optiplexes cost very similarly to Aliexpress Protectli alternatives (such as [2]), while being larger, having a fan, and being overall more hassle. TP Link/Ubiquiti WiFi APs seem to be overall inferior to Aruba Instant On, which is exactly the same hardware HP sells in their Aruba line, but for the same SOHO price.
[1] https://teklager.se/en/knowledge-base/opnsense-performance-o...
[2] https://www.aliexpress.com/item/1005004272231167.html
The biggest reason I like xSense is because of unbound and using root dns. OpenWRT supports that, but otherwise I have to setup a local linux instance to serve DNS via unbound. With pfsense and opnsense I find that the dns is a lot more stable than unbound as using a resolver seems to work more reliably than a forwarder like dnsmasq.
I like the idea of VyOS. Networking software tends to have so much hidden, hard to control state for reasons I don't understand. As a programmer this melts my brain.
VyOS finally lets you have all your configuration in one, easily controlled place. Nice!
For some reason I ended not actually trying it out too actively. I think I was weirded out by the distribution model and concerned by the small community.
VyOS finally lets you have all your configuration in one, easily controlled place. Nice!
For some reason I ended not actually trying it out too actively. I think I was weirded out by the distribution model and concerned by the small community.
The idea of VyOS is awesome: have your configuration all in one place, apply it atomically.
After using it for several years, the implementation is clearly lacking. It seems that the maintainers are overloaded, because contributions to fix minor issues, or add config options get ignored in my experience. As a result, the configuration is missing some nice options in the IPv6 space (so-called tethering), and it's still using iptables for packet filtering. It's also rather hard to roll your own, with your own modules: the module system is rather hard to use.
Not to mention the lack of interest to roll out a version for ARM.
My next router will be based on NixOS, and will attempt to recreate the awesome UI of VyOS.
After using it for several years, the implementation is clearly lacking. It seems that the maintainers are overloaded, because contributions to fix minor issues, or add config options get ignored in my experience. As a result, the configuration is missing some nice options in the IPv6 space (so-called tethering), and it's still using iptables for packet filtering. It's also rather hard to roll your own, with your own modules: the module system is rather hard to use.
Not to mention the lack of interest to roll out a version for ARM.
My next router will be based on NixOS, and will attempt to recreate the awesome UI of VyOS.
Unfortunately they went full "Vyatta way" or "RedHat Way" by basically giving rolling release for free only. I remember times when Vyatta went behind paywalls Vyos was completely free.
Not sure who would want something rolling on device like router.
Nowdays everyone wanting something good and free go OpnSense way.
Not sure who would want something rolling on device like router.
Nowdays everyone wanting something good and free go OpnSense way.
The biggest problem I have is that they only support (or at least that's what they release publicly) x86_64. I am forced to use openwrt because the vast majority of consumer low-power hardware is using ARM or exotic architectures. VyOS interface is vastly superior, though.
VyOS is not targeted for such usage scenarios. It is ISP who are paying for support. I have VyOS systems with BGP, MPLS, PPPoE termination, etc that are pushing gigabits of traffic. This is VyOS usecase. Not 5W router handling home or small office.
I've used EdgeOS in the past and it's perfect for SOHO routers too, but unfortunately is half full of proprietary extensions and semi-abandoned. I think VyOS could take its place nicely, openwrt feels like a toy compared.
You can build your own LTS image relatively easy and free of charge
The process is not straightforward, especially if you want to customize the image.
And I have been unable to build a cloud-init image.
But the ISO is buildable from the LTS branch and it works well.
And I have been unable to build a cloud-init image.
But the ISO is buildable from the LTS branch and it works well.
It's been a while since I tried it, but I remember this roughly being my experience as well.
I'm usually all for up to date software, but on my networking equipment??? I don't really want to beta test that stuff, but that's what they seem to want to make me do.
I'm usually all for up to date software, but on my networking equipment??? I don't really want to beta test that stuff, but that's what they seem to want to make me do.
It is not really beta software though. If you don't want to go into the trouble of building your own ISO, then, yes, you are a beta tester because the only thing they make available pre-built is the ISO from the 1.4 branch which is in flux.
You can build the ISO from the LTS branch though and that branch doesn't move much. Though, I don't know how you can tell which commit was used to release, say v1.3.2. For the moment, I simply build an LTS ISO using the latest commit of the LTS branch. That strategy has been rock solid for years now.
You can build the ISO from the LTS branch though and that branch doesn't move much. Though, I don't know how you can tell which commit was used to release, say v1.3.2. For the moment, I simply build an LTS ISO using the latest commit of the LTS branch. That strategy has been rock solid for years now.
hey... I know this post....
I need to write a follow-up so bad.
VyOS has evolved a BUNCH since I wrote this, but the same basic ideas apply. Mostly some configuration nodes have moved around.
I need to write a follow-up so bad.
VyOS has evolved a BUNCH since I wrote this, but the same basic ideas apply. Mostly some configuration nodes have moved around.
Hey! Thanks for writing this, this really helped me setup VyOS as my main router/firewall for my homelab.
Do you have any plans to expand into IPv6 functionality?
Do you have any plans to expand into IPv6 functionality?
Thank you for writing the blog, the way it is written was very engaging :)
I thought this would be about building VyOS from source, but fun article nonetheless.
I use VyOS whenever I need layer 3 routing in vSphere for test environments. NSX-T is (way) faster by dint of being deeply integrated into VMkernel, but VyOS is pretty performant for what it is and is easier to install to boot.
I use VyOS whenever I need layer 3 routing in vSphere for test environments. NSX-T is (way) faster by dint of being deeply integrated into VMkernel, but VyOS is pretty performant for what it is and is easier to install to boot.
I was wondering if this was Vyatta... it seems it is the community continuation after Broccade ceased development. Used to run this over a decade ago on a 'router' (ThinkCentre Tiny), but eventually went to a Fedora installation a few years back. Might have a look again.
... have to say that the offering is confusing; it is a subscription or a rolling release.
Wondering how much can be automated of the installation/state; would it be possible to use version control? If not, I can see the appeal to suggest Nix over this.
Wondering how much can be automated of the installation/state; would it be possible to use version control? If not, I can see the appeal to suggest Nix over this.
Both. Pay for a subscription or use the RR.
You can also build your own images with docker using the annotated releases.
You can also build your own images with docker using the annotated releases.
From my experience, Nix makes lot more reliable and easier to configure router. ymmv
I just finished building a router based on NixOS and I must disagree.
NixOS modules mostly targets desktop and servers, but the network specific configuration is still very lacking. A few examples:
- up until a couple of weeks ago the hostapd module was basically a toy: could only manage a single SSID, no way to configure the radios, hardcoded to WPA2-PSK;
- the NixOS firewall is still based on iptables and conflicts with nftables, so you must disable and manually write rules;
- the `networking.nat` module (NAT44) doesn't do NAT reflection;
- I had to write a module for Jool (NAT64, SIIT);
- I had to write a module for libreswan (IPsec);
- I had to write a module for automatic rollbacks, otherwise you can loose access if you make a mistake.
Vyatta and VyOS also provide a much higher level abstraction over the software that is being configured (e.g. you don't have to deal with a specific IPsec implementation). Finally, once you do `nixos-rebuild switch` you're on your own, while with vyatta you have a clean command line interface to inspect the state of the router and manage it.
- up until a couple of weeks ago the hostapd module was basically a toy: could only manage a single SSID, no way to configure the radios, hardcoded to WPA2-PSK;
- the NixOS firewall is still based on iptables and conflicts with nftables, so you must disable and manually write rules;
- the `networking.nat` module (NAT44) doesn't do NAT reflection;
- I had to write a module for Jool (NAT64, SIIT);
- I had to write a module for libreswan (IPsec);
- I had to write a module for automatic rollbacks, otherwise you can loose access if you make a mistake.
Vyatta and VyOS also provide a much higher level abstraction over the software that is being configured (e.g. you don't have to deal with a specific IPsec implementation). Finally, once you do `nixos-rebuild switch` you're on your own, while with vyatta you have a clean command line interface to inspect the state of the router and manage it.
And this is just basic routing and firewall with NAT.
Now try to setup 20 MPLS L3-VPN's + dozen VPLS services on top of IS-IS routed network with that ;-)
Thanks for the warning; sounds like my mileage would have varied quite a bit.
Do you think that, had your use case been simpler, you'd have enjoyed the experience?
Do you think that, had your use case been simpler, you'd have enjoyed the experience?
Well, I actually enjoyed contributing to NixOS, but running it on a router is certainly not a "it works out-of-the-box" experience.
Even if you are building a simple SOHO router you need to know a lot more about the implementation details (which daemon does what, how to write firewall rules, how to setup the network interfaces, etc.). This is because NixOS is general-purpose OS, unlike VyOS or openwrt.
You are comparing bananas and windows.
do you have an example of a router setup?
curious what this would look like.
[deleted]
https://vyos.io/
The VyOS website says:
> Democratizing how we access networks through a universal Router and Open source software.
> Our vision at VyOS is to dramatically change how we access networks so that we can all build the solutions we always dreamed of, without restrictions, limitations, or prohibitive costs.
But I'm still kind of none the wiser. Does this thing use Linux or something else?