Wireless-to-Ethernet island for RPi cluster: IPv6, NDP proxy, mDNS reflector(vladimir.varank.in)
vladimir.varank.in
Wireless-to-Ethernet island for RPi cluster: IPv6, NDP proxy, mDNS reflector
https://vladimir.varank.in/notes/2021/04/wireless-to-ethernet-island-for-homelab-cluster-ipv6-ndp-proxy-and-mdns-reflector/
59 comments
I ended up doing something very similar, if for different reasons. There was an elongated period of time when I had no home Internet service. It was more or less a nonissue, since I had 5GB of LTE-speed tethering, and most of my devices spoke Wi-Fi.
Note that I said most, not all. I had a few Raspberry Pi's, and an old Xbox and 486 PC that I used for old games.
I ended up using DD-WRT in access point mode, and rather than having it run in station mode, it would connect to my phone as a Wi-Fi client, and forward Wi-Fi packets out the Ethernet switch.
It was quite handy, and made the summer much more bearable. As an added bonus, I now keep that router config as a backup. If my actual ISP is ever having an outage, I can reupload that config, and get all of the devices on my network back online, without having to switch them over to Wi-Fi.
Note that I said most, not all. I had a few Raspberry Pi's, and an old Xbox and 486 PC that I used for old games.
I ended up using DD-WRT in access point mode, and rather than having it run in station mode, it would connect to my phone as a Wi-Fi client, and forward Wi-Fi packets out the Ethernet switch.
It was quite handy, and made the summer much more bearable. As an added bonus, I now keep that router config as a backup. If my actual ISP is ever having an outage, I can reupload that config, and get all of the devices on my network back online, without having to switch them over to Wi-Fi.
I use one of those GL.inet routers. It's been reliable for 1.5 years. I configured it to route all traffic through a proxy service (aka VPN). Most VPN clients fail open, which defeats the purpose of using a proxy for privacy. The GL.inet firmware has an option to fail-closed when the proxy connection is down.
I use a ZTE MF820B USB 4G LTE modem as a backup Internet connection. I configured the GL.inet router to talk to the modem on /dev/cdc-wdm0 and added my LTE service provider's APN hostname. With those two settings, it just works. I keep the LTE modem unplugged and on-hand for when my fiber (Webpass) goes down.
GL.inet is based in Hong Kong and is now under the control of the Chinese red party.
I use a ZTE MF820B USB 4G LTE modem as a backup Internet connection. I configured the GL.inet router to talk to the modem on /dev/cdc-wdm0 and added my LTE service provider's APN hostname. With those two settings, it just works. I keep the LTE modem unplugged and on-hand for when my fiber (Webpass) goes down.
GL.inet is based in Hong Kong and is now under the control of the Chinese red party.
> I use a ZTE MF820B USB 4G LTE modem as a backup Internet connection...GL.inet is based in Hong Kong and is now under the control of the Chinese red party.
ZTE is a Chinese state-owned enterprise, so if CCP-sanctioned action is part of your threat model I'm not sure GL.inet would be the most concerning part of that stack.
ZTE is a Chinese state-owned enterprise, so if CCP-sanctioned action is part of your threat model I'm not sure GL.inet would be the most concerning part of that stack.
Yes, and several points reduce the risk of the ZTE device:
1. The ZTE device is on the untrusted side of the network. It carries encrypted OpenVPN packets.
2. The ZTE device does not download new firmware.
3. The ZTE device is not addressable on the public Internet. It participates in the GSM network. It tunnels data between its USB interface and the NAT proxy server (APN) provided by the carrier.
And one point increases the risk: The ZTE device is physically attached to the USB port of the router.
CCP-sanctioned action directed at me is not part of my thread model. I'm more concerned with them breaking the security of the firmware to facilitate monitoring of PRC citizens who use GL.inet devices. There is precedent for this. For example CCP forces everyone in an entire province to install spyware on their phones that logs their activities and communications and transmits them to a central server unencrypted. The data goes over whatever network the user is connected to, with zero privacy or even any protection against MITM.
If GL.inet broke their firmware security, how long would the blackhats use it before the rest of us found out about the problem?
1. The ZTE device is on the untrusted side of the network. It carries encrypted OpenVPN packets.
2. The ZTE device does not download new firmware.
3. The ZTE device is not addressable on the public Internet. It participates in the GSM network. It tunnels data between its USB interface and the NAT proxy server (APN) provided by the carrier.
And one point increases the risk: The ZTE device is physically attached to the USB port of the router.
CCP-sanctioned action directed at me is not part of my thread model. I'm more concerned with them breaking the security of the firmware to facilitate monitoring of PRC citizens who use GL.inet devices. There is precedent for this. For example CCP forces everyone in an entire province to install spyware on their phones that logs their activities and communications and transmits them to a central server unencrypted. The data goes over whatever network the user is connected to, with zero privacy or even any protection against MITM.
If GL.inet broke their firmware security, how long would the blackhats use it before the rest of us found out about the problem?
Those little radios also have use in ham radio, there's a project to flash them with a cool self-healing mesh networking system called AREDN.
I played with it some, with two nodes, although there's nobody near me to mesh with unfortunately.
I played with it some, with two nodes, although there's nobody near me to mesh with unfortunately.
Those little devices are in high demand. I've been buying them in quantities of 10-at-a-time (Especially when they were cheaper) since they are useful for putting VOIP phones securley behind VPN. I've seen a lot of folks recommend them on HN. They support OpenVPN + Wireguard out of the box and are easy enough for non-techs to reconfigure in the field w/ remote instructions.
Betcha it wouldn't be too hard to use an rpi to do these tasks. ;)
Hence why I love seeing posts like the one's submitted: lessons learned, that we can all possibly use to upgrade ourselves.
It would be nice for mainstream linux (not just openwrt) to grow more user-friendly tooling. I look at opnsense in envy, but I'm not really interest in splitting my expertise, taking on learning FreeBSD too. I know I wouldn't really have to touch the OS much, that opnsense is a pretty complete UI package, but I like to keep a fuller-stack view of things, have some up & down mastery.
Hence why I love seeing posts like the one's submitted: lessons learned, that we can all possibly use to upgrade ourselves.
It would be nice for mainstream linux (not just openwrt) to grow more user-friendly tooling. I look at opnsense in envy, but I'm not really interest in splitting my expertise, taking on learning FreeBSD too. I know I wouldn't really have to touch the OS much, that opnsense is a pretty complete UI package, but I like to keep a fuller-stack view of things, have some up & down mastery.
I deployed a remote/home office printer sharing setup with these little guys for my friend's company. The VPN integration is fantastic, at least for low bandwidth uses like that.
Make sure you flash them / check their traffic. They make some telemetry requests IIRC.
By extracting the firmware^1 one can see that GL.inet provides a cloud storage service. This is opt-in. It is not enabled by default. The cloud storage has a telemetry address listed in /etc/config/glbigdata (and a ping address to Google at 8.8.8.8). Perhaps one could use the service and change "telemetry.goodcloud.xyz" to the loopback or delete it. I have no inclination to test this as I do not use cloud storage.
1. https://s3.us-east-2.amazonaws.com/download.gl-inet.com/firm...
With default configs IME one will see some constant traffic to 8.8.8.8. This is from OpenWRT mwan3's default configs. These defaults are left unchanged by GL.inet and probably other routers running OpenWRT. I have never seen anyone complain about this traffic. I doubt this is what is being referred to as "telemetry" in the above comment. Nevertheless, it is constant pinging to Google, OpenDNS or some other third party. In typical Linux fashion, mwan3, a daemon not every user is going to need, is enabled by default anyway.
https://github.com/gl-inet/gli-pub/raw/master/mwan3/files/et...
1. https://s3.us-east-2.amazonaws.com/download.gl-inet.com/firm...
With default configs IME one will see some constant traffic to 8.8.8.8. This is from OpenWRT mwan3's default configs. These defaults are left unchanged by GL.inet and probably other routers running OpenWRT. I have never seen anyone complain about this traffic. I doubt this is what is being referred to as "telemetry" in the above comment. Nevertheless, it is constant pinging to Google, OpenDNS or some other third party. In typical Linux fashion, mwan3, a daemon not every user is going to need, is enabled by default anyway.
https://github.com/gl-inet/gli-pub/raw/master/mwan3/files/et...
simpler is boring & un-educatuonal but thanks anyways for the good resources. those trying to learn & improve ought know what provided offerings there are about. that is openwrt based is extra compelling that we should better document & explain explain what is really afoot here!!!
OpenWRT is unfortunately rather limiting.
For example: can you run tcpflow on your mini router?
For example: can you run tcpflow on your mini router?
this is a terrible post! almost all software will compile on openwrt. the sdk build option builds a very nice self contained toolset that builds aost anything. it's just linux. this "openwett is unfortunately rather limiting" shallow posting sounds like ridiculously unsubstantiated corporate anti-open-source software droneposting. spreading shit to confuse the field. f this. bad posting. stop kneecapping great flexible distros. this shallow challenge made, "tcpflow", is some ultra ultra particular horseshit that doesnt reflection what we could do, creates a sense of limit where really possibilities are endless as we wish. terrible conservative posting, I hate it.
this is just a really narrow gripe, picking on one very tiny small perspective angle. and it doesn't take any responsibility for what it might take to do better. this is extremely cheap talk shade casting. i'm incredibly unimpressed. it reeks of sabotage.
I'd rather run Debian too. but everything about this post stinks of meanspirited senseless misonfo sabotage.
this is just a really narrow gripe, picking on one very tiny small perspective angle. and it doesn't take any responsibility for what it might take to do better. this is extremely cheap talk shade casting. i'm incredibly unimpressed. it reeks of sabotage.
I'd rather run Debian too. but everything about this post stinks of meanspirited senseless misonfo sabotage.
Haven't played with softflow, thanks will definitely try it, but an initial quick look shows me it's not as easy and straightforward as tcpflow.
I might be wrong, will look at it longer.
But my initial point, which I should have elaborated on: compiling stuff for openwrt is a gigantic pain in the butt.
I tried to compile tcpflow for OpenWRT, installed a VM with the whole toolchain, messed with it for 4 hours and finally gave up when I realized I would have to also recompile a recent openssl to to get tcpflow to link.
Much better to run a "proper" linux distro on these firewall/routers hardware.
I might be wrong, will look at it longer.
But my initial point, which I should have elaborated on: compiling stuff for openwrt is a gigantic pain in the butt.
I tried to compile tcpflow for OpenWRT, installed a VM with the whole toolchain, messed with it for 4 hours and finally gave up when I realized I would have to also recompile a recent openssl to to get tcpflow to link.
Much better to run a "proper" linux distro on these firewall/routers hardware.
The post isn't mentioning DHCPv6-PD[*] (prefix delegation.) I feel like murphy's law is kicking into effect right now and the ISP-provided router actually supports PD to give downstream routers their own /64.
(Or maybe not. Who knows. I feel like the post would've mentioned PD if they tried it.)
Also:
---
[*]: https://en.wikipedia.org/wiki/Prefix_delegation
[*]: https://tools.ietf.org/html/rfc3633
[*]: https://github.com/openwrt/odhcp6c (-P option)
(Or maybe not. Who knows. I feel like the post would've mentioned PD if they tried it.)
Also:
I divided it into a smaller subnet 2001:db8:abc:123:40::/76
---
[*]: https://en.wikipedia.org/wiki/Prefix_delegation
[*]: https://tools.ietf.org/html/rfc3633
[*]: https://github.com/openwrt/odhcp6c (-P option)
Upvoting as good idea, but most routers I know, evening running custom firmware, don't support delegating a wan prefix on request from lan side.
Really depends on the ISP; it's actually on the certification mandatory feature list for some network operators. (Comcast, AFAIK.) But yeah, no way to tell if the particular device and setup the author has supports it.
(But it should be a very early thing to try, if it's available everything else becomes much easier.)
(But it should be a very early thing to try, if it's available everything else becomes much easier.)
[deleted]
> But it should be a very early thing to try, if it's available everything else becomes much easier
I've tried to set up PD but it didn't work, so I've moved on with other options. Now, after you mentioned that, this feels like a good excuse to delve into what exactly didn't work back then.
I've tried to set up PD but it didn't work, so I've moved on with other options. Now, after you mentioned that, this feels like a good excuse to delve into what exactly didn't work back then.
Happy starlink and pfsense user here doing just that.
I don't know about *WRT and Tomato, but OPNsense / pfSense and OpenBSD definitely support PD.
*WRT supported it last I checked, but that's no help in this case, as the ISP-provided locked-down router would need to support this.
(This isn't about ISP-provided locked-down routers either, though those are an abomination too IMHO. Regardless of whether the "last" ISP-controlled device is a plastic router in your home or an aggregator somewhere else, it needs to support and offer PD for you...)
(This isn't about ISP-provided locked-down routers either, though those are an abomination too IMHO. Regardless of whether the "last" ISP-controlled device is a plastic router in your home or an aggregator somewhere else, it needs to support and offer PD for you...)
Yes, it requires using your own router instead of the ISP's. I know the article says the ISP router is locked down, but it's not clear whether the ISP router is mandatory.
Fritz!Box supports it (needs to be enabled in the IPv6 settings) so does OpenWRT
Fantastic, love my FritzBox, I'll look into this. Stand happily corrected!
Out of interest, why didn't you just bridge the two interfaces? Did you want everything on a separate subnet? (maybe I missed that but but you mentioned before how everything used to be directly plugged in anyway)
You can't bridge regular 802.11 wireless into ethernet at the client side. The on-air addressing requires the client MAC address to be the same as the ethernet packet's sender.
802.11 has the concept of "transmitter address" and "receiver address" in addition to source and destination. Those are MAC addresses too, but they're relevant for the on-air radio management. Things like RTS, CTS, ACKs, and fancier things like beamforming and sounding. The problem is that the design only includes 3 address fields in on-air frames; the AP can specify separate SA and TA (i.e. send a packet for somebody else, SA=real source, TA=AP MAC, RA=DA=client.) There is no mechanism for the client to do the same thing; that would require 4 address fields.
Coincidentally, 4 address fields is exactly what you get with "WDS" / "Wireless Extender" / ... modes. However, these need to be supported, enabled and configured on both the AP and client. The author of the post seems to have no access to the AP to do so (and the AP possibly doesn't support it anyway.)
802.11 has the concept of "transmitter address" and "receiver address" in addition to source and destination. Those are MAC addresses too, but they're relevant for the on-air radio management. Things like RTS, CTS, ACKs, and fancier things like beamforming and sounding. The problem is that the design only includes 3 address fields in on-air frames; the AP can specify separate SA and TA (i.e. send a packet for somebody else, SA=real source, TA=AP MAC, RA=DA=client.) There is no mechanism for the client to do the same thing; that would require 4 address fields.
Coincidentally, 4 address fields is exactly what you get with "WDS" / "Wireless Extender" / ... modes. However, these need to be supported, enabled and configured on both the AP and client. The author of the post seems to have no access to the AP to do so (and the AP possibly doesn't support it anyway.)
How did my AirPort Express or do my Sonos do it? Just rewrite the MAC Address? (is that then not technically bridging from terminology pov?)
Yes, one of the worst hacks in all of home networking is exactly this... rewriting the MAC address. aka "MAC NAT." Alternatively, proxy ARP + proxy ND can also work. The Apple stuff may be able to autonegotiate over some proprietary handshake if it's an Apple AP and Apple Extender.
There is also a new standard that covers 4-address frames, 802.11ak; I have no idea how widely that is adopted though — it was only released in 2018[1].
[1] https://standards.ieee.org/news/2018/ieee_802_11ak-2018.html
There is also a new standard that covers 4-address frames, 802.11ak; I have no idea how widely that is adopted though — it was only released in 2018[1].
[1] https://standards.ieee.org/news/2018/ieee_802_11ak-2018.html
Thank you for your expertise! Out of interest, why is it a hack? Does it break things elsewhere? Why is it better to do it at the IP level?
NAT, in general, is a hack that requires holding a lot of state in order to squeeze multiple devices into a limited number space.
On a technical/complexity level, IP NAT is much worse because you need to hold much more state, i.e. you need the UDP/TCP flow information to rewrite correctly.
However, MAC NAT is the pinnacle of stupidity for an entirely different reason: there should be no need for it. MAC addresses only have local significance, and while there are some long-term concerns about them running out, there is absolutely enough of them right now. There should simply be no need to do MAC NAT, if only it wasn't for the shortsighted 802.11 design decision to go with 3 addresses in the header.
FWIW, MAC NAT is almost the same thing as a router with ARP/ND proxying turned on, though possibly implemented on a different level. This is the technical reason it's an extremely stupid hack: proxy ND/ARP provides pretty much the same thing, but in a much cleaner way. (The difference is that with proxy ND/ARP, the "router-ish-bridge" assumes ownership of the lower-layer exchanges, i.e. ARP & ND, and just does normal routing with that. MAC NAT, meanwhile, tries to be clever and just forward ARP and ND. Reasons for doing that are ... extremely thin IMHO.)
On a technical/complexity level, IP NAT is much worse because you need to hold much more state, i.e. you need the UDP/TCP flow information to rewrite correctly.
However, MAC NAT is the pinnacle of stupidity for an entirely different reason: there should be no need for it. MAC addresses only have local significance, and while there are some long-term concerns about them running out, there is absolutely enough of them right now. There should simply be no need to do MAC NAT, if only it wasn't for the shortsighted 802.11 design decision to go with 3 addresses in the header.
FWIW, MAC NAT is almost the same thing as a router with ARP/ND proxying turned on, though possibly implemented on a different level. This is the technical reason it's an extremely stupid hack: proxy ND/ARP provides pretty much the same thing, but in a much cleaner way. (The difference is that with proxy ND/ARP, the "router-ish-bridge" assumes ownership of the lower-layer exchanges, i.e. ARP & ND, and just does normal routing with that. MAC NAT, meanwhile, tries to be clever and just forward ARP and ND. Reasons for doing that are ... extremely thin IMHO.)
> pinnacle of stupidity
Thanks for that. Filed away for future use.
Thanks for that. Filed away for future use.
> Right from the beginning, it’s worth to mention, that the router (Sagemcom F@st) is super limited in what it allows configuring. [...] Similarly, I can’t set up any custom routing or configure which DNS servers the router’s built-in DHCP provides for the home networks — the configuration is locked by the vendor
I had the same problems with my modem/router, a Fastgate by Fastweb (in .it)
No custom static routes, dns fixed by the provider, no vpn functionalities. Some arbitrary tcp ports can't be forwarded via NAT. TR-069 was up and running, at least in ~2017, and at the same year at CCC in Hamburg (or Leipzig?) there was a nice talk about how good of an attack vector TR-069 is.
All this is quite infuriating, specifically the DNS thing.
I ended up replacing the whole thing with custom equipment (ONT + a Linksys WRT3200ACM running OpenWrt).
But I honestly think that stuff like this should be illegal.
I had the same problems with my modem/router, a Fastgate by Fastweb (in .it)
No custom static routes, dns fixed by the provider, no vpn functionalities. Some arbitrary tcp ports can't be forwarded via NAT. TR-069 was up and running, at least in ~2017, and at the same year at CCC in Hamburg (or Leipzig?) there was a nice talk about how good of an attack vector TR-069 is.
All this is quite infuriating, specifically the DNS thing.
I ended up replacing the whole thing with custom equipment (ONT + a Linksys WRT3200ACM running OpenWrt).
But I honestly think that stuff like this should be illegal.
And this is all because WiFi doesn't act like a proper layer 2... It isn't possible to have a network which is ethernet -> wifi -> ethernet and have the whole thing one broadcast domain.
Not true at all, you could bridge the wifi interface to the ethernet interface and call it a day with a single broadcast domain
Well, as at least one other commenter in this thread already pointed out, this is possible with WDS (Wireless Distribution System). However, this needs to be supported by the access points. If it is supported (for example on APs running OpenWRT), it is literally just a matter of enabling WDS on the station and client APs, and bridging the wireless interfaces to the ethernet interfaces.
I've been using this setup in my home network for years now (with a dedicated OpenWRT device for each wired "island") and it works great.
Edit: To clarify, yes, this establishes a single broadcast domain. For example, DHCP and ARP requests are propagated through the entire network.
I've been using this setup in my home network for years now (with a dedicated OpenWRT device for each wired "island") and it works great.
Edit: To clarify, yes, this establishes a single broadcast domain. For example, DHCP and ARP requests are propagated through the entire network.
> The router is set with ... a global unicast IPv6 address (GUA) prefix 2001:db8:abc:123::/64, which the ISP designates to us (of course, that’s not the real prefix, but I will use this one in all examples below).
On a side note, I have more trust in documentation that is compliant with the relevant RFCs (i.e., RFC1918, RFC3849, RFC5737, et al).
In my experience, such documentation is much more likely to be "technically correct" and get the small details right.
On a side note, I have more trust in documentation that is compliant with the relevant RFCs (i.e., RFC1918, RFC3849, RFC5737, et al).
In my experience, such documentation is much more likely to be "technically correct" and get the small details right.
Ended up doing something similar except went for wifi 6 routers at both ends so that it could carry gigabit
What if your IPv6 prefix changes? As far as i know prefixes from Vodafone cable in Germany are semi static so it could change in a few months.
As far as I can tell, I have the same IPv6 prefix for at least a year. Of course, if the prefix's changed, I'll have to reconfigure the homelab. If that started to become annoying I would automate that with an ansible task. But will probably need to search for a better and more stable solution.
Powerline is your friend when you can’t lay new cables.
How do you relay mDNS over IPv4?
The article mentioned an ARP relay.
Any recommendations?
The article mentioned an ARP relay.
Any recommendations?
avahi (the standard Linux mDNS implementation) has settings to set up proxying between multiple segments.
It's not ARP relay, it's proxy ARP. That's a builtin feature on the Linux kernel, with 2 distinct modes to configure and enable it. (a) /proc/sys/net/.../proxy_arp, or (b) ip neigh add proxy ...; the latter way is more fine grained while the former is just an interface-wide switch that you flick on.
It's not ARP relay, it's proxy ARP. That's a builtin feature on the Linux kernel, with 2 distinct modes to configure and enable it. (a) /proc/sys/net/.../proxy_arp, or (b) ip neigh add proxy ...; the latter way is more fine grained while the former is just an interface-wide switch that you flick on.
This seems like a lot of work to avoid having to run a long cable.
> This seems like a lot of work
I see this is your first introduction to the homelab hobby. Welcome!
I see this is your first introduction to the homelab hobby. Welcome!
I tried to figure out what a homelab was once, but it seemed to just be people who enjoyed having a lot of fan noise at home so they could assemble VLANs for no particular reason.
Homelab is a wide hobby. Some people assemble their lab to learn how things work as a way to skill up for work. Some people assemble a small lab (sometimes just a single box with a bunch of VMs) to run home automation or media applications.
Other people just enjoy the blinkenlights.
Other people just enjoy the blinkenlights.
If you follow the homelab sub reddit, a lot of people like to run old obsolete datacenter gear.
You don't have to do that. Mine is a pair of Ryzen 2 white boxes. They are dead silent and don't put out much heat either. All my network gear is fanless too.
I do automation, so I use mine for dev/test against the stuff I am automating. A lot of it could run in the cloud, but that gets expensive real quick. Especially when I am working with a VMware stack.
You don't have to do that. Mine is a pair of Ryzen 2 white boxes. They are dead silent and don't put out much heat either. All my network gear is fanless too.
I do automation, so I use mine for dev/test against the stuff I am automating. A lot of it could run in the cloud, but that gets expensive real quick. Especially when I am working with a VMware stack.
I have the same thing with a NAS in a reasonable quiet PC box, but when I filed some FreeNAS bugs some guy actually yelled at me for not having a 1U server next to my TV for better airflow…
I have 200TB in my garage.
too sharp a point apologies, but, some homelabs acquire gear, some acquire skill.
i don't see your claim about how much storage you have as any sort of technical claim, any proof of anything. i think it's expected that the homelab folk have an interest in going deep, in some areas. if networking isn't your bag, isn't interesting to you, fine. but the size of your storage cluster or how many vm's your running isn't really an interesting counter-claim.
i don't see your claim about how much storage you have as any sort of technical claim, any proof of anything. i think it's expected that the homelab folk have an interest in going deep, in some areas. if networking isn't your bag, isn't interesting to you, fine. but the size of your storage cluster or how many vm's your running isn't really an interesting counter-claim.
Not to mention a lot more latency.
wifi hop should be under 8ms. untethering sounds like a great & sophisticated way to skill up.
also within this island latencies will remain low.
also within this island latencies will remain low.
I use it for wireless Wake-on-Lan for my homelab PC, integrated with homeassistant and Google Assistant voice command “Hey Google, turn on homelab”.
[1] - perhaps due to chip shortage, it is closer to $30 now - https://www.amazon.com/dp/B073TSK26W