Pulling MikroTik into the Limelight Demystifying and Jailbreaking RouterS(margin.re)
margin.re
Pulling MikroTik into the Limelight Demystifying and Jailbreaking RouterS
https://margin.re/2022/06/pulling-mikrotik-into-the-limelight/
58 comments
You've thrown out a lot of performance numbers here but most of us aren't compliling rust on our routers. Have you stood up iperf to see how fast it routes? The task is harder than you think if you want any features to go with that and there's a good reason most vendors (Mikrotik and Ubiquiti included) offload most of the task to silicon.
Yeah, sure, but it's frustrating. Sure casual users who won't saturate the network get fast offloaded performance.
However more aggressive users will end up enabling more complicated firewall rules, QoS, buffer bloat mitigation, or deep packet inspection and disabling the silicon based acceleration or bottlenecking elsewhere.
Thus my interest in the RK3588, that provides 4x the PCIe of the RPi4 and a fair bit more CPU.
I do wish there was a popular benchmark that used a more realistic benchmark. Smaller packets, dozens or 100s of TCP sessions, non-trivial firewall rules, etc. Doesn't seem like you learn much from iperf managing 95% of of line rate with large packets.
But here's the iperf numbers I stumbled across: https://www.cnx-software.com/2022/11/12/nanopi-r6s-review-un...
However more aggressive users will end up enabling more complicated firewall rules, QoS, buffer bloat mitigation, or deep packet inspection and disabling the silicon based acceleration or bottlenecking elsewhere.
Thus my interest in the RK3588, that provides 4x the PCIe of the RPi4 and a fair bit more CPU.
I do wish there was a popular benchmark that used a more realistic benchmark. Smaller packets, dozens or 100s of TCP sessions, non-trivial firewall rules, etc. Doesn't seem like you learn much from iperf managing 95% of of line rate with large packets.
But here's the iperf numbers I stumbled across: https://www.cnx-software.com/2022/11/12/nanopi-r6s-review-un...
I suspect the majority of consumer gigabyte routers are never saturated. The technology has eclipsed typical demand. Not that there should be deceptive marketing or performance cheating, but I suspect 95%+ of people will never be limited by their network gear.
I think building it yourself works well for routers and NASes, but there's a gap for WiFi. Lot of the bells and whistles that make modern wifi perform well (beamforming, MIMO, ...) is locked behind proprietary software and your homemade access point will never work all that well.
Interestingly, this is also something MikroTik isn't very good at - their long-range wifi is nice, but their consumer wifi used to be stuck at 2x2 and basically worse than cheap Asus "wifi router" which is somewhat embarrassing.
For switches, if you want fast, for small installation it is hard to beat the Mikrotik 4xSFP+ 10G switch for $150.
Interestingly, this is also something MikroTik isn't very good at - their long-range wifi is nice, but their consumer wifi used to be stuck at 2x2 and basically worse than cheap Asus "wifi router" which is somewhat embarrassing.
For switches, if you want fast, for small installation it is hard to beat the Mikrotik 4xSFP+ 10G switch for $150.
you are drinking too much of their marketing.
Not a single wifi consumer vendor have any say on those features. it is one hundred percent done on the closed OS in the radio chip.
your expensive ubiquiti/oanda/cisco have the same mimo/beanforming performance as any opensource/clone using the same wifi radio chipset.
the consumer facing OS just flip a bit somewhere in the configuration flash. granted, knowledge of the right bit to flip mighty be missing on the opensource still, but there's no magic happening in ubiquity owned code.
uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.
Not a single wifi consumer vendor have any say on those features. it is one hundred percent done on the closed OS in the radio chip.
your expensive ubiquiti/oanda/cisco have the same mimo/beanforming performance as any opensource/clone using the same wifi radio chipset.
the consumer facing OS just flip a bit somewhere in the configuration flash. granted, knowledge of the right bit to flip mighty be missing on the opensource still, but there's no magic happening in ubiquity owned code.
uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.
> uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.
They're not - I have no idea about the hardware itself, but what makes Ubiquiti so popular is the software integration layer. Their stuff Just Works even for incredibly complex and large installations while still being easy to configure.
They're not - I have no idea about the hardware itself, but what makes Ubiquiti so popular is the software integration layer. Their stuff Just Works even for incredibly complex and large installations while still being easy to configure.
I've also replaced a Ubiquiti EdgeRouter with a spare PC I had around. It was a small form factor thing, I don't need WiFi, so I replaced the WiFi card with a M.2 Ethernet card[1], which gives me two network interfaces. The point is pretty much anything is powerful enough to be a router now.
(Actually my setup is more complex, as the machine is powerful enough to run a few VMs, so I pass through one of the PCI devices to a firewall VM, which means the VM host doesn't even see the outside world.)
One thing to watch out for is Intel released some bad 2.5Gbe hardware (e.g. earlier versions of the I225-V). I even have a more recent one that seems to have issues (even when running the latest Linux from kernel.org). You can find Realtek cards though, which seem to work better (yeah, who'd have thought that would be the case).
[1]: It was something like: https://www.amazon.com/Suroene-Ethernet-Adapter-Multi-Gigabi... -- there are others.
(Actually my setup is more complex, as the machine is powerful enough to run a few VMs, so I pass through one of the PCI devices to a firewall VM, which means the VM host doesn't even see the outside world.)
One thing to watch out for is Intel released some bad 2.5Gbe hardware (e.g. earlier versions of the I225-V). I even have a more recent one that seems to have issues (even when running the latest Linux from kernel.org). You can find Realtek cards though, which seem to work better (yeah, who'd have thought that would be the case).
[1]: It was something like: https://www.amazon.com/Suroene-Ethernet-Adapter-Multi-Gigabi... -- there are others.
There also are dual 2.5G Ethernet (and multiple SATA) adapters for M.2 sockets, which make things even more interesting for those little ARM boards.
https://www.aliexpress.us/item/3256804337466480.html
https://www.aliexpress.us/item/3256804400925663.html
OpnSense/XigmaNAS devs, are you listening?
https://www.aliexpress.us/item/3256804337466480.html
https://www.aliexpress.us/item/3256804400925663.html
OpnSense/XigmaNAS devs, are you listening?
Apparently I225 went through 2-3 revisions, and the "b3" is supposedly ok. But the newer $200 PCs are now coming with the I226, which it's claimed is much better.
Re that M.2 Ethernet card, that's a long untwisted ribbon cable. 100Mbit should be fine, but 1G or 2.5G? Sus.
The untwisted ribbon cable is fine for PCIe in short runs by virtue of PCIe data lines being a differential signal and if the length of cable is kept short(-ish). The max length of the cable should vary inversely with the generation, that is, max length of a flexible copper extender for gen v1 is longer than gen v3, more length introduces more noise which increases error rates. A well terminated cable should be able to use the full bandwidth of the bus. I'd be completely comfortable running x16 flat out at max rate with one of those.
I don't have any hard-and-fast sources to back the above up, but punching 'pcie extender max length' into the Gargler should find you something to suffice.
I don't have any hard-and-fast sources to back the above up, but punching 'pcie extender max length' into the Gargler should find you something to suffice.
I take it you've seen PCIe riser cables like this one:
https://www.amazon.com/Zer-one-7-5Inch-Extension-Flexible/dp...
If that sort of thing can work at PCIe rates I don't see any reason why that M.2 card can't. It's a lower bit rate plus theoretically a more resilient encoding for long distance as opposed to PCIe.
https://www.amazon.com/Zer-one-7-5Inch-Extension-Flexible/dp...
If that sort of thing can work at PCIe rates I don't see any reason why that M.2 card can't. It's a lower bit rate plus theoretically a more resilient encoding for long distance as opposed to PCIe.
I bought one of the Edgerouters thinking it would tie into their Unifi Web controller interface - no hells bells - got a Mikrotik like device where you need Cisco IOS level of skill to configure.
Got a TPLINK Omada ER605 router now - just as powerful but way easier to configure.
Got a TPLINK Omada ER605 router now - just as powerful but way easier to configure.
> got a Mikrotik like device where you need Cisco IOS level of skill to configure
That's giving it too much credit. I have experience with Cisco IOS and NexusOS, and I'll say that Mikrotik's RouterOS was hard.
That's giving it too much credit. I have experience with Cisco IOS and NexusOS, and I'll say that Mikrotik's RouterOS was hard.
The funny thing is that when I was big on ubiquiti I always went the other way edgerouter only. mainly because I did not want to get tied into the whole unifi ecosystem.
They sold two versons of their switches and routers, the black ones(good) which were stand alone and the white ones(bad) which depended on the unifi system.
But then again I am also the idiot who ran openbsd on my edgerouters.
They sold two versons of their switches and routers, the black ones(good) which were stand alone and the white ones(bad) which depended on the unifi system.
But then again I am also the idiot who ran openbsd on my edgerouters.
> got a Mikrotik like device where you need Cisco IOS level of skill to configure
You have not used a Mikrotik wifi router in the last 10 years.
You have not used a Mikrotik wifi router in the last 10 years.
If you're considering straight nftables, do straight nftables. The syntax isn't bad for editing/reading directly. The userspace tool has quirks but you'll get the hang of it. I run my config through a templating engine (the same ruleset goes to cloud VMs etc).
My main router is virtualized on a Ryzen 5600G that also does other tasks. I have two other wifi APs that are just low power x86 boxes that also run Kodi.
Getting back to the original topic, I've still got Mikrotik for switches. The hardware is decent, the software is janky. I've got three different flavors of config - SwOS, RouterOS, and old RouterOS that requires a different type of config because the older switch chips were never updated to the newer config commands.
Mikrotik continues to hold their software too tightly instead of making the transition to a hardware company. They've built so much routing functionality, that's just pointless on devices that are switches. Yet there is some switch functionality that isn't available on SwOS. And SwOS is web-only with a binary config backup, rather than something conducive to centralized administration.
I'd love to find some low power white box 1Gb+10Gb switches that could run straight open Linux, or even using my existing Mikrotik switches that way if someone were to forcibly open their the environment.
My main router is virtualized on a Ryzen 5600G that also does other tasks. I have two other wifi APs that are just low power x86 boxes that also run Kodi.
Getting back to the original topic, I've still got Mikrotik for switches. The hardware is decent, the software is janky. I've got three different flavors of config - SwOS, RouterOS, and old RouterOS that requires a different type of config because the older switch chips were never updated to the newer config commands.
Mikrotik continues to hold their software too tightly instead of making the transition to a hardware company. They've built so much routing functionality, that's just pointless on devices that are switches. Yet there is some switch functionality that isn't available on SwOS. And SwOS is web-only with a binary config backup, rather than something conducive to centralized administration.
I'd love to find some low power white box 1Gb+10Gb switches that could run straight open Linux, or even using my existing Mikrotik switches that way if someone were to forcibly open their the environment.
> Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).
I did this with my (mostly) OpenWrt-based network: https://github.com/kwesthaus/network-config.
I chose the NanoPi R4S instead of the R6S as my router since it is fully supported by vanilla OpenWrt [1] whereas the R6S is still in the works [2].
[1]: https://openwrt.org/toh/friendlyarm/nanopi_r4s_v1
[2]: https://forum.openwrt.org/t/nanopi-r6s-a-rockchip-rk3588s-ro...
I did this with my (mostly) OpenWrt-based network: https://github.com/kwesthaus/network-config.
I chose the NanoPi R4S instead of the R6S as my router since it is fully supported by vanilla OpenWrt [1] whereas the R6S is still in the works [2].
[1]: https://openwrt.org/toh/friendlyarm/nanopi_r4s_v1
[2]: https://forum.openwrt.org/t/nanopi-r6s-a-rockchip-rk3588s-ro...
I saw an Armbian release for a similar RK3588 sbc, hopefully that image works and had a mainline kernel.
Don't know about the Linux distros, but one problem with the FreeBSD versions (pfSense, OPNsense) is the mostly nonexistent WiFi support because there's no driver support for PCIe cards. You end up needing to hook up a COTS Wifi AP, which has all the same problems that a COTS router would've had.
But other than that, yeah I don't see any reason to use a COTS router when I can just make one. My current router is my old desktop PC that became redundant when I built a new one, sitting as a bare motherboard on a test bench to take up very little space, running OPNsense.
But other than that, yeah I don't see any reason to use a COTS router when I can just make one. My current router is my old desktop PC that became redundant when I built a new one, sitting as a bare motherboard on a test bench to take up very little space, running OPNsense.
How does a common wireless AP have the same issues as a common consumer NAT router? Do APs have to worry about NAT state tables? Are they exposed to the Internet? Do they have to be configured to NAT/forward special edge case devices, like Nintendo / Sony games consoles?
Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.
Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.
>Are they exposed to the Internet?
They're exposed to WiFi. They have terrible security in baked-in firmware that can't be updated without connecting them to vendor clouds, if at all.
FragAttack was just last year.
>Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.
Ah yes, because one can think of three things that APs are not affected by, therefore APs are bullet-proof.
They're exposed to WiFi. They have terrible security in baked-in firmware that can't be updated without connecting them to vendor clouds, if at all.
FragAttack was just last year.
>Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.
Ah yes, because one can think of three things that APs are not affected by, therefore APs are bullet-proof.
Cloud-based APs? Seriously? Nobody who actually takes security seriously would use cloud-based APs. Even using cloud services to download firmware updates would be shady as heck.
You just made my point for me. If you have to download binary blobs to your hardware, like wifi interfaces, you can have issues. Would you rather have vendor-dependent devices inside your firewall / NAT router, or just connected to the same network as your firewall / NAT router?
Nobody said APs are bulletproof, silly. Only people who don't know better would suggest or even imply that having separate APs is somehow worse. You want to keep the less secure stuff over which you have less control out of the important systems.
You just made my point for me. If you have to download binary blobs to your hardware, like wifi interfaces, you can have issues. Would you rather have vendor-dependent devices inside your firewall / NAT router, or just connected to the same network as your firewall / NAT router?
Nobody said APs are bulletproof, silly. Only people who don't know better would suggest or even imply that having separate APs is somehow worse. You want to keep the less secure stuff over which you have less control out of the important systems.
Are you using the images available from FriendlyElec? I have limited trust in them.... Been struggling to find a good image except for DietPi.
Heh, it came with FriendlyElec, I just used it long enough to make sure the hardware was working and stable. I always reinstall Linux from something trusthworthy upon receipt.
However all the NanoPi images are pinned to the same 5.10.x kernel, that I don't think it upstreamed, which is pretty ugly.
Armbian just had a new release, supporting a TON of SBCs, even a Risc-V board, which is weird based on the name. In any case, they support another RK3588 board, so I was going to try that. I want something using a mainline linux kernel if at all possible.
However all the NanoPi images are pinned to the same 5.10.x kernel, that I don't think it upstreamed, which is pretty ugly.
Armbian just had a new release, supporting a TON of SBCs, even a Risc-V board, which is weird based on the name. In any case, they support another RK3588 board, so I was going to try that. I want something using a mainline linux kernel if at all possible.
> I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS.
Which one did you decide to stick with? Or are you using both?
The problem I've had with OpenWRT is updates are a pain.
Which one did you decide to stick with? Or are you using both?
The problem I've had with OpenWRT is updates are a pain.
OpenWrt upgrades can be made MUCH more convenient by installing extra packages that enable "Attended Sysupgrade" - check https://openwrt.org/docs/guide-user/installation/attended.sy... for more info, and please spread the word if you like it! :)
The 1x1GBe port on your system is the controller built in to the SoC. It supports jumbo frames so you can do full size packets over PPPoE if your ISP can handle that.
I recently discovered a vulnerability and jailbroke RouterOS 7.4 (beta). As far as I know, this is the latest jailbreak for RouterOS (as of today):
https://nns.ee/blog/2022/08/05/routeros-container-rce.html
https://nns.ee/blog/2022/08/05/routeros-container-rce.html
Awesome work! Also, thanks for pointing out that RouterOS supports Docker now, there's no way this is the only vuln in that implementation.
It is a feature with warning everywhere. You'll need physical access to the device to enable docker.
I think is still a very nice feature to release even if it's no complete and fully secure yet.
> your router is as secure as anything you run in container; > if you run container, there is no security guarantee of any kind; > running a 3rd party container image on your router could open a security hole/attack vector/attack surface;
https://help.mikrotik.com/docs/display/ROS/Container#Contain...
I think is still a very nice feature to release even if it's no complete and fully secure yet.
> your router is as secure as anything you run in container; > if you run container, there is no security guarantee of any kind; > running a 3rd party container image on your router could open a security hole/attack vector/attack surface;
https://help.mikrotik.com/docs/display/ROS/Container#Contain...
I think the key difference here is that it enabled root access to the host RouterOS, which is generally not something that RouterOS permits, enables, or allows. It's why the word "jailbreaking" can even be used in a RouterOS context - it's similar to jailbreaking iOS or rooting an Android phone, where the end-user is NOT meant to have root access to the device.
It started with Synology, I'm not sure I like the world where docker runs on anything.
Slide deck from the con's site: https://cfp.recon.cx/media/2022/submissions/TZUFS8/resources...
Slide deck from margin.re: https://margin.re/content/files/2022/11/Pulling_MikroTik_int...
Blurb about the presentation at the con's site: https://cfp.recon.cx/media/2022/submissions/TZUFS8/resources...
Blurb about the presentation at margin.re: https://margin.re/pulling-mikrotik-into-the-limelight-2/
Slide deck from margin.re: https://margin.re/content/files/2022/11/Pulling_MikroTik_int...
Blurb about the presentation at the con's site: https://cfp.recon.cx/media/2022/submissions/TZUFS8/resources...
Blurb about the presentation at margin.re: https://margin.re/pulling-mikrotik-into-the-limelight-2/
Thanks.
Here is the archive.org link now that the original link is 404'ing.
https://web.archive.org/web/20221208070504/https://margin.re...
https://web.archive.org/web/20221208070504/https://margin.re...
It's back
Why do we have to jailbreak thins using opensource software? EU right to repair, should be extended to right to patch, hack, tinker.
AFAIK, for a long they were pretty hostile and there's still this webpage which asks a $45 wire transfer in order to receive a CD with open source components used in the routerOS: https://mikrotik.com/downloadterms.html
That might have changed now, but I still can't find an official repo. You could look for older versions like this: https://github.com/robimarko/routeros-GPL or try to contact them directly (or sue if you think you have a standing).
That might have changed now, but I still can't find an official repo. You could look for older versions like this: https://github.com/robimarko/routeros-GPL or try to contact them directly (or sue if you think you have a standing).
I have asked for sources recently and they just sent me a link to a .tar the next day.
The EU is actively hindering consumers from getting into the radio of wifi-APs with the Radio Equipment Directive.
Blame the Germans, those were the ones responsible for the FCC Radio Lockdown.
The EU imposed software radio lockdown, there was a public consultation about it where most of the answers from industry and users were negative, but they kept on pushing for it.
The same kind of consultation happened in the US, with the same kind of ignorance.
The EU imposed software radio lockdown, there was a public consultation about it where most of the answers from industry and users were negative, but they kept on pushing for it.
The same kind of consultation happened in the US, with the same kind of ignorance.
Same thing with the FCC.
We could have cooperative use of the airwaves and true open-source, open-hardware radios, but there's too much money to be made auctioning spectrum and assuring that most of it is is 99.99% unutilized. Would be horrible if people had cheap 5-10W handhelds that use advanced codecs, or could run 5mw of power on any part of the spectrum.
We could have cooperative use of the airwaves and true open-source, open-hardware radios, but there's too much money to be made auctioning spectrum and assuring that most of it is is 99.99% unutilized. Would be horrible if people had cheap 5-10W handhelds that use advanced codecs, or could run 5mw of power on any part of the spectrum.
I never run anything that can't be flashed with OpenWrt!
A lot of MikroTik devices run OpenWRT just fine. I never really see the point in it - RouterOS is an excellent.. well, router OS.
No doubt about the quality, I already have some Routerboards around and will be purchasing some of their ceiling APs and a couple switches at the new house soon. All the worries come from the software being closed; I'm not painting Mikrotik evil, it's just that closed software and firmware are the ultimate place in which place spyware, especially where all software is FOSS. Routers and firewalls are the devices seeing every single piece of data on the network, thus is mandatory for them to be trustworthy, a quality the good reputation of their manufacturer can indeed suggest, but not guarantee.
> Routers and firewalls are the devices seeing every single piece of data on the network, thus is mandatory for them to be trustworthy
If you're transmitting (and trusting) plaintext data around, you have bigger issues that just the trustworthiness of your router.
If you're transmitting (and trusting) plaintext data around, you have bigger issues that just the trustworthiness of your router.
I try to avoid proprietary OSes.
Not going to run that locked down crap on my network.
Nice work. Pity all the images in the article are broken.
apologies, website migration and resource issue. Images and links are fixed now
and also the GitHub repo with the presentation link is a 404.
Slides are here, their link on the paghe to the PDF did not work https://margin.re/content/files/2022/11/Pulling_MikroTik_int...
Some guy yesterday was trying to crack this on /g/ board with ChatGPT. Search the threads archive and you might find something related to this code.
[deleted]
Should it really require reverse engineering to figure out how a Linux box passes packets?
I gave up on Ubiquiti and bought a tiny $120 router ($140 with a nice metal case) that's a NanoPi R6s. Pretty impressive specifications. 8GB ram, 8 cores (4 fast Ax76 and 4 slow A55s) and no fans. Has 2x2.5GBe and 1x1GBe for networking. I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS. I didn't bother cross compiling, it's plenty fast for native compiles.
I've been impressed so far. It compiles Rust about half as fast as my Quad core Xeon server from years ago, and is SEVEN times faster than a RPi 4 8GB! I have an Apple M1 mini around that manages the same compile in 13 minutes. A nice bonus is neither network or storage is USB attached, unlike the RPi.
Burned it in overnight, running all 8 cores flat out, with no problems. Haven't decided what to use, maybe shorewall or just raw nftables/iptables.
Similarly there's 4 and 6 port 2.5 Gbe boxes with various N5000 and N6000 celerons at around $200. ServerTheHome has reviewed many of them, they seem to be evolving nicely, better cases, better heat sinks, better 2.5Gbe Intel chips, etc. Run *WRT, pfsense, or whatever else floats your boat. STH even demod running the firewall under proxmox on one of these cheap 4-6 port 2.5GBe widgets. Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).
Guess I'm just getting less trusting in my old age.