Ghost in the ethernet optic(blog.benjojo.co.uk)
blog.benjojo.co.uk
Ghost in the ethernet optic
https://blog.benjojo.co.uk/post/smart-sfp-linux-inside
67 comments
> Ben, do you plan on playing around some more with these to find out if they can be hacked to run your own OS?
Nope, I'm probably going to use it as a remote monitoring/out of band device when the time comes. The device is useful enough in it's slightly closed source state than as a device for hacking (and maybe a bricking risk)
Nope, I'm probably going to use it as a remote monitoring/out of band device when the time comes. The device is useful enough in it's slightly closed source state than as a device for hacking (and maybe a bricking risk)
Last year on another transceiver (QSFP28) teardown[0] I was surprised to find out that transceivers not marketed as "smart" also have SOCs inside them to regulate internal temperature. I had always thought the devices were "dumber" and never bothered to look inside.
So programmable CPUs in your transceivers might be more common than one would think.
[0] https://twitter.com/kwf/status/1470508119725805570
So programmable CPUs in your transceivers might be more common than one would think.
[0] https://twitter.com/kwf/status/1470508119725805570
(random thought) It surprises me how, despite working in the tech industry for over a 15 years now, I struggled in following the details of this blog post (which is well written). It's so impressive how things got very complex over time, and how verticalize the role of an Engineer is becoming.
I feel like things have gotten a lot simpler over time. Examples:
1. There's only one relevant type of optics (Q)SFP(+)(28) instead of many incompatible ones. Line cards have become uncommon.
2. Everything runs Linux so he can just run tcpdump instead of some vendor specific monitor command.
3. 10G ethernet and 56G infiniband are dirt cheap so now there’s a large online community for homelabs that use data center hardware. Nobody needs to be a network guy to run a data center.
1. There's only one relevant type of optics (Q)SFP(+)(28) instead of many incompatible ones. Line cards have become uncommon.
2. Everything runs Linux so he can just run tcpdump instead of some vendor specific monitor command.
3. 10G ethernet and 56G infiniband are dirt cheap so now there’s a large online community for homelabs that use data center hardware. Nobody needs to be a network guy to run a data center.
> Nobody needs to be a network guy to run a data center.
the complexity from networking at a DC mainly comes from scale.
running enterprsie/dc hardware in a homelab is vastly different from running it in production.
the complexity from networking at a DC mainly comes from scale.
running enterprsie/dc hardware in a homelab is vastly different from running it in production.
>running enterprsie/dc hardware in a homelab is vastly different from running it in production.
This applies to all of software. Enterprise and "home" diverged quite a lot. Back in the 2000s running a webserver at home was roughly similar to doing it for a company. Nowadays they are nothing alike with all the load balancing, caching etc. etc.
This applies to all of software. Enterprise and "home" diverged quite a lot. Back in the 2000s running a webserver at home was roughly similar to doing it for a company. Nowadays they are nothing alike with all the load balancing, caching etc. etc.
your argument seems to hinge on the perspective on "scale"
i.e. enthusiast homelabs from today have sufficient capacity/performance to accomodate production scenarios ("at scale") from 20y ago.
besides the need for appreciation of ones work, everybody, all the time, is cooking with water.
i.e. enthusiast homelabs from today have sufficient capacity/performance to accomodate production scenarios ("at scale") from 20y ago.
besides the need for appreciation of ones work, everybody, all the time, is cooking with water.
what i actually meant to say was that multitenancy makes networking complex. (and nearly any network is multi-tenant once it reaches a certain scale).
Also, depending on the services which run on the network, more complex technology like MPLS/VXLAN-EVPN is needed.
Also, depending on the services which run on the network, more complex technology like MPLS/VXLAN-EVPN is needed.
I'm looking to upgrade my 1G home network to a mix of 10+2.5G (including PoE) Which "large online communities" where you thinking of?
The main one is https://forums.servethehome.com/index.php
Also /r/homelab
Search eBay for “10g card” and google the cheapest model number to find the rest :)
Also /r/homelab
Search eBay for “10g card” and google the cheapest model number to find the rest :)
While researching EEPROM flashing on AliExpress SFPs, I discovered that many of these do support SSH. I suspect that even the smaller ones might have tiny Linux onboard.
Example:
https://forum.mikrotik.com/viewtopic.php?t=116346
https://github.com/hwti/G-010S-A
Example:
https://forum.mikrotik.com/viewtopic.php?t=116346
https://github.com/hwti/G-010S-A
I am surprised they used debian and not Yocto or buildroot, for an embedded device. Would anyone speculate on why debian would be preferred?
(Author of post)
They advertise this as a network appliance, so it makes sense that they use Debian since being able to install packages via things like apt makes the whole experience much easier. A lot of career network engineers are not that savvy with sysadmin type roles, so something like buildroot would alienate their market a little
They advertise this as a network appliance, so it makes sense that they use Debian since being able to install packages via things like apt makes the whole experience much easier. A lot of career network engineers are not that savvy with sysadmin type roles, so something like buildroot would alienate their market a little
ARMv7, with 512M of RAM, isn't that embbeded and you win a full os whit ready to use tested packages, package managers, and boot system, it saves a lot of times and expertice, maybe they need to opt out of it if they want to have better termals, or more optimization.
I've had to work with embedded devices in the past (develop an alternative firmware for $THING), and irgh I hated buildroot which was the original firmware's base. Way too much effort to compile stuff yourself... a simple "debootstrap" and installation of your private packages is way easier to get stuff up and running. And when you run into some package you need, you directly run "apt install" on the device itself, add the install command to the script that creates your firmware image and that's it.
The best thing is, whatever issues you run into will already have been solved a million times before by the wide Debian community and whatever remains can be solved by a quick hop into #debian on whatever IRC network they migrated to.
The best thing is, whatever issues you run into will already have been solved a million times before by the wide Debian community and whatever remains can be solved by a quick hop into #debian on whatever IRC network they migrated to.
There are two main uses for this kind of thing.
- Legit: debugging/monitoring. Other legit uses are theoretically possible but the device has severe limitations that likely make them impractical or unwise.
- Surreptitious: This is clearly where the value proposition of this device lies. An optic could be "unknowingly" swapped out on an interesting link to snoop on or infiltrate a network.
Swapping out optics in a large network is not uncommon as they do fail. More often they are swapped out as a troubleshooting step where the original optic may not even be bad. This way log messages indicating link flap and replacement of an optic could likely go unnoticed.
- Legit: debugging/monitoring. Other legit uses are theoretically possible but the device has severe limitations that likely make them impractical or unwise.
- Surreptitious: This is clearly where the value proposition of this device lies. An optic could be "unknowingly" swapped out on an interesting link to snoop on or infiltrate a network.
Swapping out optics in a large network is not uncommon as they do fail. More often they are swapped out as a troubleshooting step where the original optic may not even be bad. This way log messages indicating link flap and replacement of an optic could likely go unnoticed.
Ehh this chonker doesn’t look very surreptitious. I can only think of one other device that sticks out even further in front of the switch. Said device is a very old SFP+ to 10G 100m copper module with a huge heatsink and an external power brick!
That aside, I could imagine a gov’t agency programming these things and forcing ISPs to put them on customer ports. No need to wait for a maintenance window or free up rack space.
That aside, I could imagine a gov’t agency programming these things and forcing ISPs to put them on customer ports. No need to wait for a maintenance window or free up rack space.
This is one reason why I am a big proponent of network engineers physically visiting their equipment periodically. In large networks that may never happen, data center techs (often not even your own employees) are the only ones who ever see the racks in person.
Even if the 3rd party tech notices something unusual about the optic the certainly aren't going to touch it without a ticket and will probably not even mention it.
Even if the 3rd party tech notices something unusual about the optic the certainly aren't going to touch it without a ticket and will probably not even mention it.
This is the one they sell publicly to anyone at a relatively affordable price.
If they make ones that run cooler and look the same as a normal optic, they might not be so forthcoming about it.
If they make ones that run cooler and look the same as a normal optic, they might not be so forthcoming about it.
Looking at the pictures this thing isn't exactly the typical SFP form factor, it protrudes ~5 cm or so over a normal SFP. So if you swap one of these out it's going to stick out like the proverbial sore thumb.
Who says you can see the side they swapped the optic on? For datacenter cross connect links, its not uncommon for the other side of the link to be somewhere you can't see/have access to.
That's a fair point though in that case someone with physical access can do essentially anything with the cable anyway - this "just" makes it easier.
I wonder if it is GPL compliant, including the ability to update the Linux kernel/etc.
There's no requirement in GPL 2 (the one the kernel uses) to be able to update the kernel.
This is debatable - the license is pretty clear about it being a requirement, but it has been patchily enforced:
> The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable [emphasis mine].
Even in the infamous TiVo case, customers of TiVo had complete access to modify the Linux kernel or any other GPL binary - the proprietary TiVo software would just refuse to run if any modification had been applied this way.
> The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable [emphasis mine].
Even in the infamous TiVo case, customers of TiVo had complete access to modify the Linux kernel or any other GPL binary - the proprietary TiVo software would just refuse to run if any modification had been applied this way.
Yes, there's no requirement for the sfp to boot any specific kernel, thus no requirement
"to update the Linux kernel/etc"
If they aren't using a stock kernel with drivers already in the kernel, then clearly they have to distribute that code (which might just be a shim like with nvidea), but not boot from it -- it's not GPL3
"to update the Linux kernel/etc"
If they aren't using a stock kernel with drivers already in the kernel, then clearly they have to distribute that code (which might just be a shim like with nvidea), but not boot from it -- it's not GPL3
See my other comments above, the GPLv2 requires installation/updates too.
Interestingly, I found this scholarly article [0] claiming that is actually a misunderstanding of the text of the GPLv2, and an ahistorical reading/discussion of this problem.
For example, here is a citation they have from the (then) chief lawyer of the FSF:
> TiVo is a provider of hardware and software …. Our concern with them is that they have rights as users, but they should respect the rights of the users to whom they sell. Having a personal video recorder … which won't run software if you modify the box … is not user-respecting conduct. (TiVo) complied with GPL 2 by the skin of its teeth.
On the other hand, it seems it is actually pretty hard right now to tell what exactly TiVo was preventing at the time. There are some articles, like the one you cited, that claim TiVo devices would check the signature of the kernel and other software, and refuse to run their own proprietary software. However, other articles (some cited in this paper) claim that in fact TiVo devices would not boot a modified kernel. This seems more plausible to me, given the FSF's reaction and additional langauge added to GPLv3 specifically to prevent what TiVo was doing - modifications which would prohibit the latter case but not the former (were the kernel to ever adopt GPLv3).
[0] https://www.researchgate.net/publication/353194088_Does_GPLv...
For example, here is a citation they have from the (then) chief lawyer of the FSF:
> TiVo is a provider of hardware and software …. Our concern with them is that they have rights as users, but they should respect the rights of the users to whom they sell. Having a personal video recorder … which won't run software if you modify the box … is not user-respecting conduct. (TiVo) complied with GPL 2 by the skin of its teeth.
On the other hand, it seems it is actually pretty hard right now to tell what exactly TiVo was preventing at the time. There are some articles, like the one you cited, that claim TiVo devices would check the signature of the kernel and other software, and refuse to run their own proprietary software. However, other articles (some cited in this paper) claim that in fact TiVo devices would not boot a modified kernel. This seems more plausible to me, given the FSF's reaction and additional langauge added to GPLv3 specifically to prevent what TiVo was doing - modifications which would prohibit the latter case but not the former (were the kernel to ever adopt GPLv3).
[0] https://www.researchgate.net/publication/353194088_Does_GPLv...
This post I already posted elsewhere in the thread debunks McCoy's article:
https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t...
https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t...
That article clashes with many of Richard Stallman's claims. In particular, the article claims that the Series 2 TiVos allowed the user to run any modified kernel, but disabled the TiVo proprietary software. In contrast, Stallman has many public statements claiming explicitly that the TiVo devices would fail to boot or shut down immediately if a modified kernel was detected:
> For instance, the Tivo itself is the prototype of tivoisation. The Tivo contains a small GNU/Linux operating system, thus, several programs under the GNU GPL. And, as far as I know, the Tivo company does obey GPL version 2. They provide the users with source code and the users can then modify it and compile it and then install it in the Tivo. That's where the trouble begins because the Tivo will not run modified versions, the Tivo contains hardware designed to detect that the software has been changed and shuts down. So, regardless of the details of your modification, your modified version will not run in your Tivo. [emphasis mine]
> One major danger that GPLv3 will block is tivoization. Tivoization means computers (called “appliances”) contain GPL-covered software that you can't change, because the appliance shuts down if it detects modified software. The usual motive for tivoization is that the software has features the manufacturer thinks lots of people won't like. The manufacturers of these computers take advantage of the freedom that free software provides, but they don't let you do likewise.
Linus Torvalds, from many public statements about cryptographically signed kernels, shares this same view of what the GPLv2 allows:
> And it’s important to realize that signed kernels that you can’t run in modified form under certain circumstances is not at all a bad idea in many cases. For example, distributions signing the kernel modules (that are distributed under the GPL) that _they_ have compiled, and having their kernels either refuse to load them entirely (under a “secure policy”) or marking the resulting kernel as “Tainted” (under a “less secure” policy) is a GOOD THING.
> For instance, the Tivo itself is the prototype of tivoisation. The Tivo contains a small GNU/Linux operating system, thus, several programs under the GNU GPL. And, as far as I know, the Tivo company does obey GPL version 2. They provide the users with source code and the users can then modify it and compile it and then install it in the Tivo. That's where the trouble begins because the Tivo will not run modified versions, the Tivo contains hardware designed to detect that the software has been changed and shuts down. So, regardless of the details of your modification, your modified version will not run in your Tivo. [emphasis mine]
> One major danger that GPLv3 will block is tivoization. Tivoization means computers (called “appliances”) contain GPL-covered software that you can't change, because the appliance shuts down if it detects modified software. The usual motive for tivoization is that the software has features the manufacturer thinks lots of people won't like. The manufacturers of these computers take advantage of the freedom that free software provides, but they don't let you do likewise.
Linus Torvalds, from many public statements about cryptographically signed kernels, shares this same view of what the GPLv2 allows:
> And it’s important to realize that signed kernels that you can’t run in modified form under certain circumstances is not at all a bad idea in many cases. For example, distributions signing the kernel modules (that are distributed under the GPL) that _they_ have compiled, and having their kernels either refuse to load them entirely (under a “secure policy”) or marking the resulting kernel as “Tainted” (under a “less secure” policy) is a GOOD THING.
I wasn't around then, but I would be surprised if that description of TiVo's actions is accurate. I'm more inclined to believe the blog post from the person who "led the GPLv2 enforcement effort against TiVo" than the definition of tivoization that exists in the popular consciousness, which I expect is a political invention of the community over time. The article says "At the time, TiVo was doing the right thing in providing what the GPLv2 requires — including the ability to reinstall GNU and Linux software onto the actual device" and "TiVo never prevented such reinstallation". There is a whole section "How Discussion Focused on Cryptographic Lockdown Generally" about where the cryptographic lockdown worries came from; it was years after TiVo, during the GPLv3 drafting process. They even link to resources about how to update Linux on TiVo devices, one of them mentions breaking the "encryption" involves modifying the "tivoapp" userspace binary in a way that looks to me like disabling checking of the Linux kernel hash.
Linus is saying that signed Linux kernels are a good thing (and I concur), the situations he was describing there are for Secure Boot based systems, which are explicitly designed to allow for software freedom. IIRC this happens in a couple of ways:
Of course Microsoft controlled Secure Boot isn't the only kind of cryptographic lockdown in use today. The method used on mainstream Android phones is different and I don't know the details but I think it allows wiping the phone and then booting unsigned Linux kernel builds but I don't think it allows the MOK style setup from the PC UEFI world. The Apple M1 devices have yet another system.
Linus is saying that signed Linux kernels are a good thing (and I concur), the situations he was describing there are for Secure Boot based systems, which are explicitly designed to allow for software freedom. IIRC this happens in a couple of ways:
1. the UEFI firmware requirements set by Microsoft require the ability to disable Secure Boot, and ISTR also require or encourage the ability to enroll secondary keys.
2. the shim firmware built by a distro and signed by Microsoft and booted by the UEFI firmware allows a physically present user to enroll secondary keys, and then all the layers beyond shim support verifying things using those keys.
Of course Microsoft controlled Secure Boot isn't the only kind of cryptographic lockdown in use today. The method used on mainstream Android phones is different and I don't know the details but I think it allows wiping the phone and then booting unsigned Linux kernel builds but I don't think it allows the MOK style setup from the PC UEFI world. The Apple M1 devices have yet another system.
And that's the major change from GPLv2 to GPLv3 - explicit prevention of scenarios like TiVo. However, Linus said he really didn't care if companies prevented users from running modified code, only that any changes companies made to the kernel were available. The Linux kernel is explicitly licensed under GPLv2 for that reason.
https://lkml.org/lkml/2007/6/13/289
(hard to believe that 2007 is 15 years ago...)
https://lkml.org/lkml/2007/6/13/289
(hard to believe that 2007 is 15 years ago...)
That is another common misconception. RMS wanted GPLv3 to prevent what TiVo did (breaking proprietary software when you exercise the GPL installation rights), but actually the changes in GPLv3 do not do that and what TiVo did is allowed by the GPLv3.
https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t... https://events19.linuxfoundation.org/wp-content/uploads/2017...
The GPLv2 does not do what Linus wants either - it doesn't require code to be publicly available (only to customers) and it doesn't require them to give code back to upstream. Linus is also against GPLv2 enforcement in general too, even if the GPLv2 installation requirements were not enforced.
https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t... https://events19.linuxfoundation.org/wp-content/uploads/2017...
The GPLv2 does not do what Linus wants either - it doesn't require code to be publicly available (only to customers) and it doesn't require them to give code back to upstream. Linus is also against GPLv2 enforcement in general too, even if the GPLv2 installation requirements were not enforced.
That is a common misconception. Both the FSF (the license authors) and Software Freedom Conservancy have always held the opposite view, and always enforced that view.
https://sfconservancy.org/blog/2021/mar/25/install-gplv2/ https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t... https://events19.linuxfoundation.org/wp-content/uploads/2017...
https://sfconservancy.org/blog/2021/mar/25/install-gplv2/ https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t... https://events19.linuxfoundation.org/wp-content/uploads/2017...
I was about to say the same thing, but I think he’s talking about how GPL2 requires the company to give OP the kernel source code. OP mentioned that he didn’t have the kernel headers for DKMS.
Ah sure, if it's not a stock kernel then yes you'd need to distribute the code for it (although there's no requirement for it to be able boot from any replacement kernel you provide)
I wonder if the kernel's license covers any device tree overlays you create for a board. Technically, the kernel provides the device tree compiler. You write one, compile it, and supply it to the kernel at boot time (or compile it in).
The end goal of Linux for ARM would be to completely avoid needing anything other than the upstream kernel (all drivers and CPU quirks being upstream), the .config for the kernel, and a device tree overlay.
The end goal of Linux for ARM would be to completely avoid needing anything other than the upstream kernel (all drivers and CPU quirks being upstream), the .config for the kernel, and a device tree overlay.
Also there are SFP ONU modules for GPON/EPON, I've been using one (with RTL9601C chipset) for a year and it works great, fiber cable directly to the router or switch, no more shitty ISP ONU & router.
Things like this have been around for nearly (at least?) 10 years, but are not well-known outside of spaces that care about telco-style demarc & OAM systems.
An example from ~2012 would be the RAD MiNID which is available in a handy "sleeve" format where you can use your normal SFP with the smart SFP.
Pretty cool to see a writeup of this sort of thing (and increased vendor representation).
An example from ~2012 would be the RAD MiNID which is available in a handy "sleeve" format where you can use your normal SFP with the smart SFP.
Pretty cool to see a writeup of this sort of thing (and increased vendor representation).
This seems so nice for NSA-like implants...
I think those are designed to avoid detection, so they (having near-unlimited budgets for these sorts of covert things) prefer to go with replacement firmwares and suchlike. They've been known to intercept packages of networking equipment and do all-software implants so that you can't even tell a device has been tampered by looking at it, and if they want to exit they can remove all traces that they were there. Leaving physical hardware in place is the opposite of that.
There are so many places to hide malicious code in any computer or networking system these days, using weird nonstandard obvious hardware like this is pretty amateur.
There are so many places to hide malicious code in any computer or networking system these days, using weird nonstandard obvious hardware like this is pretty amateur.
Perfectly makes sense! But such hardware can be an additional place for their implants... supposing that the hardware is already there. Not that they are supplying it or replacing the passive one with the (clearly detectable) one in question here.
He said
> But such a feature could also be used to create a fake 169.254.169.254 (AWS/Cloud metadata IP address endpoint) and serve requests from it.
Wouldn’t such a thing be impossible if the application is using end-to-end encrypted requests to AWS?
> But such a feature could also be used to create a fake 169.254.169.254 (AWS/Cloud metadata IP address endpoint) and serve requests from it.
Wouldn’t such a thing be impossible if the application is using end-to-end encrypted requests to AWS?
the metadata service is http and not encrypted.
What caught my eye was the pixelated serial numbers. Also, the fact that the part and serial numbers are plainly visible in a few pictures, but not others.
Fascinating! Tho I can’t think of that many use cases where a 1G hub and raspberry pi can’t do the job.
KennyBlanken(4)
Yeah I feel like this would be mostly used for nefarious purposes since it wouldn't be obvious it was there. Other than that I can't think of any way that it's better than just a raspberry pi.
The author mentions that 'In a more premium software package for the smart-sfp you can configure ERSPAN sessions with filters'. Selling a more expensive software package for the SFP would be a reason to lock it down and prevent others from offering competing (including open-source) software.
Another interesting aspect is the communication with the programmable logic. What is implemented in the FPGA? Is it purely signal processing? Is there packet inspection and filtering? Could the communication between the CPU and the FPGA be reverse engineered to provide a driver?
Edit: Ben, do you plan on playing around some more with these to find out if they can be hacked to run your own OS?