Struck by a Thunderbolt(lightbluetouchpaper.org)
lightbluetouchpaper.org
Struck by a Thunderbolt
https://www.lightbluetouchpaper.org/2019/02/26/struck-by-a-thunderbolt/
61 comments
Thunderbolt is an awkward implementation, but a good product. There is no alternative offering that provides what it does.
Reading the rant you linked, there seems to be no real content or references to information. It mostly complains about USB-PD negotiation of the Thunderbolt alt-mode (which doesn't have anything to do with thunderbolt), and that the controller is proprietary (which sucks, sure, but so is most other controllers in the system).
IOMMU not being on by default as it should be is a motherboard manufacturer problem. Vulnerabilities found with it active should have CVE's created so that they can get fixed, like any other safety issue.
As someone using Thunderbolt on my Linux laptop, I can inform you that it works quite well.
(EDIT: Also note that Intel technically has promised to make Thunderbolt open and royalty-free, although they have missed their deadline on that promise.)
Reading the rant you linked, there seems to be no real content or references to information. It mostly complains about USB-PD negotiation of the Thunderbolt alt-mode (which doesn't have anything to do with thunderbolt), and that the controller is proprietary (which sucks, sure, but so is most other controllers in the system).
IOMMU not being on by default as it should be is a motherboard manufacturer problem. Vulnerabilities found with it active should have CVE's created so that they can get fixed, like any other safety issue.
As someone using Thunderbolt on my Linux laptop, I can inform you that it works quite well.
(EDIT: Also note that Intel technically has promised to make Thunderbolt open and royalty-free, although they have missed their deadline on that promise.)
I mean, I want 10 gbit+ peripherals and short of using PCIe extenders, what choice do I have? USB 3? Wait no, USB 3.1 gen 1, wait no USB 3.1 gen 2! (Yes, I know there's different Thunderbolt versions, too).
If Dell's going to do silly things like solder Wifi chips down on the new laptops, then you're damn right I'm going to be looking for Thunderbolt for fast connectivity out of a very expensive laptop.
If Dell's going to do silly things like solder Wifi chips down on the new laptops, then you're damn right I'm going to be looking for Thunderbolt for fast connectivity out of a very expensive laptop.
…renamed to USB 3.2 gen 1, 2 and the upcoming 2x2 as of yesterday.
What we really need is USB 4.7 gen 1.2^3
What was the motivation for building it this way, instead of by more directly exposing PCIe? There must be some upside to they way it’s designed, no?
PCIe as a phy is not designed for long distances or complex topologies. A lot of motherboards, including high-end server boards, cannot even route the PCIe lanes from CPU to PCIe socket without there being connectivity issues, which manifest as things like PCIe retransmits (which result in plummeting performance).
Designing a purpose-built phy/mac, designed to handle long links and complex topologies, is the obvious thing to do.
Designing a purpose-built phy/mac, designed to handle long links and complex topologies, is the obvious thing to do.
I think to have the flexibility to have various topologies of many devices (daisy-chained, etc.) from a single port.
Probably also timing etc. - I would think making it packet based is probably easier to make work over longer cables than raw PCI-e.
The bit I really don’t get is why they keep all the specs secret. What do they think they have to gain? All that it has meant is less adoption...
Probably also timing etc. - I would think making it packet based is probably easier to make work over longer cables than raw PCI-e.
The bit I really don’t get is why they keep all the specs secret. What do they think they have to gain? All that it has meant is less adoption...
PCIe is also packet-based BTW [1]. The benefit from the custom protocol is being able to run PCIe and DisplayPort (and I guess networking which no one cares about) over the same cable at the same time with dynamic bandwidth allocation. USB-C has alternate modes but they're not as powerful. In theory PCIe alternate mode over USB-C should be possible and it would be much simpler than Thunderbolt but nobody has tried to build it.
I imagine the spec doesn't really exist because Intel has been too lazy to properly write it (I wouldn't be surprised if Thunderbolt is basically defined as whatever the original Ridge controller implemented) and it's probably also gross. Adoption seems to be limited by cost, not by secrecy (thinking back to Firewire, it was open but there were very few controller chips for it).
[1] Check out PCIe over RS-232: https://www.youtube.com/watch?v=QMiubC6LdTA
I imagine the spec doesn't really exist because Intel has been too lazy to properly write it (I wouldn't be surprised if Thunderbolt is basically defined as whatever the original Ridge controller implemented) and it's probably also gross. Adoption seems to be limited by cost, not by secrecy (thinking back to Firewire, it was open but there were very few controller chips for it).
[1] Check out PCIe over RS-232: https://www.youtube.com/watch?v=QMiubC6LdTA
> What was the motivation for building it this way, instead of by more directly exposing PCIe?
I think when intel was doing Light Peak, they didn't have IOMMU, which in my limited knowledge seems to be a solution to at least some of what is wrong with normal hot plugging PCIe (aside from somewhat complicated hardware design, I imagine). Of course IOMMU was clearly never implemented correctly, so all was for naught.
I think when intel was doing Light Peak, they didn't have IOMMU, which in my limited knowledge seems to be a solution to at least some of what is wrong with normal hot plugging PCIe (aside from somewhat complicated hardware design, I imagine). Of course IOMMU was clearly never implemented correctly, so all was for naught.
But there's no alternative.
USB
USB has more more limited bandwidth constraints and, IIRC, has much higher CPU overhead and overall higher latency.
It also just can't do things that Thunderbolt can. TB to the end user is a free PCIe slot for Firewire, 10GbE, eGPU, etc. The fastest USB can't do that. I get that it has issues but it's still incredibly valuable.
"We built a fake network card that is capable of interacting with the operating system in the same way as a real one, including announcing itself correctly, causing drivers to attach, and sending and receiving network packets. To do this, we extracted a software model of an Intel E1000 from the QEMU full-system emulator and ran it on an FPGA."
I wanted to call that out; that's nifty!
I wanted to call that out; that's nifty!
> we extracted a software model of an Intel E1000 from the QEMU full-system emulator and ran it on an FPGA."
I need to look up what they did for this - lots of the tools promise the ability to turn a pile of C into SystemVerilog, but require a bit of manual massaging.
Edit: "used the Arm Cortex A9 CPU on an Intel Arria 10 FPGA" well that's boring and much simpler.
It does not surprise me that drivers are not secure against maliciously-constructed messages from the "hardware". Most drivers are tested in a fairly limited fashion, and nobody's even got the infrastructure to fuzz them like this.
(There are lots of FPGA-on-PCIe boards, but they tend to be horribly expensive. This one is almost $5000. https://www.digikey.com/products/en?mpart=DK-DEV-10AX115S-A&...)
I need to look up what they did for this - lots of the tools promise the ability to turn a pile of C into SystemVerilog, but require a bit of manual massaging.
Edit: "used the Arm Cortex A9 CPU on an Intel Arria 10 FPGA" well that's boring and much simpler.
It does not surprise me that drivers are not secure against maliciously-constructed messages from the "hardware". Most drivers are tested in a fairly limited fashion, and nobody's even got the infrastructure to fuzz them like this.
(There are lots of FPGA-on-PCIe boards, but they tend to be horribly expensive. This one is almost $5000. https://www.digikey.com/products/en?mpart=DK-DEV-10AX115S-A&...)
There was actually an interesting talk on fuzzing drivers in the session before ours:
https://www.ndss-symposium.org/ndss-paper/periscope-an-effec...
We're working on porting to a cheaper FPGA-on-PCIe board, but it's still ~EUR 800.
We're working on porting to a cheaper FPGA-on-PCIe board, but it's still ~EUR 800.
Have you looked at PicoEVB ($300 mini-PCIe x1) or Numato Aller ($400 M.2 x4) with Xilinx Artix-7 FPGAs?
We need to use an Intel Arria or Stratix FPGA due to the 'config bypass' feature we use to allow QEMU to implement its own PCIe config registers. Xilinx, Lattice and Cyclone FPGAs don't support this as far as I can tell.
If you later move beyond QEMU, these OSS projects may be of interest.
https://github.com/texane/vpcie
> virtual environment consisting of QEMU, LINUX and GHDL glued alltogether by a small TCP based protocol. It allows PCIE devices to be implemented as standard userland processes, answering actual PCIE requests coming from QEMU. It supports PCIE configuration headers, requests, memory read/write operations and MSI. Different abstractions are provided to simplify the implementation of PCIE devices.
https://github.com/KastnerRG/riffa & http://kastner.ucsd.edu/wp-content/uploads/2014/04/admin/fpl...
> RIFFA (Reusable Integration Framework for FPGA Accelerators) is a simple framework for communicating data from a host CPU to a FPGA via a PCI Express bus. The framework requires a PCIe enabled workstation and a FPGA on a board with a PCIe connector. RIFFA supports Windows and Linux, Altera and Xilinx, with bindings for C/C++, Python, MATLAB and Java. On the software side there are two main functions: data send and data receive ... Users can communicate with FPGA IP cores by writing only a few lines of code.
https://github.com/texane/vpcie
> virtual environment consisting of QEMU, LINUX and GHDL glued alltogether by a small TCP based protocol. It allows PCIE devices to be implemented as standard userland processes, answering actual PCIE requests coming from QEMU. It supports PCIE configuration headers, requests, memory read/write operations and MSI. Different abstractions are provided to simplify the implementation of PCIE devices.
https://github.com/KastnerRG/riffa & http://kastner.ucsd.edu/wp-content/uploads/2014/04/admin/fpl...
> RIFFA (Reusable Integration Framework for FPGA Accelerators) is a simple framework for communicating data from a host CPU to a FPGA via a PCI Express bus. The framework requires a PCIe enabled workstation and a FPGA on a board with a PCIe connector. RIFFA supports Windows and Linux, Altera and Xilinx, with bindings for C/C++, Python, MATLAB and Java. On the software side there are two main functions: data send and data receive ... Users can communicate with FPGA IP cores by writing only a few lines of code.
It is getting increasingly easy to buy off-the shelf FPGA boards that plug into exposed computer ports. Fomu is particularly unobtrusive.
But not as likely to be a security vulnerability since it's strictly USB. Hmm, well it depends very much on the probably very horrible USB driver..
Hmmm, on Windows whenever I plug in a new Thunderbolt device, it requests permission to activate it. I assumed this was for this very reason. If I hand someone my computer logged in and unlocked, it's as good as compromised.
Author here. We don't break that prompt, however there are some problems. First is that Macs have no prompt at all, so you just need to use a device on the whitelist and you're approved.
Second, the prompt says 'you attached a thing, do you want to allow it?'. The name is often not descriptive, eg 'CalDigit TS3' in our example, so users can't tell what rights are being asked for. As we've found on mobile, when apps pop up prompts about permissions users are conditioned to agree. Also, users can be deceived by putting a sticker on the malicious device with the name on the prompt - the writing on the plastic agrees with the prompt, so the device must be legit. An attacker can also play social engineering tricks - "say 'approve' to enable fast charging" for example.
Finally, it's possible to swap out the PCIe devices behind a Thunderbolt bridge without re-prompting. We took a commercial Thunderbolt dock, removed the PCB with the dock's PCIe peripherals and replaced it with another PCIe card. Windows did not notice that the device had changed and didn't prompt again.
Second, the prompt says 'you attached a thing, do you want to allow it?'. The name is often not descriptive, eg 'CalDigit TS3' in our example, so users can't tell what rights are being asked for. As we've found on mobile, when apps pop up prompts about permissions users are conditioned to agree. Also, users can be deceived by putting a sticker on the malicious device with the name on the prompt - the writing on the plastic agrees with the prompt, so the device must be legit. An attacker can also play social engineering tricks - "say 'approve' to enable fast charging" for example.
Finally, it's possible to swap out the PCIe devices behind a Thunderbolt bridge without re-prompting. We took a commercial Thunderbolt dock, removed the PCB with the dock's PCIe peripherals and replaced it with another PCIe card. Windows did not notice that the device had changed and didn't prompt again.
Ah, so if I may reiterate, the issues are:
1. Users cannot tell if a device is malicious before authorizing it.
2. Even if the user could tell, the device can be made malicious post-authorization.
Both situations seem like quite universal and unsolvable issues, in that you will never know the exact behavior of a device unless you tear it down atom by atom.
(Of course, #2 could be made harder if sub-devices were made part of the authorization, but it will always be possible to make a device malicious given physical access to it, even if a simple PCIe swap is no longer possible.)
However, assuming proper fix for the bugs discovered, the IOMMU should make DMA access pointless, reducing any attack to be a simple case of a malicious device that can at most be malicious in the execution of its own functions (e.g. eavesdropping or traffic modification). Why would any peripheral receive device mappings outside of the buffers needed to operate it?
1. Users cannot tell if a device is malicious before authorizing it.
2. Even if the user could tell, the device can be made malicious post-authorization.
Both situations seem like quite universal and unsolvable issues, in that you will never know the exact behavior of a device unless you tear it down atom by atom.
(Of course, #2 could be made harder if sub-devices were made part of the authorization, but it will always be possible to make a device malicious given physical access to it, even if a simple PCIe swap is no longer possible.)
However, assuming proper fix for the bugs discovered, the IOMMU should make DMA access pointless, reducing any attack to be a simple case of a malicious device that can at most be malicious in the execution of its own functions (e.g. eavesdropping or traffic modification). Why would any peripheral receive device mappings outside of the buffers needed to operate it?
The situation hasn't improved significantly in the years since the same vulnerability appeared with Firewire.
There are so many ways to mitigate or stop this, and keep the user in control. Android, at least, knows to prevent privileged device connections with the screen locked.
There are so many ways to mitigate or stop this, and keep the user in control. Android, at least, knows to prevent privileged device connections with the screen locked.
This device management functionality is also available for linux desktops and best combined with enabling the IOMMU at boot time.
https://gitlab.freedesktop.org/bolt/bolt
It also seems to be available on most distros.
https://pkgs.org/download/bolt
https://gitlab.freedesktop.org/bolt/bolt
It also seems to be available on most distros.
https://pkgs.org/download/bolt
The same vulnerability goes back even farther with nearly all 90's laptops having one or more PCMCIA/PC Card/CardBUS slots.
Or when they had mini-PCI/mini-PCIe slots under a cover on the bottom for network cards etc.
As it says in the article, it can also be done with desktops where you can just pop off the case panel and plug a malicious device into the PCIe bus.
I think the biggest issue with Thunderbolt specifically is that it works over the USB-C port that is also used for charging - so you could make a device that looks like a charger but also attacks the device (maybe with a cellular modem in it to exfiltrate data or something!), making it way easier to trick users to plug it in themselves.
As it says in the article, it can also be done with desktops where you can just pop off the case panel and plug a malicious device into the PCIe bus.
I think the biggest issue with Thunderbolt specifically is that it works over the USB-C port that is also used for charging - so you could make a device that looks like a charger but also attacks the device (maybe with a cellular modem in it to exfiltrate data or something!), making it way easier to trick users to plug it in themselves.
Aside from the security vulnerability, it's very cool that you can now get a high bandwidth connection between an FPGA and a system via a cable instead of a card.
I have already been using Cypress FX-3 to get 5 Gbps USB connection to an FPGA, but this is better.
I have already been using Cypress FX-3 to get 5 Gbps USB connection to an FPGA, but this is better.
If I'm reading this correctly, this means that physical access to any modern machine can be leveraged to root access. Encrypted storage that hasn't been unlocked might continue to be secure; that's about it.
This is the “cost” of DMA with untrusted external connections. If you’re basically allowing some device to sit on your PCI bridge, that’s an insane amount of trust of course. But we live with it because we like performance.
There are EFI/BIOS level “workarounds” like on Dell laptops: they have a setting to only negotiate thunderbolt with approriate dell docks.
Sadly, their thunderbolt dock is entirely garbage because they used a really crappy USB3 controller which has the habit of dropping devices and corrupting CRC checksums on Ethernet packets. Additionally, this defies the _point_ of thunderbolt itself. But if we assume we can disable thunderbolt capability while the host OS is running then that’s already a huge win.
FWIW I already do this with USB, the ports are disabled until I run a command to enable them in Linux. Because I’m one of those “paranoid” types.
There are EFI/BIOS level “workarounds” like on Dell laptops: they have a setting to only negotiate thunderbolt with approriate dell docks.
Sadly, their thunderbolt dock is entirely garbage because they used a really crappy USB3 controller which has the habit of dropping devices and corrupting CRC checksums on Ethernet packets. Additionally, this defies the _point_ of thunderbolt itself. But if we assume we can disable thunderbolt capability while the host OS is running then that’s already a huge win.
FWIW I already do this with USB, the ports are disabled until I run a command to enable them in Linux. Because I’m one of those “paranoid” types.
As long as Thunderbolt is enabled. If I'm not mistaken disabling Thunderbolt completely should be a viable workaround. It's essentially the same situation we had with Firewire: Just blacklist the driver and all should be good.
As DisplayPort Alternate Mode for USB-C is nowadays used a lot to connect to external displays, instead of the approach Apple chose to tunnel DisplayPort over Thunderbolt, Thunderbolt is probably not even relevant for most users outside of the Apple universe.
As DisplayPort Alternate Mode for USB-C is nowadays used a lot to connect to external displays, instead of the approach Apple chose to tunnel DisplayPort over Thunderbolt, Thunderbolt is probably not even relevant for most users outside of the Apple universe.
> Thunderbolt is probably not even relevant for most users outside of the Apple universe.
Actually not the case -- the TB chipsets are from Intel and promoted by them.
TB drives are pretty fast and great to use if you can afford them. TB external GPUs are useful for ML too (not sure if anyone is using them for actual rendering; that's outside my field of experience).
Actually not the case -- the TB chipsets are from Intel and promoted by them.
TB drives are pretty fast and great to use if you can afford them. TB external GPUs are useful for ML too (not sure if anyone is using them for actual rendering; that's outside my field of experience).
To an extent - many laptops and nearly all desktops don't have Thunderbolt at all. TB external GPUs are probably the main use-case for the technology today (people indeed do use them for rendering)
No statement can overestimate how critical this vulnerability is.
Taking your laptop or phone away for ten minutes allows somebody with a malicious USB-C device to read out everything without rebooting or leaving any mark that anything had been done, bypassing filesystem encryption and passwords.
Taking your laptop or phone away for ten minutes allows somebody with a malicious USB-C device to read out everything without rebooting or leaving any mark that anything had been done, bypassing filesystem encryption and passwords.
How does have a computer that is turned off affect this? Take a MacPro laptop with an encrypted SSD. If the Mac is turned off, and the attack is attempted after a fresh reboot without knowing the password, is the laptop still susceptible?
Yeah, it would have to be someone 'borrowing' or stealing a laptop that's running and has you logged in.
If I'm reading it correctly, any newer Mac with T2 chip and full disk encryption would not allow data to be exfiltrated unless a user were logged in.
If I'm reading it correctly, any newer Mac with T2 chip and full disk encryption would not allow data to be exfiltrated unless a user were logged in.
I would think itd matter in which order devices are initialized in, for example, if you had a compromised device connected, is it initialized before your keyboard is so you can enter your full disk encryption password.
Id wager youd probably be safe because I think the bare USB keyboard support is provided by the UEFI BIOS and no OS level drivers would be initialized yet (e.g. why you can use basic USB keyboard and mouse support in the BIOS), but this is my mildly educated speculation. This is definitely not my area of expertise.
Id wager youd probably be safe because I think the bare USB keyboard support is provided by the UEFI BIOS and no OS level drivers would be initialized yet (e.g. why you can use basic USB keyboard and mouse support in the BIOS), but this is my mildly educated speculation. This is definitely not my area of expertise.
It seems to leverage the DMA features of Thunderbolt to dump the data, so if no sensitive data is in memory, then it wouldn't be leakable.
Fpr the same reason, note that if the computer was booted and locked, it would be affected.
Fpr the same reason, note that if the computer was booted and locked, it would be affected.
so if you're carrying a susceptible device through a border check, it would be a good idea to shut the laptop down first. after that, it's all about how much pain you can take for refusing to login after they pull out the $5 wrench.
what about using their device to install something onto your system rather than retrieving data from it? is it possible to do one of those mischievous flash roms from something like this?
what about using their device to install something onto your system rather than retrieving data from it? is it possible to do one of those mischievous flash roms from something like this?
Forget borders, it could be a problem for domestic flights in the US. I've had the TSA demand I give them my laptop for special screening. Presumably they 'thought' it was a bomb, they didn't ask me to turn it on. But had it been turned on, it would have been vulnerable.
Are you sure IOMMU is not enabled by default in at least some versions of Linux? I thought RHEL 6 had it enabled.. (because I remember having to disable it..)
May be a warning not to plug anything except eGPU and power in that slot. But then you have apple only have that slot ... Still need eGPU in any case
Hmm, seems that plenty at intel haven't been thinking about security at all for a while... sleeping at the helm?
Intel gave us the IOMMU but OSes haven't been using it.
Intel didn't stop using IOMMU support for product segmentation until a month after Windows 10 was released. It's really no surprise that operating systems have been slow to make use of a feature that is still unavailable on a large part of the install base.
I have been warned against turning on IOMMU because the code to operate it is so little exercised, and therefore buggy. A crashed kernel is secure only in that it stops running.
On the other hand IOMMU helps protecting system memory not just against malicious but also faulty devices that could otherwise write anywhere.
And people who use GPU passthrough rely on IOMMUs, so it's not like those code paths are not being exercised.
And people who use GPU passthrough rely on IOMMUs, so it's not like those code paths are not being exercised.
Using IOMMU through the "userspace PCI" vfio driver, as done in pass-through scenarios, is a different path than using it through the DMA API.
IOMMU is also exercised by Xen PCI passthrough (e.g. QubesOS network and USB driver VMs) and SR-IOV (e.g. AWS Nitro networking).
The problem isn't bugs in the IOMMU, it's bugs or performance issues in using the DMA API with an IOMMU.
Well, they have used it a little, but it’s hard to use it enough.
You need to ensure that different network cards can’t see each other’s buffers, for example, and that will not be an easy change to most OSes.
You need to ensure that different network cards can’t see each other’s buffers, for example, and that will not be an easy change to most OSes.
So.. odd question for everyone. Does this mean, all of a sudden, we, potentially, have full access pass to plain text version of everything ( including things various driver vendors do not want you to know ? )?
It doesn't grant you any more access than root on the system already does.
I think it's worth pointing out, the way this firm is going about the research and working with the vendors is commendable.
The “firm” in this case is the Computer Lab at the University of Cambridge, UK.
> “MacOS fixed the specific vulnerability we used to get administrator access in macOS 10.12.4 in 2016”
USB-C Macbook Pros we’re only announced on Oct 27, 2016. Not sure what version of 10.12 they shipped with, but seems like it was a pretty fast fix.
USB-C Macbook Pros we’re only announced on Oct 27, 2016. Not sure what version of 10.12 they shipped with, but seems like it was a pretty fast fix.
The actual paper cites using devices with Thunderbolt 2 ports (iMac, Mac Mini both have Thunderbolt 2 via DisplayPort) to experiment on as well. It sounds like the vulnerability is in the Thunderbolt protocol in general, not just the USB-C implementation of it.
The vulnerability applies to all DMA-capable devices (ie Thunderbolt, PCIe, wifi and NVMe chips on mobiles, on-chip peripherals like mobile basebands). Thunderbolt 3 makes drive-by attacks easier, but in principle the attack could be carried out from any DMA device (subject to practical limitations like reverse engineering of firmware). Our earlier attacks were done with PCIe and Thunderbolt 2.
The 2015 Macbook had USBC, and the Macbook Pro got Thunderbolt in 2011.
https://twitter.com/whitequark/status/1097777102563074048
> Thunderbolt, uh, does not expose PCIe lanes directly. Thunderbolt is an MPLS-like packet switching network that can encapsulate PCIe TLPs over a PHY and MAC without a spec, chips without documentation, and software with barely any support. […] Ah, and let's not forget that the absurdly complicated MPLS-like system with topology written in EEPROMs in every node that, as far as I can tell, you have to bitbang via the half-configured packet path (what the actual fuck?), well, it's very general. You can describe a lot of different networks in it. But of course the Linux driver has two or three hardcoded ones, including indexes into the EEPROMs on the way, that it recognizes and loads a hardcoded configuration for. The entire thing is made -even worse than USB-, which is an achievement.
It pains me to hear people complaining about the lack of Thunderbolt in AMD Ryzen Mobile laptops, and wishing for it in the next generation now that the spec is no longer Intel-exclusive in some way. Lack of Thunderbolt is a feature.