USB-C Hubs and Ethernet (2020)(lucumr.pocoo.org)
lucumr.pocoo.org
USB-C Hubs and Ethernet (2020)
https://lucumr.pocoo.org/2020/7/6/usb-c-network-hubs/
35 comments
You cannot exploit this in a network which uses non-broken switches - pause frames must not be forwarded to other ports [1].
But vendors who make home switches often ignore standards.
1. https://en.wikipedia.org/wiki/Ethernet_flow_control#Pause_fr...
But vendors who make home switches often ignore standards.
1. https://en.wikipedia.org/wiki/Ethernet_flow_control#Pause_fr...
Thanks for the info.
This malfunctioning behaviour doesn't seem that uncommon though. Here [1] is another story from 2016 with the same root cause.
I'd like to know how widespread that non-standard behaviour really is.
I could imagine that many small businesses, e.g. cafes, restaurants etc, use cheap consumer switches to manage their guest wifi. Those would seem to be vulnerable.
Another thing: Maybe this is nonsense, but couldn't you spoof PAUSE frames coming from another MAC address than your own, e.g. on wifi? If that's possible, you could disrupt service for selected other network participants even on well-functioning switches (provided you know the victim's MAC address).
[1] http://jeffq.com/blog/the-ethernet-pause-frame/
HN discussion: https://news.ycombinator.com/item?id=12339902
This malfunctioning behaviour doesn't seem that uncommon though. Here [1] is another story from 2016 with the same root cause.
I'd like to know how widespread that non-standard behaviour really is.
I could imagine that many small businesses, e.g. cafes, restaurants etc, use cheap consumer switches to manage their guest wifi. Those would seem to be vulnerable.
Another thing: Maybe this is nonsense, but couldn't you spoof PAUSE frames coming from another MAC address than your own, e.g. on wifi? If that's possible, you could disrupt service for selected other network participants even on well-functioning switches (provided you know the victim's MAC address).
[1] http://jeffq.com/blog/the-ethernet-pause-frame/
HN discussion: https://news.ycombinator.com/item?id=12339902
PAUSE frames are only for the link in question, you should never look at the source address, only the port that the frames were received on
I have USB 3.0 and 3.1 hubs; I am not surprised about this article. USB 3.0/3.1 hubs are fickle as heck! And I noticed that they seem to emit some kind of EMF around the port which means they are unshielded. It rendered some USB dongle non-functional until I unplug the hub.
Now I have Anker 3.1 Mini-Dock (without the ethernet), it works well but it has issues with HDMI. For whatever the reason, the system think that the HDMI is plugged in the hub whereas it is not. So, It cause a strange issue with multimonitor since the system believes that there is a second monitor plugged to it. But yet it couldn’t show the second monitor in the display setting but it insists that it is plugged.
Now I have Anker 3.1 Mini-Dock (without the ethernet), it works well but it has issues with HDMI. For whatever the reason, the system think that the HDMI is plugged in the hub whereas it is not. So, It cause a strange issue with multimonitor since the system believes that there is a second monitor plugged to it. But yet it couldn’t show the second monitor in the display setting but it insists that it is plugged.
I don't know if it was just my hub or the crappy router that Spectrum gives me, but whenever I used the ethernet on my hub with my laptop, it killed the ethernet connection to the desktop.
I've had good luck with a Thunderbolt 4 Hub from Anker as of late. The hub has a power button of its own, and when the hub is turned off it turns off the attached USB-C-to-Ethernet adapter.
over under on when USB-C will finally "just work"?
five years ago I would have said five years
now I'd say 10
five years ago I would have said five years
now I'd say 10
This has very little to do with USB-C. The same problem can be observed using a classic USB-A to ethernet adapter when connected to a class USB-A hub that has its own power supply.
The problem is the 'feature' of some USB Ethernet controllers where they keep the ethernet link active when the host disconnects. As described in the article, it's the PAUSE frame that some switches forward to all devices on the network causing them to... pause sending data. The link stays up but nothing happens. Perhaps the idea was that you get a fast connection resume, but it's not like ethernet handshaking takes a lot of time so they might as well simply disconnect the ethernet link when the USB ethernet controller notices the host computer has disconnected.
The problem is the 'feature' of some USB Ethernet controllers where they keep the ethernet link active when the host disconnects. As described in the article, it's the PAUSE frame that some switches forward to all devices on the network causing them to... pause sending data. The link stays up but nothing happens. Perhaps the idea was that you get a fast connection resume, but it's not like ethernet handshaking takes a lot of time so they might as well simply disconnect the ethernet link when the USB ethernet controller notices the host computer has disconnected.
No, the problem is some broken switches forwarding PAUSE frames to other ports, in violation of the 802.3 spec. It's perfectly compliant for the adapter to send PAUSE frames when it can't empty the incoming packet queue, but that should only pause the single port on the switch.
While that is of course technically true, we are living in an imperfect world and USB Ethernet controller implementations should probably be aware of those millions of switches out there that use a handful of switch chips with known bad implementations.
Keeping to whatever the spec supports is nice, but if the world doesn't work that way you still get a broken product at the end of the day.
Instead of pause, it should (at least have the option to) disconnect. Ethernet link training isn't a computationally expensive and lengthy process anymore, at least not from the user's perspective. Between plugging it in and moving your hands to your input device L1 and L2 are up before you even get a chance to look at your display with L3 being available pretty quickly thereafter.
Keeping to whatever the spec supports is nice, but if the world doesn't work that way you still get a broken product at the end of the day.
Instead of pause, it should (at least have the option to) disconnect. Ethernet link training isn't a computationally expensive and lengthy process anymore, at least not from the user's perspective. Between plugging it in and moving your hands to your input device L1 and L2 are up before you even get a chance to look at your display with L3 being available pretty quickly thereafter.
> Perhaps the idea was that you get a fast connection resume
This seems unlikely for the reasons you listed. Since USB is a polled bus and has shared bandwidth, implementing the flow control is likely very beneficial in case the bus can't temporarily handle the full Ethernet speed.
So, a USB-to-Ethernet gizmo just monitors its internal Ethernet-to-USB queue and if it starts to overflow, it attempts to flow control the switch. But, if no USB host is connected, no USB data requests ever arrive, the queue overflows and never gets cleared...
There may also be some WoL/USB remote wakeup stuff at play, which may be why these ICs do not drop the link on USB host disconnect by default. In fact, I wonder if messing with those settings on the host side could remediate the issue...
This seems unlikely for the reasons you listed. Since USB is a polled bus and has shared bandwidth, implementing the flow control is likely very beneficial in case the bus can't temporarily handle the full Ethernet speed.
So, a USB-to-Ethernet gizmo just monitors its internal Ethernet-to-USB queue and if it starts to overflow, it attempts to flow control the switch. But, if no USB host is connected, no USB data requests ever arrive, the queue overflows and never gets cleared...
There may also be some WoL/USB remote wakeup stuff at play, which may be why these ICs do not drop the link on USB host disconnect by default. In fact, I wonder if messing with those settings on the host side could remediate the issue...
I meant fast resume on the ethernet side, not the USB side. While the (often) 8051 cores do work for things like magic packets and ASF, it's often not really exposed to the user or kernel unless some device-specific driver is used. Some devices have windows-specific drivers that come with device manager properties tabs for extra settings, but then they hide it in some unclear magic packet handling and energy saving setting instead of a host-baed link control setting (the ethernet link control usually is still there).
The weird thing is that there is a whole lot of cool stuff you can do with the on-chip firmware like mDNS proxy for offline devices, but it's all so hidden and vague that nobody bothers to do a whole lot about it.
The weird thing is that there is a whole lot of cool stuff you can do with the on-chip firmware like mDNS proxy for offline devices, but it's all so hidden and vague that nobody bothers to do a whole lot about it.
2 months before USB-D hits consumer devices.
tl;dr: cheap hubs with ethernet failed, dedicated Apple ethernet adapter worked. Author still isn't sure exactly why.
TBF, hubs in general seem to fail regardless of brand. See my other comment about Dell hubs vs docking stations. A model of ‘hubs are cheap because they fail frequently’ is a poor one. Hubs should be cheap because they can’t deliver power and device connections or they have limited port types.
I have been consistently blown away at just how bad usb hubs really are. The market is absolutely saturated with garbage, but it seems like even the high quality ones aren't really that good.
It's the network switch that failed there. Apple adapter works only because it has less features.
I just went through exactly this. I’d unplug my laptop, walk downstairs and suddenly the WiFi would stop working.
To troubleshoot, I’d plug into the hub to have Ethernet connectivity, and the network would seem to work fine. Took me weeks of guessing before considering that a powered-but-disconnected USB hub could be the culprit.
Odd that this is an issue across so many vendors. Perhaps they share a chip, but still, it reads as a mistake on the Hub as well as the Switch side of things.
To troubleshoot, I’d plug into the hub to have Ethernet connectivity, and the network would seem to work fine. Took me weeks of guessing before considering that a powered-but-disconnected USB hub could be the culprit.
Odd that this is an issue across so many vendors. Perhaps they share a chip, but still, it reads as a mistake on the Hub as well as the Switch side of things.
[deleted]
USB-C hubs that I’ve used have been garbage. They’ve all crapped out after less than a year or so of use. The Dell USB-C docking stations I’ve had have been flawless. Not sure what the diff is as my Dell puck USB-C hub sucked (went through 2 of them in a year), but the docking stations are solid.
The one difference I see is that the docking stations are meant to be independently powered and I think it has its own fan. I’m sure the internals are different too, but I don’t know how. Curious if anyone knows more about the difference between the hardware/firmware on the hubs vs the stations.
The one difference I see is that the docking stations are meant to be independently powered and I think it has its own fan. I’m sure the internals are different too, but I don’t know how. Curious if anyone knows more about the difference between the hardware/firmware on the hubs vs the stations.
I have used two Lenovo Thunderbolt docks, a Cable Matters, and a Dell; they all had some odd behavior but the Dell was by far the best.
I think the use of Display Port a plays a big factor. The Cables Matters and Dell docks only had dual HDMI and had far fewer issues.
I think the use of Display Port a plays a big factor. The Cables Matters and Dell docks only had dual HDMI and had far fewer issues.
Belkin TB3 Pro dock customer here. Is it garbage? no. Does it like to go wrong in new and mysterious ways every couple of days? yes
My Dell hub keeps randomly turning off the DP monitors. Cooling it hasnt helped, and it does it with both 1 or 2 monitors connected and on various laptops and OSs
The article is about crappy network switches though, the USB-C hub did nothing wrong :P
Note that this doesn't seem to actually be related to USB-C, but rather to a quirk of how the docking station the author was using behaved when not connected to a computer. This could just as easily happen with any dock. But maybe USB-C and Apple's dongle life has made such docks more common.
I'm not sure how this relates to Apple. I have a Thinkpad, and still vastly prefer the one-cable-solves-all for monitor/power/peripherals.
This has happened with other ethernet controllers too. It's about the fact that they can send pause frames when the host disconnects while the power is still on.
You can also do this with USB-A, get an USB-A controller that sends pause frames, plug it into a powered USB hub and connect to your computer. Unplug, watch the ethernet link still being online and suddenly the network dies. Mostly happens with Realtek USB Ethernet controllers.
This is a firmware feature so technically you can turn it off.
If you want to try it with PCI or PCIe: get a powered backplane or bifurcation board or thunderbolt-to-pci bridge. Connect/use as normal then disconnect the host while leaving power on. Same problem happens with controllers that send pause frames.
You can also do this with USB-A, get an USB-A controller that sends pause frames, plug it into a powered USB hub and connect to your computer. Unplug, watch the ethernet link still being online and suddenly the network dies. Mostly happens with Realtek USB Ethernet controllers.
This is a firmware feature so technically you can turn it off.
If you want to try it with PCI or PCIe: get a powered backplane or bifurcation board or thunderbolt-to-pci bridge. Connect/use as normal then disconnect the host while leaving power on. Same problem happens with controllers that send pause frames.
About 10 years ago, I was working in a startup that had a hadoop cluster on some bare metal servers in a datacenter. At one point when the dataset size and sustained traffic between nodes grew to a particular level, the network throughput crashed to a trickle, until the hadoop jobs were all canceled, then network was fine a minute later.
It turned out to be pause frames! The switches were too eager to propagate them, or something? The fix was to just disable them on all NICs - TCP can handle packet loss and rate control pretty well in those conditions anyway (gigabit switched ethernet).
(I can't take any credit for the work debugging and mitigating this issue.)
Anyway, it seems this is one of those bugs that pops back up from time to time... people try to implement this bit of the spec along with the rest, but don't realize how it's really supposed to behave under stress, and it's never tested under this kind of stress...
It turned out to be pause frames! The switches were too eager to propagate them, or something? The fix was to just disable them on all NICs - TCP can handle packet loss and rate control pretty well in those conditions anyway (gigabit switched ethernet).
(I can't take any credit for the work debugging and mitigating this issue.)
Anyway, it seems this is one of those bugs that pops back up from time to time... people try to implement this bit of the spec along with the rest, but don't realize how it's really supposed to behave under stress, and it's never tested under this kind of stress...
I use a USB-C/Thunderbolt dock from CalDigit that seemingly has the same problem. Luckily they released a firmware update that supposedly fixes it several months back.
I'm guessing a lot of the manufacturers just ended up using the same chip for their docks/hubs, and some manufacturers just don't care if there was a firmware update from the chip vendor.
I'm guessing a lot of the manufacturers just ended up using the same chip for their docks/hubs, and some manufacturers just don't care if there was a firmware update from the chip vendor.
A year ago spend relatively long time troubleshooting my home network problem and the root cause was the same: 1. USB-C hub with an Ethernet port send pause frames when it is powered but a laptop is disconnected 2. TP-Link Wi-Fi router broadcasts pause frames to all ports which causes all connected devices to stop sending traffic.
TP-Link (or more specifically Atheros AR8327 used inside it) here is likely breaks a IEEE standard (may be even the most basic one - 801.2d), but when no-one send pause frames this goes unnoticed.
As workaround now don't connect power to the hub and instead connect charger to a laptop directly (if a hub is powered you can connect only one cable to s laptop to get both - power and external USB+Ethernet).
TP-Link (or more specifically Atheros AR8327 used inside it) here is likely breaks a IEEE standard (may be even the most basic one - 801.2d), but when no-one send pause frames this goes unnoticed.
As workaround now don't connect power to the hub and instead connect charger to a laptop directly (if a hub is powered you can connect only one cable to s laptop to get both - power and external USB+Ethernet).
Author here: so since I wrote this blog post I get an email once every few weeks about this. Apparently that blog post ranks quite well now for the keywords that frustrated users are typing into Google.
I have now heard that this problem happens with a lot of devices and the most common element in most of them is that people also have netgear switches on their network but not exclusively.
The most reliable workaround is to ensure that the USB-C adapter with an ethernet card in it is powered off when the laptop goes to sleep. Many do that if they are not also used for USB-C PD or whatever it's called.
I have now heard that this problem happens with a lot of devices and the most common element in most of them is that people also have netgear switches on their network but not exclusively.
The most reliable workaround is to ensure that the USB-C adapter with an ethernet card in it is powered off when the laptop goes to sleep. Many do that if they are not also used for USB-C PD or whatever it's called.
The what now? I've never heard of "PAUSE" frames before, but those sound like a feature just waiting to be exploited.
Do we know how common the behaviour to forward the frames is? This sounds like it could bring down any vulnerable public network of their choice by just logging on and sending that special packet...