Intel I219-LM Running at ~60% of Maximum Speed Due to Linux Driver Bug(phoronix.com)
phoronix.com
Intel I219-LM Running at ~60% of Maximum Speed Due to Linux Driver Bug
https://www.phoronix.com/news/Intel-i219-LM-Linux-60p-Fix
46 comments
It's nice to see them giving this some attention. I hope they also look into the I225-V at the myriad of different negotiation bugs that change with each firmware revision on 2.5gb networks that also vary with different offloading features enabled/disabled including EEE. Each 2.5gb switch also exhibits different behavior which is probably not Intel's fault.
When people say you need the third hardware revision to get functional performance, and it still doesn’t work, you should conclude it’s poorly designed. The fact that these chips were being sold with old hardware revisions also seems anti-consumer.
Agreed, Intel gave up and released the I226 because their I225 brand was so tarnished. I removed the I225 from my server after I noticed the driver crashing during heavy loads (Ubuntu 22.04). For USB I've been having success with the rtl8156bg from juplink.
I226 has the same bugs as i225, so I'm pretty sure it's just another hardware revision of the terrible i225 but renamed to try and trick consumers.
I got tired of fighting that NIC, and got a Realtek 2.5g one.
All my problems gone, life is good.
All my problems gone, life is good.
I've had good experience with Realtek NICs too, not 2.5g yet but their 10/100/1000 models. I suspect that's because they use a far simpler design with not much in the way of fancy features[1]. They also tend to benchmark slower with higher CPU usage, but I'll take slightly slower but stable over fast but intermittent connectivity.
[1] Ironically, in this case the Intel has to have a fancy feature, intended for increasing performance, disabled in order to increase performance.
[1] Ironically, in this case the Intel has to have a fancy feature, intended for increasing performance, disabled in order to increase performance.
I've heard good things about Realtek's 2.5G nics, and bad things about Intel's; I've seen speculation that underlying cause is cross compatability between the two 2.5g specs that were merged into the standard and that most 2.5g switches use chips from companies that were behind the spec Intel was not behind.
For 1G realteks, I've had ok luck with them, but I do see some issues. I recall having some problem with them while running the Linux tree drivers, but I've since switched to all FreeBSD. With my current realtek NICs and FreeBSD 13.1, I have the choice of the kernel driver where sometimes the NIC will stop processing packets, and the NIC acknowledges reset, but doesn't actually immediately reset and sometimes processes old packets after resetting, resulting in wild writes and bad behavior. There's a vendor driver, which doesn't seem to get into that bad state, but it does enable ethernet PAUSE frames which are an abomination. The vendor driver is full of undocumented magic values and there's no public documentation on the NICs at all, so figuring out what to frob to make the NIC not get stuck, but not send out PAUSE frames would be an exercise in frustration, that I'm not willing to do.
Either way, the interrupt design is deficient: there's a shared interrupt for rx, tx and administration, and the status register doesn't really work right either --- it's possible for the host and device to disagree about what irqs were acknowleged and then things will get stuck (that doesn't seem to be the FreeBSD driver issue though; you can work around the stuck status communications by just assuming something probably happened aftet a few seconds, or checking for descriptor progess on rx and tx on any interrupt, etc). Having only a single interrupt means you can't meaningfully process incomming packets and finished outgoing descriptors in parallel which makes it hard to get full throughput in both directions simultaneously.
Anyway --- if they work for you, great. I'm going to avoid them where practical, and be careful with them elsewhere.
For 1G realteks, I've had ok luck with them, but I do see some issues. I recall having some problem with them while running the Linux tree drivers, but I've since switched to all FreeBSD. With my current realtek NICs and FreeBSD 13.1, I have the choice of the kernel driver where sometimes the NIC will stop processing packets, and the NIC acknowledges reset, but doesn't actually immediately reset and sometimes processes old packets after resetting, resulting in wild writes and bad behavior. There's a vendor driver, which doesn't seem to get into that bad state, but it does enable ethernet PAUSE frames which are an abomination. The vendor driver is full of undocumented magic values and there's no public documentation on the NICs at all, so figuring out what to frob to make the NIC not get stuck, but not send out PAUSE frames would be an exercise in frustration, that I'm not willing to do.
Either way, the interrupt design is deficient: there's a shared interrupt for rx, tx and administration, and the status register doesn't really work right either --- it's possible for the host and device to disagree about what irqs were acknowleged and then things will get stuck (that doesn't seem to be the FreeBSD driver issue though; you can work around the stuck status communications by just assuming something probably happened aftet a few seconds, or checking for descriptor progess on rx and tx on any interrupt, etc). Having only a single interrupt means you can't meaningfully process incomming packets and finished outgoing descriptors in parallel which makes it hard to get full throughput in both directions simultaneously.
Anyway --- if they work for you, great. I'm going to avoid them where practical, and be careful with them elsewhere.
and there's no public documentation on the NICs at all
There is plenty if you search around, and they're one of the recommended NICs if you want to write an OS/driver because of that; they also show up in virtual form in various VM hosting software due to their simplicity:
https://wiki.osdev.org/RTL8139
https://wiki.osdev.org/RTL8169
https://www.iitg.ac.in/asahu/cs421/RealTek.pdf
There is plenty if you search around, and they're one of the recommended NICs if you want to write an OS/driver because of that; they also show up in virtual form in various VM hosting software due to their simplicity:
https://wiki.osdev.org/RTL8139
https://wiki.osdev.org/RTL8169
https://www.iitg.ac.in/asahu/cs421/RealTek.pdf
Ok yes. I've done that actually. Pretty sure I used that pdf (or similar, although I think the one I used was marked confidential). But a pdf from 2002 tha is hosted from a university, not the vendor, isn't really up to the standard of public documentation. It wouldn't be so bad if their vendor drivers had comments or even sensible names for values when they frob the hardware, but it's just hex values with no explanations.
It's an OK NIC to work with hobby wise, although I quickly ran into issues with the status register, and I can only hope one day I'll get enough throughput to overwelm the nic. And then I'll move to an Intel nic of which I have several.
It's an OK NIC to work with hobby wise, although I quickly ran into issues with the status register, and I can only hope one day I'll get enough throughput to overwelm the nic. And then I'll move to an Intel nic of which I have several.
Same. From lore and experience, Intel is more stable than Realtek for 1G on Linux.
Intel managed to break some of their older consumer 1G NICs under Linux pretty reliably while adding support for newer NICs, and failed to fix them for a very long time.
At the end, Debian reverted Intel's patches and shipped a modified version which probably didn't enable some features on the newer NICs, but didn't break the older ones in the process.
It's ironic that the underdog Realtek has much more reliable cards which doesn't shudder under constant load and/or very long uptime scenarios. Realtek won't cut in server scenarios, but Intel's and Broadcom's server class NICs are completely different beasts when compared to Intel's consumer NICs, too.
Wish they had given some more attention to them while developing their drivers.
At the end, Debian reverted Intel's patches and shipped a modified version which probably didn't enable some features on the newer NICs, but didn't break the older ones in the process.
It's ironic that the underdog Realtek has much more reliable cards which doesn't shudder under constant load and/or very long uptime scenarios. Realtek won't cut in server scenarios, but Intel's and Broadcom's server class NICs are completely different beasts when compared to Intel's consumer NICs, too.
Wish they had given some more attention to them while developing their drivers.
This is the classic Sutherland "wheel of reincarnation" cycle -- a function (commonly graphics acceleration or network connectivity) will be provided by the CPU, which is found to be a bottleneck so the function is offloaded to a dedicated device, which then becomes a bottleneck as the CPU receives more dedicated focus on performance improvement than some bespoke, single-purpose offload engine.
The Realtek NIC embedded in my Asus motherboard hits maybe 600mbps with a tailwind. It only has 1 receive queue and doesn't support interrupt coalescing. Junk.
I would totally do that but I have a few mini-PC's that have the I225-V and I can't swap them out. I've used USB3.2->Ethernet dongles as work arounds until I could find a switch that was happy with it. Same with the latest Protectli firewalls, they also have the I225-V. For those I have to use ethtool to force a specific speed which is not ideal. I'm thinking mini-PC's are maybe a mistake in this generation of NICs.
Same boat here. Care to share what switch you ended up landing on? I'm still searching… :'(
They all have issues but the worst issues I ran into were on the TrendNet switches. The NIC's would start to get into carrier reset loops or sometimes would reset with no predictable pattern. I225-V + Trendnet seem to amplify each others bugs. Those switches also have problems with the I219-V. I threw those switches away.
The TP-Link and Netgear switches had the least issues for me in that I only had to force a particular speed to avoid sometimes negotiating at 100mb. I am not sure people will notice the negotiating bugs unless their family are heavy bandwidth users. I noticed it because of streaming and gaming at the same time plus I shut down my machines at night so I have a coin-flip chance of negotiating incorrectly every day.
The TP-Link and Netgear switches had the least issues for me in that I only had to force a particular speed to avoid sometimes negotiating at 100mb. I am not sure people will notice the negotiating bugs unless their family are heavy bandwidth users. I noticed it because of streaming and gaming at the same time plus I shut down my machines at night so I have a coin-flip chance of negotiating incorrectly every day.
Which mini PC's have the issue?
I do not have an all-inclusive list of which mini-PC's have the I225-V's. I have a few versions of the AMD Ryzen's and they all have them. The latest Protectli firewalls also mini-PC's have them. Several of the off-brand mini-PC's on Amazon have them. Some of the Intel NUCs may have them. If buying one of these online I suggest first posting a customer question about what NICs they have.
> AMD Ryzen
Damn. That's pretty much a requirement for any upcoming home lab gear here, so I'd better be careful about this aspect when researching.
Thanks. :)
Damn. That's pretty much a requirement for any upcoming home lab gear here, so I'd better be careful about this aspect when researching.
Thanks. :)
> is probably not Intel's fault
consensus is Intel went down the shitter quality wise, just one example https://www.youtube.com/watch?v=DXNyHFOWx_k
consensus is Intel went down the shitter quality wise, just one example https://www.youtube.com/watch?v=DXNyHFOWx_k
Could be. Most of the folks I interacted with in the past have retired. I think I am just going to revert back to full tower PC's for anything I do, even firewalls and appliances so that I am not locked into what may be abandonware. I was hoping to use less power and take up less space with the mini-PC's but in hind-sight that may have been a mistake.
Fujitsu S920 seems to be a good option for a mini PC that is cheap and reliable. Not very fast, but has usb 3.0 and a x4 pcie slot. It has been very stable as my wifi access point. First heard about it on Wolfgang's channel.
you will want to look forward to upgrading your Linux kernel build soon... [...] due to a regression introduced back in 2020.
If anything, that message seems to be implying to not upgrade if everything is already working well.
A bit tangential, but ever since they came out with the '217 I've thought this is one of the worst-named products and could never remember what the actual letter is. Here it's an uppercase I, but even Intel seems to think it's a lowercase L (for LAN?) sometimes:
https://www.intel.com/content/www/us/en/support/articles/000...
If anything, that message seems to be implying to not upgrade if everything is already working well.
A bit tangential, but ever since they came out with the '217 I've thought this is one of the worst-named products and could never remember what the actual letter is. Here it's an uppercase I, but even Intel seems to think it's a lowercase L (for LAN?) sometimes:
https://www.intel.com/content/www/us/en/support/articles/000...
That makes me wonder how much waste there is, just because there is something broken without anyone noticing. Recently a customer had two nearly identical systems behaving completely different in regards to performance. It turned out it simply was some bios setting.
> That makes me wonder how much waste there is, just because there is something broken without anyone noticing.
I'd say a lot. Not seldom when I go over to a colleague I notice their laptop is breathing hard. "Oh yeah, so annoying! It's been doing that for a while now". Check Task Manager and sure enough, some process is stuck sucking 100% of a core... 5 CPU days worth.
Even I find it difficult to spot this, with my 16 cores. I got an efficient cooler, so no fan spins up noticeably. Randomly check Task Manager, oh explorer.exe is pegging a core, and has been using 15 days worth of CPU... Gee thanks!
I'd say a lot. Not seldom when I go over to a colleague I notice their laptop is breathing hard. "Oh yeah, so annoying! It's been doing that for a while now". Check Task Manager and sure enough, some process is stuck sucking 100% of a core... 5 CPU days worth.
Even I find it difficult to spot this, with my 16 cores. I got an efficient cooler, so no fan spins up noticeably. Randomly check Task Manager, oh explorer.exe is pegging a core, and has been using 15 days worth of CPU... Gee thanks!
I have an Intel I219-V adaptor on a Windows desktop. I had a hell of a time trying to figure out the problem when I upgraded to Verizon FIOS internet; upload speeds were abysmal. The chipset has a hardware bug making it incompatible with the Verizon Optical Network Terminal. The workaround is to disable checksum offloading in the network driver advanced settings. It sounds like this is the linux version of the same thing.
One would expect Intel to do periodic regression tests on Linux kernel releases on their hardware. A lot of Intel customers use Linux, and they may switch to other vendors if Intel hardware has bad Linux support.
They do, and they provide some of the most extensive regression testing capabilities provides to kernel development.
But to be fair, this is an old NIC with a hardware bug that requires an odd workaround disabling an otherwise very useful performance feature. Missing that edge case is understandable.
But to be fair, this is an old NIC with a hardware bug that requires an odd workaround disabling an otherwise very useful performance feature. Missing that edge case is understandable.
Hmmm, that doesn't really seem to gel though, unless their extensive regression testing is a recent development?
Saying that because at some point, that NIC was a new thing, not an old one. So extensive regression testing should have picked it up then, yeah?
Saying that because at some point, that NIC was a new thing, not an old one. So extensive regression testing should have picked it up then, yeah?
https://www.intel.com/content/www/us/en/developer/topic-tech...
It is not a new thing, but no test is exhaustive and they must prioritize hardware tested for practical reasons. This particular test would need to check if the working NIC could reach full gigabit speeds in TX, as everything else checked out.
Regardless, it cannot be said that they do not have regression testing or are irresponsible in that sense, as they provide important services to the community in exactly this area.
It is not a new thing, but no test is exhaustive and they must prioritize hardware tested for practical reasons. This particular test would need to check if the working NIC could reach full gigabit speeds in TX, as everything else checked out.
Regardless, it cannot be said that they do not have regression testing or are irresponsible in that sense, as they provide important services to the community in exactly this area.
> This particular test would need to check if the working NIC could reach full gigabit speeds in TX, as everything else checked out.
To me, that sounds weird.
My expectation is that every non-dodgy manufacturer of a network device would test the features they're developing.
So, when (say) offloading support of some variety is newly added to a chip set, whoever is adding that support would be testing it at the full speed of the device.
If the developers aren't verifying the things they're adding work (somehow), there's something wrong. :(
To me, that sounds weird.
My expectation is that every non-dodgy manufacturer of a network device would test the features they're developing.
So, when (say) offloading support of some variety is newly added to a chip set, whoever is adding that support would be testing it at the full speed of the device.
If the developers aren't verifying the things they're adding work (somehow), there's something wrong. :(
Not a network guy, but couldn't this be detected (for any other card too) by running some mandatory tests?
Interesting that in a era of 100+ Gbit Ethernet we seem to need the offload features working just right to get even 1 Gbit.
Well in this case the bug is that the feature doesn't work and needs to be disabled.
But generally speaking you want all the offload you can get. For 1G, it allows pushing power efficiency much further. For 100G, it allows you to spend time on something other than parsing and writing packets.
But generally speaking you want all the offload you can get. For 1G, it allows pushing power efficiency much further. For 100G, it allows you to spend time on something other than parsing and writing packets.
Does this mean if someone with an I219-LM did a regular file transfer they would only get 67MiB/s instead of 112MiB/s? Surely this would be immediately noticeable and not take 3 years to fix? Was it just a throughput bug or a packets-per-second bug? Did any other conditions need to be present for it to trigger the slowdown? So many questions... I just don't understand how this can go unnoticed. If you buy a 1Gbps NIC and it performs <1Gbps for a simple file transfer, you would think that's not something Intel would miss, or ignore if reported given how easy it would be to reproduce.
This dumb chip caused myself and my colleagues some headaches when the driver kept getting deleted as suspected-malicious but reinstalled right away by Windows, causing the whole lot to bounce like on some giant trampoline.
Reading the comments here, we aren't the only ones taken by surprise and/or aback.
Reading the comments here, we aren't the only ones taken by surprise and/or aback.
Doesn't disabling offloading (of anything) result in increased CPU usage when transferring lots of data? Is it now that it's faster but uses more CPU, or is it actually faster with the same CPU usage (a "proper" fix)?
There are probably a couple of independent things:
(1) HW GRO, most often LRO (TCP)
(2) SW GRO aka GRO
Hardware GRO was typically called LRO for TCP and is often disabled in practice. One reason is XDP, with XDP some offloads need to stay off. Another is device quirks: while idea of waiting for more data, and passing larger packets into kernel soiunds nice, it also means that device must do packet concatenation. Think about ECN or selective-acks. The concatentation is lossy and unless the flow is perfect, it might not be possible to concatenate packets. Furthermore devices often had issues with being too aggrressive and loosing important data.
In Linux there is a software approach that does kindof the same thing, but in software, called GRO, and this is often sufficient. It also supports more than just TCP.
In other words - having LRO disabled is common.
(1) HW GRO, most often LRO (TCP)
(2) SW GRO aka GRO
Hardware GRO was typically called LRO for TCP and is often disabled in practice. One reason is XDP, with XDP some offloads need to stay off. Another is device quirks: while idea of waiting for more data, and passing larger packets into kernel soiunds nice, it also means that device must do packet concatenation. Think about ECN or selective-acks. The concatentation is lossy and unless the flow is perfect, it might not be possible to concatenate packets. Furthermore devices often had issues with being too aggrressive and loosing important data.
In Linux there is a software approach that does kindof the same thing, but in software, called GRO, and this is often sufficient. It also supports more than just TCP.
In other words - having LRO disabled is common.
I wonder if this is an issue with the BSDs too. Sometimes they peek at Linux for these things.
It isn't. It's a pure driver bug with all of these. There's a pending patch to clear the MSI registers on disconnect that I'm planning on driving to testing tomorrow. I hounded them for 4 months by sending full stacks with symbols and they don't have any hardware in the lab (which is very common now, unfortunately)...
If you want to recover, set a ifdown script to `rmmod igc && modprobe igc` and you will never panic or hang, just have a longer interface bounce time. I'm running this on a 6 nic system which is completely egregious, but it hasn't crashed / hung / panic'd in the field since doing that.
If you want to recover, set a ifdown script to `rmmod igc && modprobe igc` and you will never panic or hang, just have a longer interface bounce time. I'm running this on a 6 nic system which is completely egregious, but it hasn't crashed / hung / panic'd in the field since doing that.
This is a hardware bug so I would expect it to affect all OSes. Turning off TSO is counterintuitive so drivers tend to turn it on by default.
Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring. I am not saying BDS is better, just a few things are quite different vs Linux.
> Probably not, BSD network drivers were a bit more advanced than Linux ones as some companies (like Netflix) invested time and effort in optimizations and monitoring.
for those 3 nics they use.
I doubt some random 2.5Gbit nic is on their list of stuff to fix
for those 3 nics they use.
I doubt some random 2.5Gbit nic is on their list of stuff to fix