Dual-port 100GbE PCI-E 4.0 network cards are now under $1300(broadcom.com)
broadcom.com
Dual-port 100GbE PCI-E 4.0 network cards are now under $1300
https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
56 コメント
Except that's pci-e 3.0 unlike the cards from broadcom and intel.
On the same website you can also find pci-e 4.0 Connectx-5 for slightly more than 1k$ though.
PCIe 3.0 x16 is capable of 36GB/s. A single 100G NIC is only 12.5GB/s.
Yes, if it's a one way traffic flow, but a 100GbE link as used in a router is full duplex. If looking at a traffic chart, you could have 75-80Gbps of traffic inbound and also 70-80Gbps of traffic outbound simultaneously.
Using a smaller traffic example, on a much older Juniper router that has linecards with 20Gbps full duplex to the backplane, each line card can hold two 10GbE PICs without oversubscription.
If you want to have a line-rate-capable 100GbE x86-64 whitebox router using FRR or similar, you want to budget PCI-E bus bandwidth for a maximum of approximately 24GB/s per dual port 100GbE NIC.
Using a smaller traffic example, on a much older Juniper router that has linecards with 20Gbps full duplex to the backplane, each line card can hold two 10GbE PICs without oversubscription.
If you want to have a line-rate-capable 100GbE x86-64 whitebox router using FRR or similar, you want to budget PCI-E bus bandwidth for a maximum of approximately 24GB/s per dual port 100GbE NIC.
[deleted]
I'd really love it if the 10GbE cards would drop under $13 :) And especially the switches to become cheap.
It really feels like we've had 1GbE like forever (it was already standard on my 2004 PowerBook!!). But 10GbE is still hardly affordable.
It really feels like we've had 1GbE like forever (it was already standard on my 2004 PowerBook!!). But 10GbE is still hardly affordable.
I hope all the patents expire soon and that good products at good prices follow quickly. The standard was released in 2006, but I'm not sure how close to that release date the last relevant patent involved was issued.
https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-T
"10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 Gbit/s connections over unshielded or shielded twisted pair cables, over distances up to 100 metres (330 ft)."
https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-T
"10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 Gbit/s connections over unshielded or shielded twisted pair cables, over distances up to 100 metres (330 ft)."
10GBASE-T is unreasonably power hungry. SFP+ passive cables are far superior for short lengths and fiber optics are superior for anything longer. I've got two SFP+ 10GBASE-T transceivers on my home network and they seem to sit around 80°C all the time.
What is pantented about implementations? I’m surprised that the nature of the implementation of a public specification is patentable.
I spent a few years implementing 802.3 physical layer testing. At the conferences it was clear that things were written carefully to not cover implementation, but they went so far in the necessary details that there were only a handful of reasonably performant and economical solutions.
I spent a few years implementing 802.3 physical layer testing. At the conferences it was clear that things were written carefully to not cover implementation, but they went so far in the necessary details that there were only a handful of reasonably performant and economical solutions.
> I’m surprised that the nature of the implementation of a public specification is patentable.
Prepare to be very surprised, I guess? It’s pretty routinely done, eg) look at any of the MPEG standards.
I don’t know what patents, if any, cover 10GigE, but in general IEEE, which governs that standard, will accept Essential Patent Claims in a standard provided the Patent Holder will provide a Letter of Assurance indicating that they will license said patents under FRAND (fair, reasonable, and non-discriminatory) terms – which does not mean free.
Prepare to be very surprised, I guess? It’s pretty routinely done, eg) look at any of the MPEG standards.
I don’t know what patents, if any, cover 10GigE, but in general IEEE, which governs that standard, will accept Essential Patent Claims in a standard provided the Patent Holder will provide a Letter of Assurance indicating that they will license said patents under FRAND (fair, reasonable, and non-discriminatory) terms – which does not mean free.
Yeah I don't think patents will play a major role in the price. WiFi would be a lot more patent-encumbered with technologies like MIMO. Yet it doesn't seem to preclude cheap commodity hardware. My Unifi WiFi 6 AP was only a bit more expensive than the AC one.
What about Mellanox ConnectX-3 for ~30 ? More than 13, but less than 1300
Plus Intel X520 goes for a similar price, I think older Chelsio cards too
Yeap.
Mellanox has one advantage , even though it pcie x8, if you put it to pcie x4 slot, it works nicely (according to specs it downgrades itself).
Intel - not that sure about it
It definitely does, in my PC right now. Why wouldn't it? Not downgrading would probably be an egregious violation of PCIe specs. PCIe is built to work no matter the restrictions. You can connect an x16 gen4 GPU via an adapter to an x1 gen2 miniPCIe slot and it will work.
When I was shopping for one a few months ago, it wasn't explicitly mentioned it X520 specs but mentioned in Mellanox specs.
I know that in the theory it supposed to downgrade itself, but implementations may vary depends on fantasy of designers.
At the end, I just got motherboard with few 10GbE and SPF+ on it.
At the end, I just got motherboard with few 10GbE and SPF+ on it.
You can pick up used HP NC523SFP or NC552SFP (better!) NICs for about $15-$25 each off eBay. The drivers fell out of support on RHEL8/CentOS8 but I think Fedora Server, RHEL7/CentOS7 and Ubuntu Server are fine with them. TrueNAS can make use of them as well. Then all you need is a 10GbE switch that can do bonding. Or, just make yourself a router with a bunch of dual port NICs.
https://www.ebay.com/itm/LOT-OF-2-593717-B21-HPE-NC523SFP-DU...
https://www.ebay.com/itm/LOT-OF-2-593717-B21-HPE-NC523SFP-DU...
Exactly this. Few laptops have 10GbE, though many have Thunderbolt and USB 3.1.
What kind of things will use more than 1gb?
And can most systems push this kind of data?
(home systems)
And can most systems push this kind of data?
(home systems)
A basic NAS with a handful of spinning rust drives can easily exceed 1 gigabit/s. A single SSD can as well.
A single modern mechanical HDD has more sequential throughput than a gigabit ethernet link.
SSDs are an order of magnitude faster than gigabit ethernet.
Google fiber is rolling out 2Gbps internet in many locations
1 Gigabit is definitely good enough for most general use cases, but once you start moving files around regularly it becomes a huge bottleneck.
SSDs are an order of magnitude faster than gigabit ethernet.
Google fiber is rolling out 2Gbps internet in many locations
1 Gigabit is definitely good enough for most general use cases, but once you start moving files around regularly it becomes a huge bottleneck.
My patience. :-)
People will compare browser benchmarks over milliseconds so why not file transfers over the home network?
In general terms, cynicism about the "need" may be right. Most home users in most use cases likely won't notice much by going to 10G.
But then there is the rest of us and the other use cases.
As for systems that can push this much data. Almost anything with a M.2 SSD will easily keep up.
So for a use case, backing up your zillions of photos to at least one other point of failure on your home network. May not be the best backup plan ever, but regular home users are not IT gurus and will do this.
People will compare browser benchmarks over milliseconds so why not file transfers over the home network?
In general terms, cynicism about the "need" may be right. Most home users in most use cases likely won't notice much by going to 10G.
But then there is the rest of us and the other use cases.
As for systems that can push this much data. Almost anything with a M.2 SSD will easily keep up.
So for a use case, backing up your zillions of photos to at least one other point of failure on your home network. May not be the best backup plan ever, but regular home users are not IT gurus and will do this.
A single modern HDD can push out (sequential) 100-200 MBytes/s per spindle. Commodity SSDs can saturate a SATA port.
So yes, it's trivial to saturate a 10 Gbit/s link, assuming that the kernel and userland can keep up with the hardware.
So yes, it's trivial to saturate a 10 Gbit/s link, assuming that the kernel and userland can keep up with the hardware.
2Gbit internet service is becoming semi-common.
Wifi 6E can do almost wigig/uwb speeds, near 7Gbit hypothetical.
A single modern NVMe drive does 56Gbit.
USB4 can do 40Gbit. It can connect direct computer to computer.
Wifi 6E can do almost wigig/uwb speeds, near 7Gbit hypothetical.
A single modern NVMe drive does 56Gbit.
USB4 can do 40Gbit. It can connect direct computer to computer.
My systems definitely can.
But indeed as others have said, even broadband connections are exceeding this speed now. USB exceeds it by a factor 40 and is on cheap laptops. Yet Ethernet remains so expensive.
But indeed as others have said, even broadband connections are exceeding this speed now. USB exceeds it by a factor 40 and is on cheap laptops. Yet Ethernet remains so expensive.
Even the raspberry pi 4 with USB 3 on a single PCIe 2 lane has a limit of 4gbs.
More importantly, 10GbE cards are cheap as free on eBay. That + a Mikrotik CRS305-1G-4S+IN and some direct connect cables you can have a 10GbE network in your house for well under $300.
I picked up the 8 port version from Microtik and picked up a Mellanox ConnectX-2 of eBay which came with a pair of 10G SFP+ fiber modules. A 10 meter single mode LC fiber off Amazon was like $20.
As for the NAS, I snagged a Supermicro mini-ITX 8-core Xeon D (Broadwell era) board from eBay as well for like $400. It brings out the dual integrated 10G NICs on the Xeon D, a PCIe 3.0 x16 port, a 4 lane M.2 port, and 6x SATA ports, 4 slots for DDR4 ECC RDIMMs, and an ASPEED based IPMI. Adding 64 GiB of RDIMMs, 6x 10 TB WD drives, and 2x M.2 drives on a bifurcating riser made for a killer NAS and VM host (crammed inside a Fractal Node 304)
As for the NAS, I snagged a Supermicro mini-ITX 8-core Xeon D (Broadwell era) board from eBay as well for like $400. It brings out the dual integrated 10G NICs on the Xeon D, a PCIe 3.0 x16 port, a 4 lane M.2 port, and 6x SATA ports, 4 slots for DDR4 ECC RDIMMs, and an ASPEED based IPMI. Adding 64 GiB of RDIMMs, 6x 10 TB WD drives, and 2x M.2 drives on a bifurcating riser made for a killer NAS and VM host (crammed inside a Fractal Node 304)
Can I run this on vanilla Cat5e or do I have to put in Cat6?
The cheap used 10Gb NICs and the switch mentioned use SFP+ ports rather than RJ45. 10GBASE-T SFP+ modules are expensive and power-hungry compared to direct-attach twinax cables or fiber + transceivers.
I had a flat CAT 6 cable and could only get 5GbE - upgraded to a normal cat 7 cable and got the 10GbE.
Cat5e usually works fine. For longer distances you will need the correct (cat6 or better) though.
That's what I just did! Been loving it!
Intel https://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=... and Mellanox https://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=... are around $800, and a single-port card is under $700.
Say I had two of these cards. How would I send 200Gib of traffic through it? What would the test program look like?
You could use something based on DPDK, like pktgen-dpdk: https://git.dpdk.org/apps/pktgen-dpdk/
For testing, something like iperf?
https://iperf.fr/
https://community.mellanox.com/s/article/howto-install-iperf...
https://iperf.fr/
https://community.mellanox.com/s/article/howto-install-iperf...
Are there any dedicated server hosters that give you 100Gbps private networking among your servers?
AWS has 100G in some instance types.
Where does it mention price?
The spec page there doesn't, google "broadcom p2100g" and you'll see vendor prices... I didn't want to link directly to a vendor for a top level post.
https://www.mouser.com/ProductDetail/Broadcom-Avago/BCM95750...
https://www.mouser.com/ProductDetail/Broadcom-Avago/BCM95750...
Gah that hurts. Not that it concerns HN, but I immediately thought 'oh, that's neat I have to mail that to friend x', and then while composing the mail in my head I halfway through realized he's been dead for a year. :(
Hey, sorry about that.
Good to take some time to remember them sometimes.
Good to take some time to remember them sometimes.
Oh wow, didn't know you could buy these of mouser.
My laptop has four 40 Gb/s ports on it. None of them are Ethernet.
Ethernet has wide adoption and standards tested for years. It's going to be hard to knock it for a different solution anytime soon. But we'll see :)
Are we advertising for broadcom now?
You can hardly really advertise it. Nobody invests in 100 GbE NICs taking advertisements seriously rather than knowing some reasons to choose/avoid particular brands/models.
There are no Internet providers in Canada that do more than 1.5Gbps though (Ball Canada claims to have 1.5Gbps down) and even that is mostly just claimed and rarely delivered. These days my compute workloads are mostly in the cloud and all my files are there as well. Thus local speeds do not matter much if the Internet can not also feed them.
I guess this matters for data centres, but primarily my data centres are run by AWS and GCP, so I do not buy this hardware myself.
I guess this matters for data centres, but primarily my data centres are run by AWS and GCP, so I do not buy this hardware myself.
Would be really cool to have all my local systems have little local disk, but instead share a NAS. But still a very niche use case
Speaking of Canada, Linus "Tech Tips" Sebastian mentioned this very argument in a recent video where he was installing 100G and 25G in his office for workgroup video editing. Not everything is in the cloud yet.
Not being the bottleneck doesn’t make increased throughput completely superfluous. The faster you can communicate across a channel, the quicker you can shut up and get to idle, saving power on both transmission and policing of resource contention. You also have more time to handle locally scoped retries and error correction.
Google Fiber can do 2Gbps.
https://serversupply.com/NETWORKING/NETWORK%20ADAPTER/100%20...