PCI Express over Fiber [video](youtube.com)
youtube.com
PCI Express over Fiber [video]
https://www.youtube.com/watch?v=XaDa9bBucEI
41 comments
> Although we can already buy commercial transceiver solutions that allow us to use PCIe devices like GPUs outside of a PC, these use an encapsulating protocol like Thunderbolt rather than straight PCIe.
> [snip]
> As explained in the intro, this doesn’t come without a host of compatibility issues, least of all PCIe device detection, side-channel clocking and for PCIe Gen 3 its equalization training feature that falls flat if you try to send it over an SFP link.
So, uh… what’s the benefit? How much overhead does Thunderbolt really introduce, given it solves these other issues?
> [snip]
> As explained in the intro, this doesn’t come without a host of compatibility issues, least of all PCIe device detection, side-channel clocking and for PCIe Gen 3 its equalization training feature that falls flat if you try to send it over an SFP link.
So, uh… what’s the benefit? How much overhead does Thunderbolt really introduce, given it solves these other issues?
I go over it in the video but yes, active thunderbolt is probably a very good choice for a lot of people. I went into another direction for some reasons that are not applicable to everyone:
- Learning : I want to learn about the lower level of PCIe and it's a good project. - Re-use of cabling : I have a bunch of single mode fiber bundle going around already. You can't find thunderbolt that just have a LC connector ... - Isolation : Active thunderbolt cable still often have copper for some low speed signals, they don't offer true galvanic isolation - Avoid dealing with thunderbolt. I want a custom chassis/pcb at one end and chips to convert from TB back to PCIe are not readily available to make custom stuff with ... (not as an individual anyway).
So yeah, if you want a ready to use solution, TB cable is absolutely a good choice, here I'm having some fun, learning in the process and hopefully sharing some of the knowledge.
- Learning : I want to learn about the lower level of PCIe and it's a good project. - Re-use of cabling : I have a bunch of single mode fiber bundle going around already. You can't find thunderbolt that just have a LC connector ... - Isolation : Active thunderbolt cable still often have copper for some low speed signals, they don't offer true galvanic isolation - Avoid dealing with thunderbolt. I want a custom chassis/pcb at one end and chips to convert from TB back to PCIe are not readily available to make custom stuff with ... (not as an individual anyway).
So yeah, if you want a ready to use solution, TB cable is absolutely a good choice, here I'm having some fun, learning in the process and hopefully sharing some of the knowledge.
Hey, I love a great self-educational deep dive. Don’t have time to watch the video until after the workday, but it sounds enlightening! (I swear that was not intentional.)
The benefits are twofold: physical colocation and bandwidth.
Thunderbolt 5 offers 80Gbps of bidirectional bandwidth. PCIe 5.0 16x offers 1024Gbps of bidirectional bandwidth. This matters.
TB5 cables can only get so long whereas fiber can go much farther more easily. This means that in a data center type environment, you could virtualize your GPUs and attach them as necessary, putting them in a separate bank (probably on the same rack).
Thunderbolt 5 offers 80Gbps of bidirectional bandwidth. PCIe 5.0 16x offers 1024Gbps of bidirectional bandwidth. This matters.
TB5 cables can only get so long whereas fiber can go much farther more easily. This means that in a data center type environment, you could virtualize your GPUs and attach them as necessary, putting them in a separate bank (probably on the same rack).
Active optical (yes!) Thunderbolt cables can be much longer. After all, optical fiber was the original medium for Thunderbolt, back when it was still called Light Peak.
I couldn’t find any optical TB5 cables, but here’s a 4.5m TB4 one: https://www.owc.com/blog/the-new-superlong-40gb-s-owc-active...
And if TB3 is enough, Corning makes them in lengths up to 50m: https://www.corning.com/microsites/coc/oem/documents/ocbc/OE...
As for bandwidth, the medium transition seems to actually limit the author’s capabilities by losing some of the more advanced link-training features that are necessary for the highest-bandwidth PCIe 3 connections, never mind PCIe 5.
I couldn’t find any optical TB5 cables, but here’s a 4.5m TB4 one: https://www.owc.com/blog/the-new-superlong-40gb-s-owc-active...
And if TB3 is enough, Corning makes them in lengths up to 50m: https://www.corning.com/microsites/coc/oem/documents/ocbc/OE...
As for bandwidth, the medium transition seems to actually limit the author’s capabilities by losing some of the more advanced link-training features that are necessary for the highest-bandwidth PCIe 3 connections, never mind PCIe 5.
Thunderbolt is kind of cursed. To insure maximum compatibility it mandates a legacy usb2 connection via separate connections. TB3/USB4/TB4 are packetized, but afaik there's no defined way to packetized usb2, it's expected there be a whole separate set of wires for it.
And because of timings, my admittedly so-aonunderstanding that you can only get about 7m before you absolutely have to have a hub/repeater (unless you can speed up the speed of light considerably). This limit to how long a single length can be can't really be cheated without violating usb specs.
It's awesome if folks have packetized USB2. A pity it's not in the flipping spec though!!
That Corning made it 50m is wild. You need a virtual hub at the start that can pretend to be hubs 1-5 (so it's close enough to time well). Then a hub on the other side of the cable at (skinny) tree depth 6. Allowing for 4 devices under it (the number of ports on a usb2 hub in the spec. But you could work around by faking being not a skinny tree but a fat tree, maybe?).
And because of timings, my admittedly so-aonunderstanding that you can only get about 7m before you absolutely have to have a hub/repeater (unless you can speed up the speed of light considerably). This limit to how long a single length can be can't really be cheated without violating usb specs.
It's awesome if folks have packetized USB2. A pity it's not in the flipping spec though!!
That Corning made it 50m is wild. You need a virtual hub at the start that can pretend to be hubs 1-5 (so it's close enough to time well). Then a hub on the other side of the cable at (skinny) tree depth 6. Allowing for 4 devices under it (the number of ports on a usb2 hub in the spec. But you could work around by faking being not a skinny tree but a fat tree, maybe?).
IIRC, USB-PD also requires USB 2.0 signaling. The idea of dedicated lower-bandwidth signaling wires isn’t uncommon in my very limited EE experience—level 3 charging reuses J-1772 signaling to control the charge available of the DC pins.
Hundreds of meters is considered short range in the world of *SFP. If you just plan on putting the GPUs in the same rack then I'm not sure it really matters, but you can really put anything anywhere in your DC and have things zoned with *SFP.
I don't think there is any reason TB couldn't do the same, beyond it would be even more niche to want non-modular/patchable cables+transceivers at those lengths (especially since fiber is often bundled dozens/hundreds of strands over a single trunk cable between racks).
I don't think there is any reason TB couldn't do the same, beyond it would be even more niche to want non-modular/patchable cables+transceivers at those lengths (especially since fiber is often bundled dozens/hundreds of strands over a single trunk cable between racks).
For the curious, that 50m cable is $500 MSRP. https://1sourcevideo.com/shop/corning-50-meter-thunderbolt-3...
"same rack" should still be fine for 1m passive TB5 cable though, right?
> 1024Gbps
Good luck getting a 1Tbit tranceiver. Anydirectional. Also it's 512Gbitish per direction.
Good luck getting a 1Tbit tranceiver. Anydirectional. Also it's 512Gbitish per direction.
That's 64Gb per lane across x16 lanes. That sounds not daunting?
There's already 800Gb transceivers readily available, 1.6 is probably getting preview deploys to some hyperscalers & other early adopters as we speak.
There's already 800Gb transceivers readily available, 1.6 is probably getting preview deploys to some hyperscalers & other early adopters as we speak.
Easy, fs.com has 1.6Tbps OSFP for about 570€ - though only up to 1m lenght apparently.
I was looking into the highest bandwidth optical transceivers. 400Gbps were easy enough to find so thanks for posting this. I honestly didn't know there were 1.6Tbps transceivers like this.
One note: I believe the SMF max fiber length is 2km not 1m [1]. The data sheet [2] also says:
> - 2000m max on single mode fiber
[1]: https://www.vitextech.com/products/1-6t-osfp-2fr4
[2]: https://resource.fs.com/mall/resource/cn_osfp-2fr4-16t-data-...
One note: I believe the SMF max fiber length is 2km not 1m [1]. The data sheet [2] also says:
> - 2000m max on single mode fiber
[1]: https://www.vitextech.com/products/1-6t-osfp-2fr4
[2]: https://resource.fs.com/mall/resource/cn_osfp-2fr4-16t-data-...
Bidirectional is a lot like biweekly. Biweekly depending on context means twice a week or once every two weeks and bidirectional can both mean per direction and total of both directions.
But yes I meant 512Gbps each way, to be clear.
But yes I meant 512Gbps each way, to be clear.
I'm only a single datapoint but I've never encountered that usage. My understanding of a bidirectional link is that it meets the same spec in both directions simultaneously. It's important precisely because many links aren't bidirectional, sharing a single physical link between two logical links.
The more precise terms are full-duplex and half-duplex.
The video is about a 2x1 link, which the author hopes to eventually scale up to 3x4 using 40 gig transceivers. I'd say thunderbolt is probably safe in the near future.
This was a super interesting video to watch. I honestly thought SFP required more setup, but this explains why AliExpress is so ripe with USB3 and HDMI over SFP converters that are dirt cheap.
It's been amazing having 6 years of fiber optic HDMI & DP monitor connections, that work so so so well. I bought some no name one on Amazon in ~2019 and was flabbergasted it was real & worked.
Such a huge upgrade from the heavy thick 35 ft HDMI<->dvi cable I've used for so long.
Literally the only downside is figuring out how to roll it up, which I still haven't figured out how to do well with the 150ft cable I have.
It was astoundingly cheap too. I think the first one I got was under $60?! No one really knew the segment existed, they just needed to get some sales, I assume. I heard usb3 has been available but they've been bulky & expensive. Where-as the whole fiber optic cable seamlessly integrates the transceiver on mine. I like Cable Matters, they make some fine ones.
Such a huge upgrade from the heavy thick 35 ft HDMI<->dvi cable I've used for so long.
Literally the only downside is figuring out how to roll it up, which I still haven't figured out how to do well with the 150ft cable I have.
It was astoundingly cheap too. I think the first one I got was under $60?! No one really knew the segment existed, they just needed to get some sales, I assume. I heard usb3 has been available but they've been bulky & expensive. Where-as the whole fiber optic cable seamlessly integrates the transceiver on mine. I like Cable Matters, they make some fine ones.
Amusingly PCIe is talking CopprLink now, which is amusing because it also the expected basis for future optical work (yet has coppr in the name). I'm honestly not sure what if anything it brings vs OCuLink, if relaxes timings at all/allows latency, or if it's just specifying connectors etc.
https://pcisig.com/blog/pcie%C2%AE-cabling-%E2%80%93-journey...
https://www.tomshardware.com/pc-components/gpus/copprlink-de...
Worth noting too that well respected vendors have been selling optical thunderbolt cables for a while now. I wonder if they are length limited for latency reasons (& hello hollow core fiber)? I wonder if they are usb3/multiprotocol, or if they are usb4 only. I also wonder how they handle the incredibly jank usb4 requirement to also have a separate legacy usb2. As a usb-c enjoyer, I can still admit: sure seems like USB is a lot of work to support! I can't help but wonder how blissfully simple a future CXL over cable stack might look by compare. https://www.owc.com/solutions/usb4-cables
Worth noting too that well respected vendors have been selling optical thunderbolt cables for a while now. I wonder if they are length limited for latency reasons (& hello hollow core fiber)? I wonder if they are usb3/multiprotocol, or if they are usb4 only. I also wonder how they handle the incredibly jank usb4 requirement to also have a separate legacy usb2. As a usb-c enjoyer, I can still admit: sure seems like USB is a lot of work to support! I can't help but wonder how blissfully simple a future CXL over cable stack might look by compare. https://www.owc.com/solutions/usb4-cables
So you're saying I can put a handful of 4090's out in the middle of snowy Michigan with a handful of OM4 cables snaking into my basement to run legit arctic cooling with no noise?
Might as well put your entire computer outside and use thunderbolt/usb-4 over fiber docks
Watercooling loop light be better, the radiator fins will still rust from condensation.
Better yet, keep it inside and save big on heating!
What you meant is you will have your GPU rig running passively cooled inside the pond tethered to your pier in your back yard
No part of Michigan is in the arctic, but sure, outside of mosquito season, that would work.
I mean yes, but you could also just place the entire computer out there as well
How does this compare to something like RDMA over Converged Ethernet (RoCE)?
A fun tangent - if someone wants to explore how Azure is performing RDMA over RoCEv2 - check this paper out - https://www.microsoft.com/en-us/research/wp-content/uploads/...
There is an interesting NSDI talk on the paper too - https://www.youtube.com/watch?v=kDJHA7TNtDk (2023)
There is an interesting NSDI talk on the paper too - https://www.youtube.com/watch?v=kDJHA7TNtDk (2023)
It seems rather educational.
Cool project! PCIe itself I think is likely to end up doing something similar soon, there are provisions in the spec now for optical retimers.
Nothing new here. Samtec were doing this in 2017 https://www.samtec.com/support/videos/pcie-over-fiber-with-f...
and consumer/prosumer pcie encapsulation over thunderbolt over fiber was happening in 2015. maybe even earlier
still, it's incredibly cool for one guy to pull this off on his own. demonstrates mastery of the subject
still, it's incredibly cool for one guy to pull this off on his own. demonstrates mastery of the subject
Does it have to be new?
There's a number of optical modules for TB3 and TB4, might be an easier (but less fun) route as TB3 and TB4 can carry PCIe.
[deleted]
While at a higher level, thunderbolt and https://en.wikipedia.org/wiki/ExpEther can both of course work over fiber too!
(Q|O)SFP are basically just raw high speed serial interfaces to whatever - you see this a lot in FPGAs, you can use the QSFP interfaces for anything high speed - PCIe, SATA, HDMI…