ASICs at the Edge(blog.cloudflare.com)
blog.cloudflare.com
ASICs at the Edge
https://blog.cloudflare.com/asics-at-the-edge/
11 comments
> we never prototyped in FPGAs. Most designs will not fit into a single FPGA and designs involving multiple FPGAs are a headache and the design diverges rapidly from your target due to all the overhead associated with coordinating multiple FPGAs.
It definitely depends on what you're building. When I worked on GPUs at Broadcom and CPUs at arm (and now working on OpenTitan: https://opentitan.org/) FPGAs were used. At both companies you could create a sufficiently small configuration of the design that a single FPGA was a viable target without requiring heroics for coordinating multiple FPGAs (For the GPU turn down the number of cores, for CPU I was working on the little designs A55 and such which you could fit in a sufficiently large FPGA).
Though I'd agree FPGAs aren't often used a prototyping tool. They're used for verification and often once the design is fairly mature (they're inevitably a pain to setup and use so you want to iron out all the stupid bugs first, FPGAs are most useful when you need to be running many cycles to find an issue).
It definitely depends on what you're building. When I worked on GPUs at Broadcom and CPUs at arm (and now working on OpenTitan: https://opentitan.org/) FPGAs were used. At both companies you could create a sufficiently small configuration of the design that a single FPGA was a viable target without requiring heroics for coordinating multiple FPGAs (For the GPU turn down the number of cores, for CPU I was working on the little designs A55 and such which you could fit in a sufficiently large FPGA).
Though I'd agree FPGAs aren't often used a prototyping tool. They're used for verification and often once the design is fairly mature (they're inevitably a pain to setup and use so you want to iron out all the stupid bugs first, FPGAs are most useful when you need to be running many cycles to find an issue).
> the tooling is good enough
What are some of your favorite simulation and verification tools?
What are some of your favorite simulation and verification tools?
> Technically, the use case in the network industry should be called an ASSP (Application Specific Standard Product), but network engineers are simple people, so we prefer to call it an ASIC.
Strictly speaking, ASIC only refers to "a custom chip for a specific customer for a narrow application". If the chip is available to the general public, it's not an ASIC but an ASSP. So a Wi-Fi chipset [0], an Ethernet controller or an LCD driver on the public market are all ASSPs, never ASICs. But colloquially speaking, for many people, "ASIC" is a synonym for all custom chips, so almost everything is ASIC, even if the chip is a general-purpose CPU, many will call it "ASIC" if it's someone's original design. Loose term, but it's not really bothering anyone, so this usage remains. Oh, and the word "chipset" has a similar problem. It originally refers to the ASIC/ASSP on the computer motherboard (originally IBM PC's I/O controller, but also graphics and sound), but nowadays it's often used as a synonym of "ASSP", in other words, "highly integrated chip designed for a specific application in mind", hence Wi-Fi chipset, 5G chipset...
Strictly speaking, ASIC only refers to "a custom chip for a specific customer for a narrow application". If the chip is available to the general public, it's not an ASIC but an ASSP. So a Wi-Fi chipset [0], an Ethernet controller or an LCD driver on the public market are all ASSPs, never ASICs. But colloquially speaking, for many people, "ASIC" is a synonym for all custom chips, so almost everything is ASIC, even if the chip is a general-purpose CPU, many will call it "ASIC" if it's someone's original design. Loose term, but it's not really bothering anyone, so this usage remains. Oh, and the word "chipset" has a similar problem. It originally refers to the ASIC/ASSP on the computer motherboard (originally IBM PC's I/O controller, but also graphics and sound), but nowadays it's often used as a synonym of "ASSP", in other words, "highly integrated chip designed for a specific application in mind", hence Wi-Fi chipset, 5G chipset...
Also, mobile and embedded device SoCs carrying CPUs plus other functions fall into the category of ASICs in many cases.
Great intro article, saving this one for folks at work!
For anyone confused why so many low buffer chips exist it depends greatly on the network needs. In some deployments going for deep buffers can actually make things slower overall. E.g. in enterprise healthcare I've really gravitated towards Trident (Trident 3 currently) level chips for most of the LAN/MAN and sometimes RAN as the feature sets are there and there isn't a strong driver for deep buffer per se just traffic priority for phones and a certain app once in a great while. I.e. it doesn't matter if you've got 16 GB of deep packet buffer built up, the telemetry packet needs to make it out the 10 Mbit pipe to the remote clinic now not at some point in the future. For WAN across regions or internet we move into some of those Juniper ASICs though and do rely on deeper buffer for the MANY concurrent sessions which are fighting understanding loss at higher latencies.
Another interesting hardware rabbit hole to go down is the modular transceivers that go into these kinds of switches to provide different kinds of copper/optical connections instead of always being a fixed RJ-45 port like most would be used to at home. Crazy levels of engineering, especially at the higher end. I saw a really interesting presentation last year from Juniper on their use of silicon photonics for high density low cost 400 gigabit Ethernet and beyond (pdf warning) https://www.juniper.net/assets/us/en/local/pdf/nxtwork/silic...
For anyone confused why so many low buffer chips exist it depends greatly on the network needs. In some deployments going for deep buffers can actually make things slower overall. E.g. in enterprise healthcare I've really gravitated towards Trident (Trident 3 currently) level chips for most of the LAN/MAN and sometimes RAN as the feature sets are there and there isn't a strong driver for deep buffer per se just traffic priority for phones and a certain app once in a great while. I.e. it doesn't matter if you've got 16 GB of deep packet buffer built up, the telemetry packet needs to make it out the 10 Mbit pipe to the remote clinic now not at some point in the future. For WAN across regions or internet we move into some of those Juniper ASICs though and do rely on deeper buffer for the MANY concurrent sessions which are fighting understanding loss at higher latencies.
Another interesting hardware rabbit hole to go down is the modular transceivers that go into these kinds of switches to provide different kinds of copper/optical connections instead of always being a fixed RJ-45 port like most would be used to at home. Crazy levels of engineering, especially at the higher end. I saw a really interesting presentation last year from Juniper on their use of silicon photonics for high density low cost 400 gigabit Ethernet and beyond (pdf warning) https://www.juniper.net/assets/us/en/local/pdf/nxtwork/silic...
I looked up one of the devices mentioned here: Juniper’s 2U-sized PTX10002. Turns out the list price for PTX10002-60C-AC-R is $720,000.00. :-o
The traffic volumes these devices are intended for are truly mind-boggling to me (as someone currently building a small 25Gbit network with sub-$1k parts).
The traffic volumes these devices are intended for are truly mind-boggling to me (as someone currently building a small 25Gbit network with sub-$1k parts).
Can you explain what the advantages of a 25gbit network are?
It’s the next step up from 10G (which our peak traffic is outgrowing).
If you look at older/used networking gear, 40G QSFP+ might look like the next step, but to me that’s starting to look a bit like an evolutionary dead end at this point.
Low-end 25G routers/switches are relatively cheap, as are 25G SFP28 optics (including bi-directional ones, which get a bit marginal at speeds above 25G).
If you look at older/used networking gear, 40G QSFP+ might look like the next step, but to me that’s starting to look a bit like an evolutionary dead end at this point.
Low-end 25G routers/switches are relatively cheap, as are 25G SFP28 optics (including bi-directional ones, which get a bit marginal at speeds above 25G).
I guess what I'm really asking is: what are you doing that requires so much bandwidth?
It's not just the immense traffic throughput, but also consider that some of these routers [0] can do line-rate MACsec encryption at 400Gbps on every port.
This kind of horsepower was unthinkable even a few yrs ago...
[0] Example Juniper PTX10000 datasheet (PDF)
https://www.juniper.net/assets/us/en/local/pdf/datasheets/10...
[0] Example Juniper PTX10000 datasheet (PDF)
https://www.juniper.net/assets/us/en/local/pdf/datasheets/10...
A few minor nits:
* As such, the development lifecycle is incredibly long. It starts with prototyping in an FPGA (Field Programmable Gate Array), in which chip designers can program their required functionality and confirm compatibility*
I’ve been involved with development of multiple generations of ASICs including 7nm and now 5nm. Going all the way back to 40nm, we never prototyped in FPGAs. Most designs will not fit into a single FPGA and designs involving multiple FPGAs are a headache and the design diverges rapidly from your target due to all the overhead associated with coordinating multiple FPGAs.
Prototype might be something you do for a very simple design and a new or inexperienced team.
If you have a good experienced team, the tooling is good enough that simulation and verification are all that’s needed. However verification is a very extensive process and the most time consuming part and can take 9-12 months. 7nm tape-outs are extremely expensive. So you want to get it right. Most designs have multiple options built in on risky sections so that a work-around for a bug can be done in a metal spin. Furthermore, most of these chips have significant FW/SW associated with them as well. If you think about it, that’s quite a lot to deliver in 2 years into something as reliable as a network processor.
When you consider all of the above, Broadcom does an amazing job going from TH3 to TH4 in only 2 years. They are tough to beat because they have such a good process that they can crank out these incredibly complex chips on a reliable cadence. Please note that this includes analog for the SERDES which is its own development.