Cerebras’ new monster AI chip adds 1.4T transistors(spectrum.ieee.org)
spectrum.ieee.org
Cerebras’ new monster AI chip adds 1.4T transistors
https://spectrum.ieee.org/tech-talk/semiconductors/processors/cerebras-giant-ai-chip-now-has-a-trillions-more-transistors
168 comments
> That chip needed some 20,000 amps of current fed to it from one million copper connections to a fiberglass circuit board atop the wafer.
Is the 20,000 amp number not an error?
Is the 20,000 amp number not an error?
They also say the peak power draw is 23kW. Now 23kW/20kA = 1.1V which is about what my NVIDIA 2080 GPU uses at load. So that does indeed seem to work out.
Of course 20kA is pretty insane, would be interesting to see just how they feed the power to this thing. Bond wires going to the middle of the chip? And the local regulator must be a beast, or rather lots of beasts.
Of course 20kA is pretty insane, would be interesting to see just how they feed the power to this thing. Bond wires going to the middle of the chip? And the local regulator must be a beast, or rather lots of beasts.
20000 amps seems plausible. A regular desktop CPU operates at 1-2 Volts and power consumption of 100-200 watts which means 50-200 Amps.
Comparison table shows nvidia A100 which has max power consumption of 400W. This has roughly 50 times the transistors and 50 surface area so having 50 times the power consumption 50*400=20000W and same order of magnitude of current seems possible.
Comparison table shows nvidia A100 which has max power consumption of 400W. This has roughly 50 times the transistors and 50 surface area so having 50 times the power consumption 50*400=20000W and same order of magnitude of current seems possible.
Time for an AI winter I guess.
This statement was made for experimental purposes, to understand HN biases.
You do realize that AI crossed human expert performance in NLP / Vision tasks ?
I'm not sure if you're assuming best intentions. If you do, it'd be nice to provide sources - to my knowledge, vision under non-optimal conditions (rain, snow, sunlight ahead) is only partially solved by resorting to sensors resistant to the disturbance.
I'd be glad to learn I'm wrong.
I'd be glad to learn I'm wrong.
TBH, I believe we need another AI winter to come up with something better than the AI "black boxes" we have right now.
Exactly how does one outperform a human expert in natural language processing?
Parent probably meant that it outperformed a human expert in some specific task in the area of natural language processing - for example the task of converting a spoken language into a written language...
See, if you were an AI, you would understand EXACTLY what the poster means by this.
Maybe the gibberish GPT-3 spits out is actually true, and our puny monke brains are just too weak to understand it.
They don't really outperform human experts on real tasks yet (no matter what some superGLUE or other benchmark shows); but in general, once a system can solve a particualr task well, it would be plausible to outperform human experts simply by not making random errors.
If we have multiple human experts annotate a NLP task and measure inter-annotator agreement, it will be far from 100%; part of that will be genuine disagreements or fuzzy gray area, but part of the identified differences will be simply obviously wrong answers given by the experts - everyone makes mistakes. The same applies for many other domains - business process automation, data entry, etc; no employee will produce error-free output in a manual process, no matter how simple and unambiguous the task is.
And for simpler tasks the computer can easily make less mistakes than a human - especially if you measure the human reliability not for a few minutes of focus, but for a whole tedious working day.
If we have multiple human experts annotate a NLP task and measure inter-annotator agreement, it will be far from 100%; part of that will be genuine disagreements or fuzzy gray area, but part of the identified differences will be simply obviously wrong answers given by the experts - everyone makes mistakes. The same applies for many other domains - business process automation, data entry, etc; no employee will produce error-free output in a manual process, no matter how simple and unambiguous the task is.
And for simpler tasks the computer can easily make less mistakes than a human - especially if you measure the human reliability not for a few minutes of focus, but for a whole tedious working day.
No, it hasn't. Not even close.
AI has "crossed human expert performance" on extremely narrow NLP/CV tasks.
AI is still light years away from human-level performance.
AI has "crossed human expert performance" on extremely narrow NLP/CV tasks.
AI is still light years away from human-level performance.
GPT-3 is not even close to a minimal and coherent text adventure made with Inform6 from a novice, even if it's written by a non-native English speaker.
Those networks didn't match "Detective", a crappy story written by a 12yo.
Those networks didn't match "Detective", a crappy story written by a 12yo.
>Time for an AI winter
with AI chips burning 15KW? No chance for the winter in sight. Some chances for AI hell though.
with AI chips burning 15KW? No chance for the winter in sight. Some chances for AI hell though.
Most interesting aspect of wafer-scale manufacturing is yield. Even if we have 95% chip yield, as the chip size approaches the wafer-level dimensions, I don't know off top of my head what the math would be but it is going to plummet drastically. My guess is that they're handling this in the chip logic. Building resiliency by turning off cells in the wafer that didn't yield. That begs the question, how are they probing them? A probe card of the size of the wafer is unheard of. How are they running validation? Pretty mindblowing to say the least!
Not an expert in chip manufacturing but my guess is that they just disable the parts that don't work and their big numbers represent ~80% of the actual number of transistors in the wafer because they account for that manufacturing loss.
Correct. You already have the same with modern CPUs and GPUs. You just disable the defect parts. Obviously, that sounds way easier than it is.
Disabling parts of the chip? The secret sauce then is to make it defect-proof
The question is, how do you detect defective cores?
Yup. This. From https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...:
> Cerebras achieves 100% yield by designing a system in which any manufacturing defect can be bypassed – initially Cerebras had 1.5% extra cores to allow for defects, but we’ve since been told this was way too much as TSMC's process is so mature.
> Cerebras achieves 100% yield by designing a system in which any manufacturing defect can be bypassed – initially Cerebras had 1.5% extra cores to allow for defects, but we’ve since been told this was way too much as TSMC's process is so mature.
Yep, not all semiconductor defects can be repaired, and at such scale repair circuits themselves will need to be redundant.
[deleted]
They probably aren't bothering to. The extreme economics for producing this type of chip are likely acceptable to the stakeholders.
Also, there is no reason they cant have some redundancy throughout the design so you can fuse off bad parts. It all really depends on the nature of the anticipated vs actual defects, which is an extraordinarily deep rabbit hole to climb into.
Also, there is no reason they cant have some redundancy throughout the design so you can fuse off bad parts. It all really depends on the nature of the anticipated vs actual defects, which is an extraordinarily deep rabbit hole to climb into.
I wonder how much this "chip" costs!
Products like these are often sold as part of a contract to deliver complete solutions + support + maintenance over some number of years.
It's hard to estimate a per-unit cost. But suffice to say it would cost similar to other datacenter compute solutions on a performance/$ level.
It's hard to estimate a per-unit cost. But suffice to say it would cost similar to other datacenter compute solutions on a performance/$ level.
According to Anandtech, an arm + leg. Also known as several million.
The previous generation was the $2-3 million range, and these are now in the neighborhood of "several million". Or, arm+leg.
How does this fusing work? I assume there are a bunch of wires than are either hot or ground and that determines if part of a chip gets run?
I believe you design special fuses on the chip that can be "blown" with a laser after testing and before putting the silicon in the protective packaging.
Fuses aren't blown with a laser, it's purely electrical. You apply a sufficiently high voltage to some port on the part from a source that can drive a high enough current and then tell the digital block of the chip which address to fuse and if you want it high or low. Repeat for the entire fuse bank and you're done.
https://en.wikipedia.org/wiki/Efuse
https://en.wikipedia.org/wiki/Efuse
It might actually use a traditional fuse block, where at some point in the packaging/testing process, you literally apply a sufficiently high voltage that you can permanently 'set' some part of it (whether that's actually melting a tiny wire, I'm not sure). But that's basically just programming a ROM that gets read in at boot time, and sets a bunch of logic on the chip to route around the bad parts. You could just use an external EEPROM to track that info too, and it would basically work the same.
20 years ago people were overclocking AMDs by drawing on them with pencil: http://www.ocmelbourne.com/tutorials/PencilTrick/
This last decade people soldered resistors onto cheaper NVIDIA cards, to make them behave as more expensive NVIDIA cards: https://www.eevblog.com/forum/general-computing/hacking-nvid...
So chipmakers have an incentive to bury the process in the chip and make it irreversible.
This last decade people soldered resistors onto cheaper NVIDIA cards, to make them behave as more expensive NVIDIA cards: https://www.eevblog.com/forum/general-computing/hacking-nvid...
So chipmakers have an incentive to bury the process in the chip and make it irreversible.
I bet they just probe the chip:
1. piece by piece
2. on die test circuits
1. piece by piece
2. on die test circuits
they exclude/disable cores with issues so an long as infrastructure parts of the chip are not affected the chip is functional
The first Cerebras Wafer Scale Engine used "... breakthrough techniques in cross-reticle patterning, but with the level of redundancy built into the design, ensured a yield of 100%, every time." I'm unsure what to think of this.
"When we spoke to Cerebras last year, the company stated that they already had orders in the ‘strong double digits’."
And they cost 2- 2.5 million each!
https://www.anandtech.com/show/15838/cerebras-wafer-scale-en...
"When we spoke to Cerebras last year, the company stated that they already had orders in the ‘strong double digits’."
And they cost 2- 2.5 million each!
https://www.anandtech.com/show/15838/cerebras-wafer-scale-en...
"the price has risen from ~$2-3 million to ‘several’ million".
"The CEO Andrew Feldman tells me that as a company they are already profitable, with dozens of customers already with CS-1 deployed and a number more already trialling CS-2 remotely as they bring up the commercial systems".
Quotes from https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
"The CEO Andrew Feldman tells me that as a company they are already profitable, with dozens of customers already with CS-1 deployed and a number more already trialling CS-2 remotely as they bring up the commercial systems".
Quotes from https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
So, less than 100 orders? That doesn't blow me away. Though at 2 million or more each, that adds up to real money...
200M would be great revenue in that space.
It's the old Wafer Scale Integration again. But now shows a successful product : https://en.wikipedia.org/wiki/Wafer-scale_integration
How do they do heat management and dissipation on such a big wafer? I can imagine some different parts heats up differently, putting a mechanical strain on wafer, and leading to cracks.
If you look at the die shots (wafer shots?), you'll notice small holes a few millimeters across spaced roughly at reticle spacing. Those are drilled holes to allow through-wafer liquid cooling. With liquid cooling not just around but through the wafer, the temperature differential is minimized.
thank you
"Carefully."
The article mentions a year of engineering went into dealing with the entire wafer thermally expanding under load.
The article mentions a year of engineering went into dealing with the entire wafer thermally expanding under load.
How much memory is on the chip, and what kind is it?
Under what circumstances does the chip need to access external memory?
What type of communication interfaces does this chip have?
Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?
Under what circumstances does the chip need to access external memory?
What type of communication interfaces does this chip have?
Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?
Tom's Hardware has some nice tables comparing the specs: https://www.tomshardware.com/news/cerebras-wafer-scale-engin...
(more than the IEEE)
This thing pulls 15-20kW of juice!
(more than the IEEE)
This thing pulls 15-20kW of juice!
I did a double that when I realised that was Kilowatts not Watts. This chip uses more energy in an hour than the average household (in my country at least) does in a day.
It may be a very large wafer but dissipating that heat is still very impressive.
It may be a very large wafer but dissipating that heat is still very impressive.
It sounds like a lot but it almost isn't? Like this is ~50x bigger than an Nvidia A100, and the A100 pulls up to 400w. 50 * 400 ~= 20kW. So in terms of thermal density it's in-line with existing GPUs.
That said, I'd be fascinated to see the cooling solution. Is it just a massive copper heatsink & a boatload of airflow? Typical approaches of using heatpipes to expand the heatsink won't really work with something this big after all. Or is it a massive waterblock with multiple inlets/outlets so it can hit up a stack of radiators? How do they get even mounting pressure across that large of an area?
That said, I'd be fascinated to see the cooling solution. Is it just a massive copper heatsink & a boatload of airflow? Typical approaches of using heatpipes to expand the heatsink won't really work with something this big after all. Or is it a massive waterblock with multiple inlets/outlets so it can hit up a stack of radiators? How do they get even mounting pressure across that large of an area?
There's an image on their website[1], pretty huge water pumps.
"To solve the 70-year-old problem of wafer-scale, we needed not only to yield a big chip, but to invent new mechanisms for powering, packaging, and cooling it.
The traditional method of powering a chip from its edges creates too much dissipation at a large chip’s center. To prevent this, CS-2’s innovative design delivers power perpendicularly to each core.
To uniformly cool the entire wafer, pumps inside CS-2 move water across the back of the WSE-2, then into a heat exchanger where the internal water is cooled by either cold datacenter water or air."
[1]: https://cerebras.net/product/
"To solve the 70-year-old problem of wafer-scale, we needed not only to yield a big chip, but to invent new mechanisms for powering, packaging, and cooling it.
The traditional method of powering a chip from its edges creates too much dissipation at a large chip’s center. To prevent this, CS-2’s innovative design delivers power perpendicularly to each core.
To uniformly cool the entire wafer, pumps inside CS-2 move water across the back of the WSE-2, then into a heat exchanger where the internal water is cooled by either cold datacenter water or air."
[1]: https://cerebras.net/product/
> This thing pulls 15-20kW of juice!
If you look at the wafer she’s holding at the top, it’s seemingly segmented into a 12x7 grid of roughly chip-sized rectangles. That’s 84 “CPUs” at 200-240 watts each, which is pretty well in line with discrete server CPUs.
The amount of heat coming off this thing must be amazing, though.
If you look at the wafer she’s holding at the top, it’s seemingly segmented into a 12x7 grid of roughly chip-sized rectangles. That’s 84 “CPUs” at 200-240 watts each, which is pretty well in line with discrete server CPUs.
The amount of heat coming off this thing must be amazing, though.
40GB of SRAM. Not quite big enough to fit the big models like GPT3
SRAM (static RAM) vs DRAM (dynamic RAM) for anybody else curious: https://computer.howstuffworks.com/question452.htm
It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications.
The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available.
However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip.
So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains.
That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.
The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available.
However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip.
So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains.
That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.
I don’t think it’s straightforward to do a head to head comparison.
from: https://www.youtube.com/watch?v=yso2S2Svdlg
@ 25:14
James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?"
Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."
from: https://www.youtube.com/watch?v=yso2S2Svdlg
@ 25:14
James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?"
Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."
Are there any good resources out there describing in practice how existing training workloads are distributed among GPUs? (using tensorflow, pytorch, or whatever else?).
I'm curious how the problem effectively gets sliced.
I'm curious how the problem effectively gets sliced.
SOTA on the biggest language models (which is where effectively the largest models are) is here: https://www.microsoft.com/en-us/research/blog/zero-infinity-...
It's 40 GB of SRAM. I doubt it supports external memory.
> Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?
Good question. I think it is. I mean the word "chip" isn't really that well defined (is HBM one chip?), but given that they sell it as a single unit and you can't really cut it in half I think it's one chip.
> Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?
Good question. I think it is. I mean the word "chip" isn't really that well defined (is HBM one chip?), but given that they sell it as a single unit and you can't really cut it in half I think it's one chip.
Looking at the picture, the thing is a platter. Not a chip.
It's a really cool picture too.
It's a really cool picture too.
It's a wafer, but that doesn't preclude it from being a chip. Where's you definition of chip coming from.
Considering the chipped edges, it is not far off.
This thing reminds me of Daniel Hillis's "The Connection Machine". Just 35 years later.
How can the chip itself consume that kind of power? Or is the 15kw value for the entire unit? That's like 10 residential space heaters all turned to max. I'm surprised that much heat could be dissipated over such a small surface area. Does it use refrigerant for cooling? If my math is correct, if you had a 6500BTU window air conditioner, you'd need 8 of them to move the heat from this chip.
Well, a single 7nm CPU (AMD Ryzen) can pull down 95 odd watts at peak, granted, the IO die is a fair bit of that as it's on an older process, but if you extrapolate that out across a giant wafer it's "only" 157 Ryzen 7's.
It would melt if you just do air cooling. I did some math for liquid nitrogen which has heat of vaporization as 200 kJ / kg. It would boil 3600*15/200 = 270 kg of nitrogen every hour. Just insane.
It’s water cooled for sure, perhaps with chilled water, but all integrated into the 15U chassis. There are cutaway images of the v1 system online.
I think I once saw one of the founders with a wafer in an In n' out with a potential investor. Looking at what Apple achieved with their M1A and the demand for "AI" - or training neutral networks, what it really is - they have a lot of potential.
At least as long as the AI bubble doesn't burst.
How can the future burst? It's like saying medicine will burst or physics
Right now everyone is betting that you can solve all problems just by throwing enough hardware at the problem. It's attractive for investors because they get to use their fat stacks of cash and corner the market but if this bet doesn't work out then we will have to go back to the drawing board and figure out how to build NNs efficiently and investors are not interested in this at all. They just want to launch a product as quickly as possible.
I don't know, expert systems haven't been so hot lately...
It's very surprising what people with downvote power do but it's *** to downvote anyone that disagrees. Anyway expert systems were a precursor to the technology and they got replaced much like new physics replace old physics.
Disagreeing doesn't mean you're right. AI is not equivalent to the future, and it will burst if it stalls and another AI winter cools the current hype cycle.
It's because you made a false analogy. AI isn't literally "the future". Billions of $ are being invested in deep-learning focused AI right now (which you call "the future"), and yes it could be a bubble and it could burst. You can disagree, but it's still a sensible thing to predict.
By bursting you mean humanity will never create artificial intelligence? Or you mean that there will be a cool off period as for example what happened with quantum physics at some point? Because it sure looks to me that there is no future without AI regardless of cool off periods. That makes my statement true. If you think humanity will never progress from where we are now then we pretty much are on very opposite schools of thought.
I agree with you, but I also think that the current wave of investment is in fact a bubble that will burst. See, I truly believe that AI will be an important part of our future. What you call a cool off, I call a bubble burst, which it is when the cool off involves investors losing tons of money they bet on deep learning.
The point being discussed here is that the current approaches (and those investing in them) might fail to yield any further significant advance.
Same way as the last time: rampant over-promising and under-delivering, maybe catalyzed by some high profile mishaps.
Bubbles popping aren’t always evaluations of the objective quality of something; it could just be about its assumed value relative to other plausible options. Homes mostly don’t become uninhabitable when a real estate bubble pops.
Bubbles popping aren’t always evaluations of the objective quality of something; it could just be about its assumed value relative to other plausible options. Homes mostly don’t become uninhabitable when a real estate bubble pops.
Personalized medicine, as an example, absolutely burst.
Much better article from anandtech at https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
Here's a video made by the author, Ian Cuttress, which goes into more detail as well.
https://www.youtube.com/watch?v=FNd94_XaVlY
https://www.youtube.com/watch?v=FNd94_XaVlY
This is very cool. The explanation of how yield/defects was interesting: they can bypass cores with defects due to channeling and account for the statistical defects, allowing them to have a 100% yield.
This is how pretty much all large chips are designed nowadays. The perfect chips are expensive higher-capacity / faster chips, the less perfect chips are cheaper lower capacity and/or slower.
I laughed at the "arm+leg" official cost of the CPU.
EDIT: Or GPU, or whatever it is.
EDIT: Or GPU, or whatever it is.
Interesting--the placement of code-to-processor so that things will be done roughly at the same time sounds a lot like the VLIW compiler problem of scheduling execution units so things are available at exactly the right time, without hardware interlocks.
Does anyone here have firsthand experience using the compiler? Can you give a rough approximation of performance tuning with the compiler compared to performance tuning with compilers targeting the tensor cores on an A100 or a TPU?
Is there any information on how this thing performs on actual AI workloads? It is priced similarly to a dozen of DGX A100s, but is it faster on something like training big transformer models(such as CLIP or GPT-3)?
This thing is awesome and, as someone working for a competitor, kind of scary. I applaud their approach though. I think we're a couple years off from it, but we'll probably see wider adoption of larger silicon, with more specialized functional units, which are used with a lower duty cycle to manage heat. If nothing else, they're probably developing some good IP and techniques to handle other sorts of ultra-mega-insane-scale-integration.
I wonder what their software stack looks like. Can they support the sort of virtualization and sharing you'd want to keep this expensive beast fully utilized 24/7?
I wonder what their software stack looks like. Can they support the sort of virtualization and sharing you'd want to keep this expensive beast fully utilized 24/7?
LSI, VLSI, UMISI
I like it!
In previous articles they've gone into some detail about how they deal with reticle limits, jumping over the scribe line area, and other n stuff. Between that, chiplets, HBM-style die stacks, etc... the developments here have been more interesting than I expected.
I like it!
In previous articles they've gone into some detail about how they deal with reticle limits, jumping over the scribe line area, and other n stuff. Between that, chiplets, HBM-style die stacks, etc... the developments here have been more interesting than I expected.
I wonder what the dev story looks like for these? I know they say "just use TF/Pytorch" but surely developers need to actually test stuff out on silicon and run CI on code... do they offer a baby version for testing?
This thing needs a GraphBLAS[1] implementation yesterday. 100 billion edge graphs and up are the new norm. This monster could smoke the competition if the implementation was tuned right!
[1] http://graphblas.org
[1] http://graphblas.org
Creating the ecosystem of both software and adjacent hardware for wafers this size is the real challenge for a company like Cerebras (which is doing amazing work). At first, they thought they just needed to make a chip 56x the size of its predecessor, and somehow get around the issue of defects and yield. After they solved those problems (which blocked Gene Amdahl, among others), they found they needed to bring an entire ecosystem into being to work with their hardware.
Agreed, that's why I think the GraphBLAS would be such a great fit for this hardware. The ecosystem is growing pretty fast. There are, for example Python bindings, you can do sparse 'A @ B' on millions of elements in parallel with this wafer-chip. MATLAB 2021a now has GraphBLAS built in, you could drive this thing directly from your notebooks.
I'm sure there's a compiler and low level primitives to really get the maximum performance out of it, but the trade-off maybe worth it in many cases to do it using an abstraction like the linear algebra approach.
I'm sure there's a compiler and low level primitives to really get the maximum performance out of it, but the trade-off maybe worth it in many cases to do it using an abstraction like the linear algebra approach.
Who said Moore’s law is dead?
It’s not dead, but melting.
its not a law but an observation.
I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way.
The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples.
And that’s just for gpt — you lose all the other libraries people have written. This matters more in GAN training, since for example you can find someone else’s FID implementation and drop it in without too much hassle. But with this specialized chip, you’d have to write it from scratch.
We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. But, for all its flaws, GL made these improvements “drop-in” —- just opt in to the new extension, and keep writing your gl code as you have been.
Ditto for direct3d, except they took the attitude of “limit to a specific API, not arbitrary extensions.” (Pixel shader 2.0 was an awesome upgrade from 1.1.)
AI has no such standards, and it hurts. The M1 GPU in my new Air is supposedly ready to do AI training. Imagine my surprise when I loaded up tensorflow and saw that it doesn’t support any GPU devices whatsoever. They seem to transparently rewrite the cpu ops to run on the gpu automatically, which isn’t the expected behavior.
So I dig into Apple’s actual api for doing training, and holy cow, that looks miserable to write in swift. I like how much control it gives you over allocation patterns, but I can’t imagine trying to do serious work in it on a daily basis.
What we need is a unified API that can easily support multiple backends — something like “pytorch, but just enough pytorch to trick everybody” since supporting the full api seems to be beyond hardware vendors’ capabilities at the moment. (Lookin’ at you, google. Love ya though.)
The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples.
And that’s just for gpt — you lose all the other libraries people have written. This matters more in GAN training, since for example you can find someone else’s FID implementation and drop it in without too much hassle. But with this specialized chip, you’d have to write it from scratch.
We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. But, for all its flaws, GL made these improvements “drop-in” —- just opt in to the new extension, and keep writing your gl code as you have been.
Ditto for direct3d, except they took the attitude of “limit to a specific API, not arbitrary extensions.” (Pixel shader 2.0 was an awesome upgrade from 1.1.)
AI has no such standards, and it hurts. The M1 GPU in my new Air is supposedly ready to do AI training. Imagine my surprise when I loaded up tensorflow and saw that it doesn’t support any GPU devices whatsoever. They seem to transparently rewrite the cpu ops to run on the gpu automatically, which isn’t the expected behavior.
So I dig into Apple’s actual api for doing training, and holy cow, that looks miserable to write in swift. I like how much control it gives you over allocation patterns, but I can’t imagine trying to do serious work in it on a daily basis.
What we need is a unified API that can easily support multiple backends — something like “pytorch, but just enough pytorch to trick everybody” since supporting the full api seems to be beyond hardware vendors’ capabilities at the moment. (Lookin’ at you, google. Love ya though.)
I'm on board with you that there should be a "drop-in" cross support of these chips, but pytorch is at a way higher abstraction level than what should be commonly supported.
Maybe people will get to their senses and switch to Julia instead of having to waste all these time on Python bindings.
Python here basically works as a binder for C, which is what everything is written in.
That is in fact the problem, yes.
Honestly I think julia missed the boat. There was an opportunity for julia to treat the gpu as an abstract distributed node (which is a first class concept in julia) and pay respect to the reality of data gravity. But they chose to instead basically treat the gpu as a synchronous entity.
I'm now bullish on elixir-nx because I think there's an outside shot they will get it right.
Python is a lost cause.
I'm now bullish on elixir-nx because I think there's an outside shot they will get it right.
Python is a lost cause.
I'm not sure that's necessarily the domain of a low-level package like CUDA.jl though (which I assume you're referring to). That kind of interface is more the domain of higher-level packages like https://github.com/JuliaGPU/DaggerGPU.jl and to a lesser extent https://juliagpu.github.io/KernelAbstractions.jl/stable/. Moreover, the jury is still out on whether the built-in Distributed module is an ideal abstraction for every use-case (clusters, heterogeneous compute, etc.)
WRT Nx, my biggest question is how they'll crack the problem of still needing big balls of C++ and the shims everywhere to get acceleration. Creating a compiler that generates efficient GPU or other accelerator code is a massive research project with no clear winners, never mind the challenge of reconciling the very mutation-heavy needs of GPU compute with a mostly immutable language model.
WRT Nx, my biggest question is how they'll crack the problem of still needing big balls of C++ and the shims everywhere to get acceleration. Creating a compiler that generates efficient GPU or other accelerator code is a massive research project with no clear winners, never mind the challenge of reconciling the very mutation-heavy needs of GPU compute with a mostly immutable language model.
There's plenty of mutability escape hatches in the erlang vm, and basically everyone who works in the language long enough is familiar with the idea of holding on to an immutable reference to a mutable thing. Like a database connection. Or a connection to a remote microservice. Or an ETS table.
> The philosophy here seems to be “if we build it, they’ll buy it.”
Supposedly Cerebras is already profitable, so it's hardly a situation where they are building something and hoping people buy it eventually.
> That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples.
Again, based on the companies representations, Cebebras transparently supports Pytorch and Tensorflow, only requiring a few lines of changed code.
Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-... (Dr. Cutress's video on TechTechPotato is also good).
Supposedly Cerebras is already profitable, so it's hardly a situation where they are building something and hoping people buy it eventually.
> That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples.
Again, based on the companies representations, Cebebras transparently supports Pytorch and Tensorflow, only requiring a few lines of changed code.
Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-... (Dr. Cutress's video on TechTechPotato is also good).
I’ve learned to be skeptical of such claims. TPUs made that same claim, but it’s hard to use it for real work, since you don’t have access to the 300GB TPU host memory / cpu, where you infeed training examples.
There is a very specific test for “supports pytorch / tendorflow”: show an MLPerf imagenet resnet benchmark. It’s impossible to fake that. If you come anywhere close to TPUs in tensorflow, people (like me) will leap: https://mlcommons.org/en/training-normal-06/
Till then, it’s a “proof, please” type of situation.
There is a very specific test for “supports pytorch / tendorflow”: show an MLPerf imagenet resnet benchmark. It’s impossible to fake that. If you come anywhere close to TPUs in tensorflow, people (like me) will leap: https://mlcommons.org/en/training-normal-06/
Till then, it’s a “proof, please” type of situation.
Software will lag new hardware. Duh. The question is whether this is solvable (given a couple of master devs)
Cerebras itself commented that DL is not their first use case. They mentioned fluid dynamics instead. (I can find the reference if someone is interested).
Cerebras also has a weird memory/compute ratio, so very hard for some uses.
Somewhat off-topic. If u have standard keras/TF training code, the GPU to TPU is a smooth-ish transition. No?
A re-write will squeeze another 20% but not 200%?
Cerebras itself commented that DL is not their first use case. They mentioned fluid dynamics instead. (I can find the reference if someone is interested).
Cerebras also has a weird memory/compute ratio, so very hard for some uses.
Somewhat off-topic. If u have standard keras/TF training code, the GPU to TPU is a smooth-ish transition. No?
A re-write will squeeze another 20% but not 200%?
I agree.
I have high hope for Graphcore. They stated "We will be participating in MLPerf in 2021, starting with the first training submission in the spring".
https://www.graphcore.ai/posts/graphcore-sets-new-ai-perform...
(I would ignore all benchmarks in that post except the sentence quoted above however.)
I have high hope for Graphcore. They stated "We will be participating in MLPerf in 2021, starting with the first training submission in the spring".
https://www.graphcore.ai/posts/graphcore-sets-new-ai-perform...
(I would ignore all benchmarks in that post except the sentence quoted above however.)
So we are looking for OpenCUDA for AI. Like OpenGL in the 90s.
I think apple expects you to use pretrained models and convert them to CoreML.
There is also webGPU which is experimental on Safari and Chrome. But that doesn’t touch the ML accelerators on the M1.
There is also webGPU which is experimental on Safari and Chrome. But that doesn’t touch the ML accelerators on the M1.
You're bearish on new hardware for AI training because it takes 2 months to setup? That's like saying you're bearish on the invention of transistor because there are no computers yet. This chip is for research organizations such as LLNL and ANL. Tooling and commoditization will happen.
> “if we build it, they’ll buy it.”
This is literally how any new thing is invented and commercialized.
> “if we build it, they’ll buy it.”
This is literally how any new thing is invented and commercialized.
Well, two months has a way of turning into two years. June 2019 was right around when I fell in love with TPUs, and we’re still happily TPUing today.
Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosions of allocations in the forward pass for unclear reasons).
Let’s put it this way. If you contract me to implement a full gpt model on this new hardware, two months full time work would be my minimum estimate. That’s 40 hours a week of focused effort, with no breaks and no other projects. I don’t know about you, but two months with nothing to show can demoralize most teams I’ve worked with.
The point was, we’re early in ML’s lifecycle. And, like React for webdev, people are slow to change their habits — for better or worse, pytorch and tensorflow are the APIs people think in. So if you want your hardware to be widely adopted, you need tooling that supports the workflows people have spent months learning.
Jax is on the horizon too. Supposedly they’re launching something soon that might tip the scale in their favor. Perhaps an ambitious hardware vendor could capture the future market by preemptively implementing Jax support. But perhaps not: at that point you’d be competing toe to toe with Google’s TPU offering, since Jax is Google’s dogfood.
Right now my money is on TPUs, partly because of their fantastic support staff. But maybe some other company will come along and offer a better integrated cloud experience.
Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosions of allocations in the forward pass for unclear reasons).
Let’s put it this way. If you contract me to implement a full gpt model on this new hardware, two months full time work would be my minimum estimate. That’s 40 hours a week of focused effort, with no breaks and no other projects. I don’t know about you, but two months with nothing to show can demoralize most teams I’ve worked with.
The point was, we’re early in ML’s lifecycle. And, like React for webdev, people are slow to change their habits — for better or worse, pytorch and tensorflow are the APIs people think in. So if you want your hardware to be widely adopted, you need tooling that supports the workflows people have spent months learning.
Jax is on the horizon too. Supposedly they’re launching something soon that might tip the scale in their favor. Perhaps an ambitious hardware vendor could capture the future market by preemptively implementing Jax support. But perhaps not: at that point you’d be competing toe to toe with Google’s TPU offering, since Jax is Google’s dogfood.
Right now my money is on TPUs, partly because of their fantastic support staff. But maybe some other company will come along and offer a better integrated cloud experience.
Do you have a longer writeup on your experience performance tuning for the TPU? Is https://cloud.google.com/tpu/docs/performance-guide the best resource in this area?
Sadly I have nothing to offer yet, other than to DM me on Twitter with questions. Always happy to answer basic ones; I love this stuff.
But you’re right, I should write up something. That guide is pretty good, but it doesn’t walk you through anything specific; it shows you a map, but doesn’t take you on a trip, so to speak.
Some tips:
Use tf.name_scopes! You probably use them for variables, but there’s a different one for operations. If you make an @op_scope decorator, you can use it on all your ML functions and immediately get lots of insights as to where the XLA ops end up mapping to in your source code.
As much as it pains me to say this, avoid tensorflow 2 style code like the plague. Pretend that if you use eager execution, someone will jump out of the bushes and shoot you. I technically use TF2.4 now, but it’s still session / graph-based, not the new tf.function magic. The new magic pipeline seems to be slower, harder to use, harder to debug, and very likely to explode when you do anything even slightly different than the tutorial examples. YMMV, and maybe things are better now (or in a future release).
The profiler is magical; leverage it whenever you can. My workflow is to start up a TPU run, then ssh into my server and fire up a Tensorboard to that model dir. Then I manually navigate to <tensorboard_url>/#profile (because the “profile” button doesn’t seem to show up in the menu anymore) which then lets you “capture profile”.
Make sure your TPU version matches your Tensorboard version. At this point we use TPU version 2.4.0, Tensorboard 2.4.1. If you get mysterious errors, this is likely the reason things are going wrong. Even TPU version 2.3.0 wasn’t enough.
Happily, the new profiling tooling is totally badass. The trace viewer is great, the op profiling is ok (though I wish it would show me all the damn ops, instead of “helpfully” hiding all but the top N ops), and the memory viewer is incredible. You can see exactly where in your pipeline is causing “peak memory usage”, what the peak is (down to the kilobyte), and have at least some idea of what’s causing it.
It’s not effortless though. The XLA fusion ops sort of make it harder to track down what’s doing what. (TF compiler is very powerful, but the trade off is that you almost never have manual control over memory usage, which can be frustrating).
All in all (or all-to-all, ha) it’s a lot of fun if you like seeing expensive hardware go brrrr, as I do. It makes it all worth it when your loss drops from 11 to 3 overnight on a 430m parameter gpt model. :)
But you’re right, I should write up something. That guide is pretty good, but it doesn’t walk you through anything specific; it shows you a map, but doesn’t take you on a trip, so to speak.
Some tips:
Use tf.name_scopes! You probably use them for variables, but there’s a different one for operations. If you make an @op_scope decorator, you can use it on all your ML functions and immediately get lots of insights as to where the XLA ops end up mapping to in your source code.
As much as it pains me to say this, avoid tensorflow 2 style code like the plague. Pretend that if you use eager execution, someone will jump out of the bushes and shoot you. I technically use TF2.4 now, but it’s still session / graph-based, not the new tf.function magic. The new magic pipeline seems to be slower, harder to use, harder to debug, and very likely to explode when you do anything even slightly different than the tutorial examples. YMMV, and maybe things are better now (or in a future release).
The profiler is magical; leverage it whenever you can. My workflow is to start up a TPU run, then ssh into my server and fire up a Tensorboard to that model dir. Then I manually navigate to <tensorboard_url>/#profile (because the “profile” button doesn’t seem to show up in the menu anymore) which then lets you “capture profile”.
Make sure your TPU version matches your Tensorboard version. At this point we use TPU version 2.4.0, Tensorboard 2.4.1. If you get mysterious errors, this is likely the reason things are going wrong. Even TPU version 2.3.0 wasn’t enough.
Happily, the new profiling tooling is totally badass. The trace viewer is great, the op profiling is ok (though I wish it would show me all the damn ops, instead of “helpfully” hiding all but the top N ops), and the memory viewer is incredible. You can see exactly where in your pipeline is causing “peak memory usage”, what the peak is (down to the kilobyte), and have at least some idea of what’s causing it.
It’s not effortless though. The XLA fusion ops sort of make it harder to track down what’s doing what. (TF compiler is very powerful, but the trade off is that you almost never have manual control over memory usage, which can be frustrating).
All in all (or all-to-all, ha) it’s a lot of fun if you like seeing expensive hardware go brrrr, as I do. It makes it all worth it when your loss drops from 11 to 3 overnight on a 430m parameter gpt model. :)
Thank you for the tips! Whenever I hear about new accelerators, my first question is: "how do people in the real world run fast code on this"? Because (related to your above points), you don't use an accelerator for things to just run, they have to run fast. Otherwise, you'd just use a CPU.
Detailed Question 1: do you have examples of fusion making things hard? Is there a way to nudge the compiler to not fuse or create a symbol table tracking fusion? Does fusion cause issues with the name_scopes?
Detailed Question 2: isn't Pytorch eager execution? Do you know how it compares to Tensorflow's eager execution?
General Request: I'm in a more theoretical position, writing papers on programming languages for accelerators https://aetherling.org/ and TAing courses on accelerators http://cs149.stanford.edu/fall20. So, I'm excited to see people's practical experiences using these accelerators in industry. It would be very enlightening (if you have time) to write up a comparison of tuning a model for an A100's tensor cores vs a TPU. This seems like the key trade-off in comparing architectures.
Detailed Question 1: do you have examples of fusion making things hard? Is there a way to nudge the compiler to not fuse or create a symbol table tracking fusion? Does fusion cause issues with the name_scopes?
Detailed Question 2: isn't Pytorch eager execution? Do you know how it compares to Tensorflow's eager execution?
General Request: I'm in a more theoretical position, writing papers on programming languages for accelerators https://aetherling.org/ and TAing courses on accelerators http://cs149.stanford.edu/fall20. So, I'm excited to see people's practical experiences using these accelerators in industry. It would be very enlightening (if you have time) to write up a comparison of tuning a model for an A100's tensor cores vs a TPU. This seems like the key trade-off in comparing architectures.
I'd be interested in contributing to the course, if you need anything specific.
Here's a tensorboard URL that will probably stop working within a few days. http://bulma.tensorfork.com:31337/#profile
You can view the memory profiler by using the dropdown menus on the left. Here's a particularly chonky CrossReplicaSum: https://i.imgur.com/CcdJzLj.png
The game here is to keep that number at the top -- peak memory usage -- below 15GB. In practice, TPUv3-8's run out of memory at around 14.5GB, which immediately crashes (and hence you can't profile it). So we're always trying to get as close to 15GB as possible.
The first thing you immediately notice is that real-life training runs are very spiky. Different parts of the pipeline end up allocating wildly different amounts of memory. There's almost no such thing as a constant memory usage pipeline (which I was dismayed to discover).
In this profiling run, you can see that there's a big ass-spike at ~4000 on the X axis. The green bar marks the lifetime of the operation causing the highest peak memory usage. Different operations depend on each other, forming a chain of allocations. a + b takes 'a' and 'b' as inputs, and any temporary tensor reachable by either 'a' or 'b' cannot be freed until a + b is finished executing. Ditto for all other operations.
So you see, it's easy to accidentally build a "tower" of allocations, rather than a flat line. Thus, your total model parameter count is severely limited compared to what it could be, since in this situation the only way to reduce memory usage (without rewriting the code) is to scale down the model params.
Hovering over the big ass-orange allocation, we see that the shape is -- gosh, tensorboard is infuriating sometimes. I tried to copy-paste the shape, but whenever I move the mouse off of the allocation, the info on the left vanishes. Anyway, the shape is F32[32,2048,1,12608][1,3,0,2]. It means the cross replica sum is happening across TPU cores 1, 3, 0, and 2; it's a float32 sum; the batch size is 32; the hidden dimension is 2048, and the vocab dimension is 12,608. Since it's across four cores, multiply that dim by 4, and the total vocab size is 50,432, which is exactly right for a GPT model (https://nv-adlr.github.io/MegatronLM has details).
So right away, we can see that (a) the non-peak memory usage is around 4GB or so, and (b) the peak mem usage of the spike is around 12GB. That means if we eliminate the spike, we can scale up our model by more than 3x, if usage scales linearly. (Sometimes you get lucky and it's linear, other times something is superlinear. It's more or less linear in my experience.)
So how do we eliminate the spike? Heck if I know how the Google pros do it, but my way of doing it is to unstack along the batch dimension and perform each operation sequentially.
In other words, the total memory usage here is O(32 * 2048 * 12,608) which is quite hefty. By unstacking along the batch dimension, you get 32 tensors, each of size 2048 by 12,608. Therefore, if you do each operation sequentially, the temporary buffer is now O(2048 * 12,608), giving us a 32x savings.
Is this slower? Surprisingly, more often than not, it's as fast or faster. The reason is subtle: slowdowns occur due to memory bandwidth and network bandwidth. As long as the unstack is strictly a memory bandwidth effect, then it's just as fast, because you're trading CPU cycles for memory -- and you have tons of CPU cycles here, since it's a TPU core. (The TPU core utilization in our experience is always around 30%, and we've never seen it higher than 65%.) So you should always, always make this trade whenever possible.
Network traffic is trickier. This is a cross-replica sum, which means it's sending the tensors across the network to each TPU core. The TPU cores are connected via a high speed interconnect nexus thingie, but like all bottlenecks, this one has a limit. It's a very high limit, but it's not endless.
The only solution I've found is to think of an idea and then test that idea. Reasoning from first principles almost never works for me. I've seen others solve problems by reasoning from first principles, so it's possible that I'm simply stupid. But I find it's much more effective to try as many ideas as possible, as quickly as possible. You often end up surprised.
I'll type more stuff later if I feel like it, or you can ask more followup questions. Feel free to DM me on twitter if you'd like to chat in realtime sometime.
Here's a tensorboard URL that will probably stop working within a few days. http://bulma.tensorfork.com:31337/#profile
You can view the memory profiler by using the dropdown menus on the left. Here's a particularly chonky CrossReplicaSum: https://i.imgur.com/CcdJzLj.png
The game here is to keep that number at the top -- peak memory usage -- below 15GB. In practice, TPUv3-8's run out of memory at around 14.5GB, which immediately crashes (and hence you can't profile it). So we're always trying to get as close to 15GB as possible.
The first thing you immediately notice is that real-life training runs are very spiky. Different parts of the pipeline end up allocating wildly different amounts of memory. There's almost no such thing as a constant memory usage pipeline (which I was dismayed to discover).
In this profiling run, you can see that there's a big ass-spike at ~4000 on the X axis. The green bar marks the lifetime of the operation causing the highest peak memory usage. Different operations depend on each other, forming a chain of allocations. a + b takes 'a' and 'b' as inputs, and any temporary tensor reachable by either 'a' or 'b' cannot be freed until a + b is finished executing. Ditto for all other operations.
So you see, it's easy to accidentally build a "tower" of allocations, rather than a flat line. Thus, your total model parameter count is severely limited compared to what it could be, since in this situation the only way to reduce memory usage (without rewriting the code) is to scale down the model params.
Hovering over the big ass-orange allocation, we see that the shape is -- gosh, tensorboard is infuriating sometimes. I tried to copy-paste the shape, but whenever I move the mouse off of the allocation, the info on the left vanishes. Anyway, the shape is F32[32,2048,1,12608][1,3,0,2]. It means the cross replica sum is happening across TPU cores 1, 3, 0, and 2; it's a float32 sum; the batch size is 32; the hidden dimension is 2048, and the vocab dimension is 12,608. Since it's across four cores, multiply that dim by 4, and the total vocab size is 50,432, which is exactly right for a GPT model (https://nv-adlr.github.io/MegatronLM has details).
So right away, we can see that (a) the non-peak memory usage is around 4GB or so, and (b) the peak mem usage of the spike is around 12GB. That means if we eliminate the spike, we can scale up our model by more than 3x, if usage scales linearly. (Sometimes you get lucky and it's linear, other times something is superlinear. It's more or less linear in my experience.)
So how do we eliminate the spike? Heck if I know how the Google pros do it, but my way of doing it is to unstack along the batch dimension and perform each operation sequentially.
In other words, the total memory usage here is O(32 * 2048 * 12,608) which is quite hefty. By unstacking along the batch dimension, you get 32 tensors, each of size 2048 by 12,608. Therefore, if you do each operation sequentially, the temporary buffer is now O(2048 * 12,608), giving us a 32x savings.
Is this slower? Surprisingly, more often than not, it's as fast or faster. The reason is subtle: slowdowns occur due to memory bandwidth and network bandwidth. As long as the unstack is strictly a memory bandwidth effect, then it's just as fast, because you're trading CPU cycles for memory -- and you have tons of CPU cycles here, since it's a TPU core. (The TPU core utilization in our experience is always around 30%, and we've never seen it higher than 65%.) So you should always, always make this trade whenever possible.
Network traffic is trickier. This is a cross-replica sum, which means it's sending the tensors across the network to each TPU core. The TPU cores are connected via a high speed interconnect nexus thingie, but like all bottlenecks, this one has a limit. It's a very high limit, but it's not endless.
The only solution I've found is to think of an idea and then test that idea. Reasoning from first principles almost never works for me. I've seen others solve problems by reasoning from first principles, so it's possible that I'm simply stupid. But I find it's much more effective to try as many ideas as possible, as quickly as possible. You often end up surprised.
I'll type more stuff later if I feel like it, or you can ask more followup questions. Feel free to DM me on twitter if you'd like to chat in realtime sometime.
Graphics APIs and deep learning frameworks are really nothing alike. In graphics, save for a few new features, the goal every year is the same: draw more and faster. Deep learning is evolving much faster than that. The answer isn’t an API; it’s powerful and mostly transparent automatic differentiation. Anything that is just an API will forever be behind and is much less useful for active research. There are a few projects around improving AD support, but it’s a relatively difficult problem because to do it best you need deep compiler integration.
The answer is an API, like NNAPI. AD is a frontend concern and doesn't really matter to accelerators.
For AD, I am bullish for Enzyme, which does AD on LLVM IR, avoiding deep compiler integration: https://enzyme.mit.edu/
For AD, I am bullish for Enzyme, which does AD on LLVM IR, avoiding deep compiler integration: https://enzyme.mit.edu/
I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the derivatives. If you’re using a Python wrapper for this you’ll need codegen for interacting with the generated IR, too. The upside with that approach is that it might more easily work across languages. The downside is that LLVM IR can’t rely on the rich invariants that even C exposes, which makes codegen and optimization harder.
An API is always going to be necessarily behind state-of-the-art because often research depends on inventing things that don’t fit within an existing API.
An API is always going to be necessarily behind state-of-the-art because often research depends on inventing things that don’t fit within an existing API.
I think we are talking past each other. sillysaurusx is talking about OpenGL/Direct3D as a portability layer for hardwares. API vs AD is not a hardware portability layer concern. The proposal is to standardize OpenGL/Direct3D-like hardware portability layer defined by API. API vs AD you are talking about happens far above that and it is irrelevant.
Today, things like PyTorch are AD systems, optimization systems, libraries of high performance interoperable common blocks, and more. This massive complexity is why there is no standard. If you have a compiler front end that can AD arbitrary code, then the framework no longer has to worry about that. If you have a compiler backend that can target multiple devices, then the framework doesn’t have to do that, either. The size and complexity of PyTorch/TensorFlow is exactly why there isn’t a standard hardware portable NN API. A sane API of hardware independent common blocks can’t happen without all the front and backend compiler stuff first.
A PyTorch/TensorFlow-like API is always going to be massive, complicated, and hard to port to new hardware targets. Additionally, the complexity and reliance on C alone will make integrating exotic new concepts like say, differential equation solvers, extremely laborious.
A PyTorch/TensorFlow-like API is always going to be massive, complicated, and hard to port to new hardware targets. Additionally, the complexity and reliance on C alone will make integrating exotic new concepts like say, differential equation solvers, extremely laborious.
You might like this talk that describes parts of the compiler ecosystem that are lacking standardization for accelerators
https://www.youtube.com/watch?v=4HgShra-KnY
https://www.youtube.com/watch?v=4HgShra-KnY
> What we need is a unified API that can easily support multiple backends
Android standardized NNAPI and on-device inference is in pretty good shape. As you said, training is different matter.
Android standardized NNAPI and on-device inference is in pretty good shape. As you said, training is different matter.
Thr Anandtech article from a other comment here indicates that both PyTorch and Tensorflow are already running on it.
>A key to the design is the custom graph compiler, that takes pyTorch or TensorFlow and maps each layer to a physical part of the chip, allowing for asynchronous compute as the data flows through.
https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
>A key to the design is the custom graph compiler, that takes pyTorch or TensorFlow and maps each layer to a physical part of the chip, allowing for asynchronous compute as the data flows through.
https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
> We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements.
You mean one of the most rapid periods of graphical improvement in history? Growth was just too breakneck to standardize for a bit.
You mean one of the most rapid periods of graphical improvement in history? Growth was just too breakneck to standardize for a bit.
sillysaurusx's comment goes on to acknowledge that the major graphics libraries (OpenGL and Direct3D) did a pretty good job of abstracting away/indirecting away the architectural changes. As you indicate, the architectural changes were made for good reason, and translated to real improvements in performance/capabilities.
If AI acceleration chips aren't able to offer similar API standardisation/stability then it's not the same as what happened with GPUs, which was sillysaurusx's point.
If AI acceleration chips aren't able to offer similar API standardisation/stability then it's not the same as what happened with GPUs, which was sillysaurusx's point.
The article mentions that developers use standard PyTorch and Tensorflow.
You might be interested in this for your M1 MBA: https://github.com/apple/tensorflow_macos
That was actually what I was referring to! It's super weird. I like it, and it's what I use primarily, but only because no other version of tensorflow will install.
Their compiler is closed source, so it's almost impossible to tell what's going on. But, when I connect using tf.Session(), then .list_devices() shows only CPU available.
However, when I enable the TF_MLC_LOGGING=1 magic variable, it does seem to be printing out messages that indicates it's doing some kind of graph substitution under the hood. Therefore, I assume that this is the intended usage mode.
In other words, there seems to be zero difference between the "CPU" and the "GPU". Normally you can say "Do this on the CPU" while "do that on the GPU." But not with this.
Hopefully they'll open source the code sometime this century so that it's clearer what the heck it's doing. For now, though, it's reasonably fast in whatever this "CPU" mode is -- I only need to run unit tests on my laptop anyway, since all training happens on TPUs. So I ended up happy.
(For the first day or so, I was panicking that I was going to have no working tensorflow whatsoever on my M1 laptop, which would've necessitated a swift return + substitution.)
Their compiler is closed source, so it's almost impossible to tell what's going on. But, when I connect using tf.Session(), then .list_devices() shows only CPU available.
However, when I enable the TF_MLC_LOGGING=1 magic variable, it does seem to be printing out messages that indicates it's doing some kind of graph substitution under the hood. Therefore, I assume that this is the intended usage mode.
In other words, there seems to be zero difference between the "CPU" and the "GPU". Normally you can say "Do this on the CPU" while "do that on the GPU." But not with this.
Hopefully they'll open source the code sometime this century so that it's clearer what the heck it's doing. For now, though, it's reasonably fast in whatever this "CPU" mode is -- I only need to run unit tests on my laptop anyway, since all training happens on TPUs. So I ended up happy.
(For the first day or so, I was panicking that I was going to have no working tensorflow whatsoever on my M1 laptop, which would've necessitated a swift return + substitution.)
That seems very apple honestly, they’d rather you emit pure metal primitives (is that the right term? Idk) and then let their backend schedule on their soc- because for all we know, they have some special asic ip that they added/or will add and can take advantage.
With the disclaimer that I know nothing about this: doesn't the M1 have a separate "Neural Engine" for this? So it's using neither the CPU nor the GPU cores for TensorFlow?
The philosophy is working as it often does in the early days/years of a technology. The customers for this have problems at an entirely different scale. It's worth it for them to drop millions on hardware and potentially millions more to (re)write the software as needed. Sure, they'd prefer to be buying commodity hardware with standardized APIs. But since that doesn't currently exist for this class of performance, they'll pay up. People did the same back in the days of Cray/SGI/Lisp Machines and so on. These custom solutions can be quite profitable until the competition gets to the 'good enough' stage and then orders start drying up.
If you're looking for standardized hardware and software for AI, Apple is the wrong platform to be on. I'm fairly confident it won't be happening there. (I don't say that to be a hater: see the direction they've gone re: GPU APIs... it's just Apple being Apple)
If you're looking for standardized hardware and software for AI, Apple is the wrong platform to be on. I'm fairly confident it won't be happening there. (I don't say that to be a hater: see the direction they've gone re: GPU APIs... it's just Apple being Apple)
If you look at it historically and put it in scale with the AI winter, 2 months is next to nothing..
I don't really agree, at least from a longer term perspective. It's early days yet, but XLA seems to be a promising intermediate representation for allowing the DL frameworks to run on a wider array of hardware without user-facing software changes. It has traction with Google, NVIDIA, IIRC Intel and maybe more (others are definitely using the same approach of compute graph splitting and subgraph scheduling, but I'm not certain if they are using XLA specifically - I know some aren't like Mindspore).
XLA has already proven its value by allowing PyTorch to run on TPUs (shittily, but that appears to be more of a VM/GCP infra problem than an XLA problem). The work done for TPUs (and to a lesser extent for GPU optimization) has started to expose some of the major issues and so work can start on addressing them (the cost of dynamic XLA compilation as tensor shapes change and how lots of important code assumes that accelerator-to-CPU communication isn't tooo expensive, but it's is a huge issue when trying to compile the graph into machine-specific code with XLA or similar to because it forces you to only be able to compile small subgraphs).
It's early, but the rise of a really effective IR in XLA combined with the huge amount of resources that Google/NVIDIA can pour into XLA makes me very bullish on purpose-built hardware for AI training. It will take a while I admit.
XLA has already proven its value by allowing PyTorch to run on TPUs (shittily, but that appears to be more of a VM/GCP infra problem than an XLA problem). The work done for TPUs (and to a lesser extent for GPU optimization) has started to expose some of the major issues and so work can start on addressing them (the cost of dynamic XLA compilation as tensor shapes change and how lots of important code assumes that accelerator-to-CPU communication isn't tooo expensive, but it's is a huge issue when trying to compile the graph into machine-specific code with XLA or similar to because it forces you to only be able to compile small subgraphs).
It's early, but the rise of a really effective IR in XLA combined with the huge amount of resources that Google/NVIDIA can pour into XLA makes me very bullish on purpose-built hardware for AI training. It will take a while I admit.
Mr/mrs anonymous HN person, please put some info in your profile. You clearly have some deep knowledge of TPUs that I didn’t expect to pop up offhandedly on HN. You’re correct on all counts: dynamic tensor shapes are more or less impossible with XLA, making it more or less impossible to train a model with arbitrary image size inputs, even though the math would allow for that; the pytorch XLA work on TPUs is indeed kind of shitty, and I’m surprised as heck that literally anyone said this except me; and XLA as an IR is promising for portability. Now I’m curious what you’ve been doing to have experienced these things, since there didn’t seem to be many others who have (or at least, who are vocal about it).
I agree with you, but I think we differ on our timetables. I am bearish for the next two years, at which point I’ll awaken from my slumber and become a flaming bull. (It helps to remember that “we overestimate the impact of years, but underestimate the impact of decades.” I try to plan accordingly.)
In other words, if you’re bullish that two years from now we’ll start seeing portability implemented in the field across various HPC chips, then we fully agree. But that’s also a glacial pace; GPT-2 changed the world almost two years ago now, and DALL-E seems to be the next frontier for doing interesting generative work. So, we’ll split the difference and say that the bears and bulls will meet in two years for a deep learning hackathon. As a bonus, the pandemic will be over by then, so it can be an in-person meetup.
I agree with you, but I think we differ on our timetables. I am bearish for the next two years, at which point I’ll awaken from my slumber and become a flaming bull. (It helps to remember that “we overestimate the impact of years, but underestimate the impact of decades.” I try to plan accordingly.)
In other words, if you’re bullish that two years from now we’ll start seeing portability implemented in the field across various HPC chips, then we fully agree. But that’s also a glacial pace; GPT-2 changed the world almost two years ago now, and DALL-E seems to be the next frontier for doing interesting generative work. So, we’ll split the difference and say that the bears and bulls will meet in two years for a deep learning hackathon. As a bonus, the pandemic will be over by then, so it can be an in-person meetup.
Ah I see - I think we're pretty much on the same page in terms of timetables. Although if you include TPU, I think it's fair to say that custom accelerators are already a moderate success.
Updated my profile. I've been working on DL training platforms and distributed training benchmarking for a bit so I've gotten a nice view into the GPU/TPU battle.
Shameless plug: you should check out the open-source training platform we are building, Determined[1]. One of the goals is to take our hard-earned expertise on training infrastructure and build a tool where people don't need to have that infrastructure expertise. We don't support TPUs, partially because a lack of demand/TPU availability, and partially because our PyTorch TPU experiments were so unimpressive.
[1] GH: https://github.com/determined-ai/determined, Slack: https://join.slack.com/t/determined-community/shared_invite/...
Updated my profile. I've been working on DL training platforms and distributed training benchmarking for a bit so I've gotten a nice view into the GPU/TPU battle.
Shameless plug: you should check out the open-source training platform we are building, Determined[1]. One of the goals is to take our hard-earned expertise on training infrastructure and build a tool where people don't need to have that infrastructure expertise. We don't support TPUs, partially because a lack of demand/TPU availability, and partially because our PyTorch TPU experiments were so unimpressive.
[1] GH: https://github.com/determined-ai/determined, Slack: https://join.slack.com/t/determined-community/shared_invite/...
+1 please put some info in your profile.
I'm dying to know what they work on, either officially or in their spare time. https://news.ycombinator.com/item?id=26586151
> from a longer term perspective
Yes.
Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed manner -- i.e., without having to backpropagate gradients, which requires a lot of complex bookkeeping and plumbing for distributed training.
Recent progress suggests this will happen. See, for example:
https://arxiv.org/abs/2006.04182
https://arxiv.org/abs/2103.03725
https://arxiv.org/abs/2010.01047
I for one am excited to see what happens over the next decade as it becomes trivial to train/use models with 1K, 1M, or 1B times more dense connections than present state-of-the-art models.
Yes.
Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed manner -- i.e., without having to backpropagate gradients, which requires a lot of complex bookkeeping and plumbing for distributed training.
Recent progress suggests this will happen. See, for example:
https://arxiv.org/abs/2006.04182
https://arxiv.org/abs/2103.03725
https://arxiv.org/abs/2010.01047
I for one am excited to see what happens over the next decade as it becomes trivial to train/use models with 1K, 1M, or 1B times more dense connections than present state-of-the-art models.
But they do have PyTorch support??? They ship a 'cerebras graph compiler' extension for pytorch that allows you to replace the default optimizer with theirs.
> That means you’re looking at two months of R&D minimum to get everything rewritten,
For a hobbiest, sure, that's a problem.
But for a big company with a big ML research team already, that isn't an issue - they just assign a few people to work on it, and in a few months it's done. If you're running any model at scale anyway you probably want to rewrite everything to make it run efficiently on your hardware.
For a hobbiest, sure, that's a problem.
But for a big company with a big ML research team already, that isn't an issue - they just assign a few people to work on it, and in a few months it's done. If you're running any model at scale anyway you probably want to rewrite everything to make it run efficiently on your hardware.
What does the programming model look like for one of these (like at the assembly level)? I’m not even sure what to google.
The article states that the programming model is TensorFlow or PyTorch. However, I'm not clear on the details of customizing TensorFlow or PyTorch code to run efficiently on this chip.
I suspect it's more like an FPGA - massively parallel structures that don't fit traditional "programming" well.
Incidentally, FPGAs are also insanely expensive at the high end.
Incidentally, FPGAs are also insanely expensive at the high end.
Super curious, how much do one of these cost? Like 10K,100K, or 1M range?
First gen was ~ 2.5M usd per machine as per the video link
Any article or headline talking about transistor count is just a puff piece. The performance matters, the number of transistors never does, and I have never purchases anything based on the number of transistors.
I'm not going to buy one, but its more because I don't have several million dollars to spare.
I get that, but it's just a thing I've noticed about tech puff pieces. If the transistor count is front and center it's probably not worth reading.
Interesting consider that if this "chip" was manufactured with the feature size of the MC68000, the chip would be about 1/4 mile square.
Or another way to grok the process shrinkage is to consider that the MC68000, if manufactured with current process dimensions, would fit within the space of a single gate of the original process.
> the size ... [is] ... 46,225 square millimeters
According to google calculations, this is just under 6 square feet.
According to google calculations, this is just under 6 square feet.
No…? https://www.google.com/search?q=46%2C225+square+millimeters+...
> 0.49756176 square foot
> 0.49756176 square foot
I think it might be closer to 215mm x 215mm or 0.49 square feet.
Though that is unimaginably large to me for a silicon chip. I can't comprehend it - the engineering behind it would be incredible.
Though that is unimaginably large to me for a silicon chip. I can't comprehend it - the engineering behind it would be incredible.
This is the first I'm coming across Cerebras and interested here to understand more. Are these already in use integrated in some pc/servers or is it some dedicated server which you need to integrate into some motherboard, ...
Polyhedral LLVM compiler optimization > Quantum Computing Technology
Well at least on relevance* to active problems.
Well at least on relevance* to active problems.
[1] https://www.eetimes.com/powering-and-cooling-a-wafer-scale-d...