Nvidia will build 700-petaflop supercomputer for University of Florida(venturebeat.com)
venturebeat.com
Nvidia will build 700-petaflop supercomputer for University of Florida
https://venturebeat.com/2020/07/21/nvidia-collaborates-with-the-university-of-florida-to-build-700-petaflop-ai-supercomputer/
111 comments
One of Nvidia's confounders is actually a UF alumnus. Good way to give back!
Sigh, I wish people wouldn’t say “petaflop” for these.
https://www.nvidia.com/en-us/data-center/a100/ is the most official reference. If you scroll to the bottom, you’ll see that an A100 part can do ~20 Teraflops (either FP32 or FP64 in little-matrix aka tensor mode). When they say “each A100 can do 5 petaflops”, they mean each DGX which has 8 such cards and thus they mean 600-ish something-ops per card. The generous assumption is that’s FP16 or bfloat16 for sparse ops, and therefore they are “flops”.
The reality is that if someone says “supercomputer” in the general sense, they mean scientific computing and so mean a double-precision LINPACK benchmark. The 1120 A100 parts (8x140) doing 20 “real” teraflops each has an absolute peak of about 22 Petaflops (and older code without tensor mode would be half that on FP64).
tl;dr: ML isn’t scientific computing, and those are different flops, but “10 petaflops” just doesn’t sound as impressive.
https://www.nvidia.com/en-us/data-center/a100/ is the most official reference. If you scroll to the bottom, you’ll see that an A100 part can do ~20 Teraflops (either FP32 or FP64 in little-matrix aka tensor mode). When they say “each A100 can do 5 petaflops”, they mean each DGX which has 8 such cards and thus they mean 600-ish something-ops per card. The generous assumption is that’s FP16 or bfloat16 for sparse ops, and therefore they are “flops”.
The reality is that if someone says “supercomputer” in the general sense, they mean scientific computing and so mean a double-precision LINPACK benchmark. The 1120 A100 parts (8x140) doing 20 “real” teraflops each has an absolute peak of about 22 Petaflops (and older code without tensor mode would be half that on FP64).
tl;dr: ML isn’t scientific computing, and those are different flops, but “10 petaflops” just doesn’t sound as impressive.
You're the "disclaimer: I work for google cloud" person, right? Maybe you can help solve this problem (and it's a real one, it massively sucks that everyone counts whatever "flops" they want).
Start here: https://cloud.google.com/tpu
> Cloud TPU v3 Pod
> 100+ petaflops
Is the Cloud TPU v3 even functionally capable of computing in fp64 at any performance?
Start here: https://cloud.google.com/tpu
> Cloud TPU v3 Pod
> 100+ petaflops
Is the Cloud TPU v3 even functionally capable of computing in fp64 at any performance?
Yes, (and I considered including that disclaimer).
I give them shit about that all the time. I'm an equal-opportunity complainer.
I give them shit about that all the time. I'm an equal-opportunity complainer.
I'm glad to hear it.
but it's sorta disheartening that whoever makes these decisions apparently don't care. Marketing going to market, I guess.
but it's sorta disheartening that whoever makes these decisions apparently don't care. Marketing going to market, I guess.
> Marketing going to market
This is one of those situations where the vulgate enhances comprehension - here eliminating ambiguity:
Marketing gonna market
This is one of those situations where the vulgate enhances comprehension - here eliminating ambiguity:
Marketing gonna market
Maybe we need a new unit of measure: the flip. Similar to how SN came up with the bi suffixes for binary (mebi vs. mega, etc.). The flip would be a small flop.
I agree with you as well as with the response to your comment, in that both nVidia and Google are engaging in misleading claims about their TPUs. But I'd like to add two points:
- I don't think they can get away with it when it comes to Top-500 rankings. Those rankings are based on LINPACK scores and this supercomputer would end up not scoring high enough and will be placed in the right spot in the list. So it's not a big concern to me.
- "ML isn't scientific computing" goes both ways. Sure tensor-tera-ops are not teraflops, they're specific to operations involving artificial neural networks (ANNs). But when you take the claims from the past about how much computing power it'll take for rivaling that of a human brain, and folks came up with 100 petaflops, or 1 exaflop. Well those should not be called "f"lops either. Because when it comes to brain inspired computing, ANN tensor-tera-ops are a more reliable number than FP32 or FP64. And if it's 100 peta-ops of ANN compute, well we're already past that, and can easily create a supercomputer with 1 tensor-exa-ops. And that means we have reached the hardware capacity required to emulate human-level intelligence in a machine (i.e., the only thing missing is the right set of algorithms).
Conclusion: Tensor-tera-ops are not FP ops and should not be used for placement in the Top-500 list. But tensor-tera-ops have enough significance that it warrants creating a new list Top-500-Tensor and ranking Tensor-supercomputers on that list.
- I don't think they can get away with it when it comes to Top-500 rankings. Those rankings are based on LINPACK scores and this supercomputer would end up not scoring high enough and will be placed in the right spot in the list. So it's not a big concern to me.
- "ML isn't scientific computing" goes both ways. Sure tensor-tera-ops are not teraflops, they're specific to operations involving artificial neural networks (ANNs). But when you take the claims from the past about how much computing power it'll take for rivaling that of a human brain, and folks came up with 100 petaflops, or 1 exaflop. Well those should not be called "f"lops either. Because when it comes to brain inspired computing, ANN tensor-tera-ops are a more reliable number than FP32 or FP64. And if it's 100 peta-ops of ANN compute, well we're already past that, and can easily create a supercomputer with 1 tensor-exa-ops. And that means we have reached the hardware capacity required to emulate human-level intelligence in a machine (i.e., the only thing missing is the right set of algorithms).
Conclusion: Tensor-tera-ops are not FP ops and should not be used for placement in the Top-500 list. But tensor-tera-ops have enough significance that it warrants creating a new list Top-500-Tensor and ranking Tensor-supercomputers on that list.
[deleted]
Fuly agree - TL;DR pick the right benchmark.
If you are doing AI the Top500,is meaningless and if you are doing classical HPC these ExaFlop int8 supercomputers are meaningless
If you are doing AI the Top500,is meaningless and if you are doing classical HPC these ExaFlop int8 supercomputers are meaningless
[deleted]
I'm a bit baffled.
How high are the tuitions at the university of florida?
This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
How high are the tuitions at the university of florida?
This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
No, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call it "AI flops").
The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)?
So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop supercomputer, not counting host CPUs.
Granted though, the hot new HPC approach is to approximate certain expensive high-precision calculations with ML inference instead, but that only works for certain algorithms.
The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)?
So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop supercomputer, not counting host CPUs.
Granted though, the hot new HPC approach is to approximate certain expensive high-precision calculations with ML inference instead, but that only works for certain algorithms.
The machine will be 140 nodes, which is half the size of the current #7 on the top500 list which is otherwise identical: https://top500.org/system/179842/
That #7 machine got 27.6 Pflops on HPL, so I would guess the Florida machine will get roughly half of that.
(Please keep in mind the tensor cores on the A100 can do FP64, so the theoretical peak flops per GPU is 19.5)
That #7 machine got 27.6 Pflops on HPL, so I would guess the Florida machine will get roughly half of that.
(Please keep in mind the tensor cores on the A100 can do FP64, so the theoretical peak flops per GPU is 19.5)
The FLOPs quoted by national lab supercomputers are usually measured double-precision FLOPs from HPL (High-performance Linpack - at least on the Top500 reports).
IIUC, HPL can use double-precision tensor operations. It might also be able to use mixed-precision for some computations.
Are the performance numbers for this HPC system at University of Florida also from HPL ?
IIUC, HPL can use double-precision tensor operations. It might also be able to use mixed-precision for some computations.
Are the performance numbers for this HPC system at University of Florida also from HPL ?
> IIUC, HPL can use double-precision tensor operations.
Lets put it this way: As of the A100 GPU generation, the tensor cores are able to do double precision calculations too. By using the tensor core the peak FP64 flops double to 19.5 Tflops. And yes, the CUDA BLAS library (that the HPL benchmark uses) uses the tensor cores on the A100.
> It might also be able to use mixed-precision for some computations.
There is currently a lot of interest in mixed precision algorithms, but I'm not sure that the algorithm used for HPL is amenable to that approach, nor whether it's allowed per the benchmark rules.
Lets put it this way: As of the A100 GPU generation, the tensor cores are able to do double precision calculations too. By using the tensor core the peak FP64 flops double to 19.5 Tflops. And yes, the CUDA BLAS library (that the HPL benchmark uses) uses the tensor cores on the A100.
> It might also be able to use mixed-precision for some computations.
There is currently a lot of interest in mixed precision algorithms, but I'm not sure that the algorithm used for HPL is amenable to that approach, nor whether it's allowed per the benchmark rules.
Jinx :). I really am surprised the marketing folks didn’t claim this was an ~3 exa-op supercomputer based on int4 w/ sparsity.
besides, the next generation of cray (Shasta) provides an on prem cloud environment where you can get 1000 computers or one supercomputer depending on your needs. I think that's a better deal than a supercomputer that can't do anything else.
one day when disaggregated hardware becomes common in data centers the idea of a supercomputer will seem dated
one day when disaggregated hardware becomes common in data centers the idea of a supercomputer will seem dated
It's $50 million. That's all. See https://blogs.nvidia.com/blog/2020/07/21/university-of-flori... for the numbers. It's probable the hardware is being offered at a deep discount.
The building and infrastructure will cost an additional $20m, covered by the University.
The building and infrastructure will cost an additional $20m, covered by the University.
https://lambdalabs.com/blog/demystifying-gpt-3/
According to ^, that's about 11 GPT-3s trainings worth in the cloud.
According to ^, that's about 11 GPT-3s trainings worth in the cloud.
No one who's training GPT-3 pays those prices.
I dunno if there's a deep discount.
This is 140 DGX A100 systems. The advertised price for each DGX A100 is $199,000, so 140 of them is ~$28 million. It's plausible the networking, storage, cooling, power distribution, etc (i.e. everything but the compute nodes) could cost $22 million.
This is 140 DGX A100 systems. The advertised price for each DGX A100 is $199,000, so 140 of them is ~$28 million. It's plausible the networking, storage, cooling, power distribution, etc (i.e. everything but the compute nodes) could cost $22 million.
Yes, I think the 50 mil number was off. Add discounts, tax breaks etc., and we are talking less than sticker price for the whole setup minus real estate.
I'm in favor of getting high powered compute in everyone's hands, so this is great to me.
Why?
I mean, we already have (my old 2013 desktop would - I guess - compete with a cray 1 from the 1980s). What do you expect will happen if people get more CPU?
I mean, we already have (my old 2013 desktop would - I guess - compete with a cray 1 from the 1980s). What do you expect will happen if people get more CPU?
The calculations we do nowaday are also completely different one did back in the 1980s. There’s still a lot you simply can not do on a home computer.
Better simulations? Successfully working on previously intractable problems?
But most people ie. not 'everyone' as the post talked about, is doing that. Most just want web browsing + LibreOffice stuff. Oh, and games I suppose.
It's "anchored" by $50 million (of which half is support). But what does "anchored" mean here? The article also lists an additional $20 million from UF. I wouldn't be surprised if there was even more.
Its a $70M effort "anchored" by $25M cash donation and $25M in hardware. I've heard anchored used to indicate an initial "getting us started" donation at the beginning of a fund raising drive. Its easier to raise the final portion once you've got momentum and people think the project is likely to be successfully funded.
I'm glad they spent that $50 million on research rather than on next year's football program. Seems to be a decision most big state colleges would not make.
> The initiatives and HiPerGator enhancements are anchored by a $50 million gift — $25 million from UF alumnus and Nvidia fellow Chris Malachowsky and $25 million in hardware, software, training, and services from Nvidia. UF will also make investments around its new machine “well beyond” $20 million, targeted at upgrading its on-campus datacenter.
> How high are the tuitions at the university of florida?
In-state tuition at the University of Florida is inexpensive compared to most major US universities and is generally considered one of the better values among universities. ~$6,400 per year in-state. Florida State is also comparable in cost.
https://www.sfa.ufl.edu/cost/
In-state tuition at the University of Florida is inexpensive compared to most major US universities and is generally considered one of the better values among universities. ~$6,400 per year in-state. Florida State is also comparable in cost.
https://www.sfa.ufl.edu/cost/
The article mentions that the money/hardware was donated by Nvidia co-founders.
One of the founders, Chris Malachowsky, is an alum:
https://en.wikipedia.org/wiki/Chris_Malachowsky
https://en.wikipedia.org/wiki/Chris_Malachowsky
Research at most US major universities is funded from federal grants, not from tuition. A place like the University of Florida, MIT, Michigan State, whatever, is conducting research with government money in a manner very similar to a national lab.
Indeed, and to paraphrase my PhD adviser, we currently have more research dollars than good students to spend them on.
In addition to what everyone else has said, UF has over 50,000 students, making it one of the five largest universities in the USA, iirc.
Correct. Florida has 3 of the top 10 by enrollment. UCF, UF, FIU, and USF. Georgia made the list, go dawgs.
I moved to Miami from Boulder, and was working out of an office that was on FIU campus for a good while.
The first time I walked on the campus, it blew my mind. I'd never heard about Florida International University as an American, but it's easily the biggest and most beautiful I've seen.
The first time I walked on the campus, it blew my mind. I'd never heard about Florida International University as an American, but it's easily the biggest and most beautiful I've seen.
[deleted]
Precisely explains why NVIDIA did that - to hook up all these 50k students to CUDA. Much more profitable in the long run.
There's a lot of ways to program these GPU supercomputers, and you are maybe overestimating how much of the HPC code is written in CUDA. Some is, for sure, but there's also a lot using OpenACC or OpenMP (C, C++, or Fortran with annotations), or frameworks like Kokkos.
Recently, it also became possible to use C++17 stdpar and get GPU acceleration (https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algor...).
Recently, it also became possible to use C++17 stdpar and get GPU acceleration (https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algor...).
All 50k Students aren't engineers...
As a UF CS grad, I bet less than 100 students will get hands on this computer each year. Maybe way less than 100.
Although I wasn't a graduate student so maybe way more of them have access than I realize.
Although I wasn't a graduate student so maybe way more of them have access than I realize.
According to the Hipergator Website:
https://www.rc.ufl.edu/access/account-request/
To be eligible for an account on HiPerGator, you must be on an education or research project that has as one of its Principal or Co-Principal investigators a faculty or staff member at the University of Florida.
I'm also a UF alumni (undergrad & grad EE), and IIRC back in my day high performance computing was a grad class. I never took it, but it wouldn't surprise me if for the grad class there would be labs on the super computer where students can get limited access.
To be eligible for an account on HiPerGator, you must be on an education or research project that has as one of its Principal or Co-Principal investigators a faculty or staff member at the University of Florida.
I'm also a UF alumni (undergrad & grad EE), and IIRC back in my day high performance computing was a grad class. I never took it, but it wouldn't surprise me if for the grad class there would be labs on the super computer where students can get limited access.
Do you know how the CS program is doing these days? I heard it was on the ropes a few years ago but this looks like they are investing in it again?
[deleted]
[deleted]
[deleted]
Great name for the computer: “HiPerGator“
That's an awesome name. Just like hipermercado or KluckKluckBell.
Just baffling that they were on the verge of closing their CS department 8 years ago.
https://www.forbes.com/sites/stevensalzberg/2012/04/22/unive...
https://www.forbes.com/sites/stevensalzberg/2012/04/22/unive...
I think this was a bit of a clickbait headline, and the real story was bit more nuanced. They were trying to do some reorgs (kind of like merging/splittling different CS, ECE, Information Systems etc. depts.).
[deleted]
Does anyone know why everyone is still buying Nvidia instead of custom AI accelerators from other vendors? For example, on paper the new Graphcore machines look like an easy win, or at least a risk worth taking. (I see this particular supercomputer was funded by Nvidia but my question is about the general trend).
Risk mitigation, maintenance, resell value (?), support, reliability. The custom AI accelerators you mentioned, how long have those been in business for? How many units have they moved? How many generations of hardware have they produced? Will they still be around to replace or upgrade units in 5-10 years? How flexible are they in their workload?
That's a lot of factors to keep in mind when you're spending millions on a supercomputer. I'd go for an established hardware provider as well. I'm not knocking the custom AI chips, but I wouldn't try and max out my budget with those - keep them for smaller applications for now.
That's a lot of factors to keep in mind when you're spending millions on a supercomputer. I'd go for an established hardware provider as well. I'm not knocking the custom AI chips, but I wouldn't try and max out my budget with those - keep them for smaller applications for now.
Your supercomputer won’t last more than a couple years, 5 at max. Then it will simply be unjustifiably expensive to run when newer, more power efficient hardware becomes available.
What matters is the software platform and compatibility: you don’t want to retool and port all your existing programs. That’s where having a platform - CUDA - becomes a deal-maker.
What I don’t understand is how consumers - big, institutional consumers- don’t insist on viable vendor neutral platforms like OpenCL. Obviously they’re the only ones that have an interest in cross-vendor compatibility.
What matters is the software platform and compatibility: you don’t want to retool and port all your existing programs. That’s where having a platform - CUDA - becomes a deal-maker.
What I don’t understand is how consumers - big, institutional consumers- don’t insist on viable vendor neutral platforms like OpenCL. Obviously they’re the only ones that have an interest in cross-vendor compatibility.
Not sure why you're downvoted, HPC systems don't last more than ~6 years before they are decommissioned because they've become too power-hungry to justify their continued use. I just checked some of the systems I'm familiar with and they were all decommissioned after around 6 years of use (e.g. SuperMUC, 2012-2018; two smaller cluster at the university where I work: 08/2012-04/2017, 01/2014-03/2020).
I also attended a presentation at IPDPS 2018 by someone from one of the big US national labs where they talked about how absolutely huge of an undertaking it was to port their codes to make efficient use of GPUs. You don't just re-do all of that if the payoff isn't enormous.
I also attended a presentation at IPDPS 2018 by someone from one of the big US national labs where they talked about how absolutely huge of an undertaking it was to port their codes to make efficient use of GPUs. You don't just re-do all of that if the payoff isn't enormous.
Out of curiosity, do you know what happens to the decomissioned hardware? is it scrapped for useful parts like perhaps the PSUs? or is gold and other metals extracted from the chips and the metal sold for recycling?
Is there anything useful someone could do with it or it's just too much of a problem to set it up and repurpose it?
Is there anything useful someone could do with it or it's just too much of a problem to set it up and repurpose it?
> Out of curiosity, do you know what happens to the decomissioned hardware? is it scrapped for useful parts like perhaps the PSUs? or is gold and other metals extracted from the chips and the metal sold for recycling?
I cannot state with 100% clarity what happens after the systems are acquired by surplus vendors, but I can say that we have received certificates of what was "recycled". Once the surplus vendors take possession of the hardware, it's theirs and they can do what they want with it. In fact, we recently had to purchase EOL'ed, refurbished Infiniband switches to continue to support an interconnect fabric still in use (~2016). Interestingly enough, some of the switches still had "core" and "edge" labels on them.
> Is there anything useful someone could do with it or it's just too much of a problem to set it up and repurpose it?
In my opinion, it really depends on the node. If the chassis supports hot swappable & redundant hardware (HDD, PSU, etc.), then we'll typically cannibalize several chassis to create an administrative and/or infrastructure node. Case in point, a large portion of our older 12 core nodes have been put aside to serve as administrative nodes, all fully redundant (RAID1 HDD's, dual PSU's, ECC memory, etc.). Since we have a stack of these, we're fairly confident that these will serve us for the next few years, worry free, given the abundance of parts lying around.
I cannot state with 100% clarity what happens after the systems are acquired by surplus vendors, but I can say that we have received certificates of what was "recycled". Once the surplus vendors take possession of the hardware, it's theirs and they can do what they want with it. In fact, we recently had to purchase EOL'ed, refurbished Infiniband switches to continue to support an interconnect fabric still in use (~2016). Interestingly enough, some of the switches still had "core" and "edge" labels on them.
> Is there anything useful someone could do with it or it's just too much of a problem to set it up and repurpose it?
In my opinion, it really depends on the node. If the chassis supports hot swappable & redundant hardware (HDD, PSU, etc.), then we'll typically cannibalize several chassis to create an administrative and/or infrastructure node. Case in point, a large portion of our older 12 core nodes have been put aside to serve as administrative nodes, all fully redundant (RAID1 HDD's, dual PSU's, ECC memory, etc.). Since we have a stack of these, we're fairly confident that these will serve us for the next few years, worry free, given the abundance of parts lying around.
Interesting! thanks for the insight.
CUDA is valuable and seen as a positive, even if it's vendor locked. It's battle tested. Engineers are familiar with it. Support is provided because it's all under one roof. Performance targets, roadmap timeframes, etc can be assured.
The same reason a big institutional consumer will happily pay Microsoft for Windows Enterprise, instead of demanding Microsoft rebuild Windows using open source technologies.
A supercomputer or DC cluster has a fixed lifespan and amortization period, if you can get something in a contract, with everything you want for a price you are happy with, why would you introduce risk to it by demanding open standards for no benefit?
The same reason a big institutional consumer will happily pay Microsoft for Windows Enterprise, instead of demanding Microsoft rebuild Windows using open source technologies.
A supercomputer or DC cluster has a fixed lifespan and amortization period, if you can get something in a contract, with everything you want for a price you are happy with, why would you introduce risk to it by demanding open standards for no benefit?
I’m not saying CUDA is not valuable. Heck, proprietary has a ton of value... see how Intel owes much of its position to its compiler (the FORTRAN one in particular) suite.
Of course you don’t get to dominate a field to such an extent without some redeeming positive quality.
What I’m arguing - or more appropriately, what I’m wondering - is why on Earth aren’t consumer consortiums not insisting on open-standard platforms.
I get how you want to eek every drop of performance out of your tools without making it your only mission (that would be the research you’re using the tool for), but tying yourself to a single proprietary vendor defeats that purpose imho
Of course you don’t get to dominate a field to such an extent without some redeeming positive quality.
What I’m arguing - or more appropriately, what I’m wondering - is why on Earth aren’t consumer consortiums not insisting on open-standard platforms.
I get how you want to eek every drop of performance out of your tools without making it your only mission (that would be the research you’re using the tool for), but tying yourself to a single proprietary vendor defeats that purpose imho
TPUs are perfectly positioned to capture this market. There are an endless number of reasons why, but to keep it short: Wanna see a magic trick? https://twitter.com/theshawwn/status/1286426454171975680
GPT-2 117M training at 1 million tokens/sec.
Now, I don't have experience with DGX clusters, so I'm not going to make a firm statement. What I will say is that I, as an outsider, managed to achieve a performance level that is ~unheard of for GPT-2 training. And you can too; TPUs are pervasive.
A TPUv2-512 isn't even as far as the gas pedal goes, either. v3-512 can train all of imagenet to 75.9% accuracy in 4 minutes: https://twitter.com/theshawwn/status/1223395022814339073
v3-1024 can do it in 2 minutes: https://twitter.com/theshawwn/status/1234654848114520065
I once attached a debugger to a training run during startup, after the infeed loop began (meaning it was feeding inputs to the TPU, but no training was happening yet; it was "winding up") and was shocked to discover that when I hit c to continue, it trained on all of imagenet in like 54 seconds. That blows the lid off of every perf result here (under "image classification"): https://mlperf.org/training-results-0-6
(It's not a fair comparison, but it was quite astonishing to see the raw horsepower in action.)
So, nVidia has some catching up to do. And I don't know if they'll be able to. The TPU ecosystem may be clunky at the moment, but boy is it effective. Your options are to invest your time in this ecosystem, which will likely be around in ten years, or in DGX-cluster-type knowledge, which ... might be less pervasive in 10 years.
The distinguishing feature of a TPU is that it has a CPU on board. In fact, it has a CPU with 300GB of memory for every 8 cores. Friggin' love these things.
GPT-2 117M training at 1 million tokens/sec.
Now, I don't have experience with DGX clusters, so I'm not going to make a firm statement. What I will say is that I, as an outsider, managed to achieve a performance level that is ~unheard of for GPT-2 training. And you can too; TPUs are pervasive.
A TPUv2-512 isn't even as far as the gas pedal goes, either. v3-512 can train all of imagenet to 75.9% accuracy in 4 minutes: https://twitter.com/theshawwn/status/1223395022814339073
v3-1024 can do it in 2 minutes: https://twitter.com/theshawwn/status/1234654848114520065
I once attached a debugger to a training run during startup, after the infeed loop began (meaning it was feeding inputs to the TPU, but no training was happening yet; it was "winding up") and was shocked to discover that when I hit c to continue, it trained on all of imagenet in like 54 seconds. That blows the lid off of every perf result here (under "image classification"): https://mlperf.org/training-results-0-6
(It's not a fair comparison, but it was quite astonishing to see the raw horsepower in action.)
So, nVidia has some catching up to do. And I don't know if they'll be able to. The TPU ecosystem may be clunky at the moment, but boy is it effective. Your options are to invest your time in this ecosystem, which will likely be around in ten years, or in DGX-cluster-type knowledge, which ... might be less pervasive in 10 years.
The distinguishing feature of a TPU is that it has a CPU on board. In fact, it has a CPU with 300GB of memory for every 8 cores. Friggin' love these things.
Last time I worked on TPUs, a lot of very pervasive LA operations (e.g. Cholesky decomposition) were unoptimized and slow compared to the NN-style operations. I’m sure that can be fixed, but for the time being, it seems inappropriate for anything beyond the obvious NN operations.
If "this market" means supercomputers (as 'Cthulhu_ was talking about), then TPUs are certainly not perfectly positioned to capture this market, since they are only capable of doing bfloat16 arithmetic with high performance, and AFAIK incapable of fp64 at all. That means a HPL score of zero.
TPUs will probably dominate AI training, but not so much supercomputing.
TPUs will probably dominate AI training, but not so much supercomputing.
1 milion per second of anything isn't very fast. Unless you're running that on a mobile phone.
Many people use HPC for things that are not AI/ML, such as CFD. Those applications are typically written using MPI/CUDA
Do those custom AI accelerators work with PyTorch and Tensorflow? The graphcore website says that support with PyTorch aten is available in 2020, but it's not -- and Aten is inference only, no autograd, and no extensions. Without those, we can't train on it.
Graphcore claims they do. I don't know true that is though.
I reimplemented one benchmark using TF and the speed up wasn’t the 10x or so hey we’re claiming, but more like 1x..
The only non-NVidia device you can actually use that has broad support outside basic models in software is Google's TPUs.
Graphcore looks great! But no one outside Graphcore has used them so who knows. Intel's Nervana looked great on paper, right up until they dumped it.
There are some interesting accelerator options around for inference. But for training it's NVidia for almost everything, and TPUs as a good option is a few cases. But you can't buy TPUs (except for the inference-only Coral board), and universities like to own hardware.
Graphcore looks great! But no one outside Graphcore has used them so who knows. Intel's Nervana looked great on paper, right up until they dumped it.
There are some interesting accelerator options around for inference. But for training it's NVidia for almost everything, and TPUs as a good option is a few cases. But you can't buy TPUs (except for the inference-only Coral board), and universities like to own hardware.
> But no one outside Graphcore has used them so who knows.
This part really confuses me. I'd love to try out their hardware, but the only cloud offering they had last time I checked made you rent a whole month's worth (for many thousands of dollars).
It seems like getting it in the hands of devs should be a top priority.
This part really confuses me. I'd love to try out their hardware, but the only cloud offering they had last time I checked made you rent a whole month's worth (for many thousands of dollars).
It seems like getting it in the hands of devs should be a top priority.
From my very limited understanding the Graphcore machines outperform CUDA only significantly in inference, in training the improvements might not be sufficient to switch technology.
There is a trend towards larger models in the state of the art of deep learning research. The total cost of the on-chip memory on the NVidia GPU is still a good value proposition compared to current custom deep learning accelerators. This cost to get to large on-chip memory, combined with the flexibility of CUDA for other types of scientific applications, makes it harder for such accelerators to compete with Nvidia in such large academic research clusters. I hope that this situation changes over the next couple of years.
Is this really true? At least for inference I've seen data showing the trend is rather for smaller models that exploit sparsity.
This is a chart from before GPT-3: https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1...
Yes, in NLP big models rule and they are getting bigger.
It seems like supercomputers are still generally many times more powerful than what AI researchers at top universities or orgs are using.
Has any well known AI research been done on supercomputers?
It seems like the case that literally just throwing money at the problem is a solid idea nowadays.
Has any well known AI research been done on supercomputers?
It seems like the case that literally just throwing money at the problem is a solid idea nowadays.
GPT-3 was another buckets worth of evidence in favor of the scaling hypothesis. Performance kept improving (and cost to train kept increasing) as more parameters were added. Even with 175 billion parameters, the performance had not yet plateaued. One take-away is that throwing a lot of compute at the problem helps tremendously :).
You can read more about GPT-3 here: https://lambdalabs.com/blog/gpt-3/
You can read more about GPT-3 here: https://lambdalabs.com/blog/gpt-3/
Are you alluding to "The Bitter Lesson" [1] by Rich Sutton [2]?
[1] http://incompleteideas.net/IncIdeas/BitterLesson.html
[2] http://incompleteideas.net/
[1] http://incompleteideas.net/IncIdeas/BitterLesson.html
[2] http://incompleteideas.net/
Cmon. Just look at requirements to train GPT-3 for example... https://lambdalabs.com/blog/demystifying-gpt-3/
I guess I had it in my head that they used something like 500 V100's -- which is still a far cry from a big supercomputer these days -- but they haven't published what hardware they used and for how long...
[deleted]
> It will also benefit from Nvidia’s suite of AI application frameworks
Sounds like lock-in.
Sounds like lock-in.
Is it lock-in when it's the only thing or the best thing on the market?
It is lock-in when it makes it very hard to move to anything else even if it's better. That's Nvidia's point. They don't play fair. They give this "for free" with hard strings attached to Nvidia.
How bad is this for the environment?
Seems like it has about 700/5 = 140 GPUs. And they are about 0.4kW each. Let's say it is 50% utilized for a year. That is 140 * 0.4 * 0.5 * 365 * 24 = 245280 kWh.
Let's say electricity generation creates 0.5 kg CO2e per kWh. So 122640 kg CO2e per year.
For comparison, driving a cars creates about 0.2kg CO2 per km. Or 0.32 kg per mile. So about the same as driving 122640 / 0.32 = 383250 miles per year. Call it 38 cars.
Edit: Corrected the above CO2 per km figure to fix a factor of 100 error.
Obviously there are a million things wrong with this analysis. For example, we don't know if they're going to use the machine to do climate modelling that will lead to a headline in the media that causes the green party to be elected ;-)
Let's say electricity generation creates 0.5 kg CO2e per kWh. So 122640 kg CO2e per year.
For comparison, driving a cars creates about 0.2kg CO2 per km. Or 0.32 kg per mile. So about the same as driving 122640 / 0.32 = 383250 miles per year. Call it 38 cars.
Edit: Corrected the above CO2 per km figure to fix a factor of 100 error.
Obviously there are a million things wrong with this analysis. For example, we don't know if they're going to use the machine to do climate modelling that will lead to a headline in the media that causes the green party to be elected ;-)
HPC system utilization is typically upwards of 90%, these things are not idle. There's almost always a queue of jobs waiting to run, at least in my experience.
Definitely, also it makes you hate those who hog up the resources. A good scheduler will take into account the past usage however.
Cars generate closer to 122g of CO2 per km [1]. 122,000kg would be ~1 million km or ~50 cars.
1 - https://ec.europa.eu/clima/policies/transport/vehicles/cars_...
1 - https://ec.europa.eu/clima/policies/transport/vehicles/cars_...
Damn. I was confused by the fuel consumption being in per 100 km units. I've edited my post. Thanks.
Yeah who isn't hahaha
It has 140 nodes, each with 8 GPU's. Add in extra infra like storage, networking etc. and you're looking at around 700 kW at 100% utilization.
You're right. The article is wrong. It said:
> A single A100 GPU’s 54 billion transistors can execute 5 petaflops of performance, according to Nvidia.
It is a single system of 8 GPUs that can do 5 petaflops.
I can't edit my post anymore. Multiply my numbers by 8, or a bit more if you want to take into account higher utilization and the other overheads you mention. So ~500 cars. This is my final offer.
> A single A100 GPU’s 54 billion transistors can execute 5 petaflops of performance, according to Nvidia.
It is a single system of 8 GPUs that can do 5 petaflops.
I can't edit my post anymore. Multiply my numbers by 8, or a bit more if you want to take into account higher utilization and the other overheads you mention. So ~500 cars. This is my final offer.
FWIW, UF is planning to be carbon neutral by 2025. I don't know how the supercomputer factors into the equation, but I am sure someone has it in mind.
https://sustainable.ufl.edu/campus-initiatives/neutral-uf-co...
https://sustainable.ufl.edu/campus-initiatives/neutral-uf-co...
[deleted]
Maybe they can use it to simulate load and stress on pedestrian bridges for some of the other Florida Universities:
https://en.wikipedia.org/wiki/Florida_International_Universi...
https://en.wikipedia.org/wiki/Florida_International_Universi...
This being NVIDIA, University of Florida would be wise to hire an independent company to benchmark their "700-petaflop".
They should build proper Linux driver first without enormous signed blobs. One of the worst companies for FOSS community.