Analyzing Modern Nvidia GPU Cores(arxiv.org)
arxiv.org
Analyzing Modern Nvidia GPU Cores
https://arxiv.org/abs/2503.20481
40 comments
In a sense, GPUs are only great at matrix-matrix multiplication. For anything else you would only get 7% of the FLOPs/s compared to it(989 vs 67 TFLOP/s for H100)[1].
[1]: https://www.nvidia.com/en-in/data-center/h100/
[1]: https://www.nvidia.com/en-in/data-center/h100/
Funnily, they're far from being optimal for GEMM ops (especially in terms of power consumption).
For GEMM you need to visit each row/vec n-times so theres a bunch of data-reuse going on, which isn't optimal for GPUs since you can't keep that all so close to your processing-units. And while the tensor-cores kinda implement this i think they don't quite scale up to a full sized systolic array, which is you would want for larger matrix multiplications.
Also just a simpler view: with GPUs most of their silicon is spent NOT tensor-core, so just from that you know its not optimal i guess.
Just referring to that FLOP/s number doesn't really mean much nowadays with tensor-cores and sparsity.
In my eyes the big win of GPUs are that not only are they pretty good at GEMMs but also really good at a lot of other easily parallelizable tasks PLUS they're comparatively easy to program ^^
For GEMM you need to visit each row/vec n-times so theres a bunch of data-reuse going on, which isn't optimal for GPUs since you can't keep that all so close to your processing-units. And while the tensor-cores kinda implement this i think they don't quite scale up to a full sized systolic array, which is you would want for larger matrix multiplications.
Also just a simpler view: with GPUs most of their silicon is spent NOT tensor-core, so just from that you know its not optimal i guess.
Just referring to that FLOP/s number doesn't really mean much nowadays with tensor-cores and sparsity.
In my eyes the big win of GPUs are that not only are they pretty good at GEMMs but also really good at a lot of other easily parallelizable tasks PLUS they're comparatively easy to program ^^
[deleted]
That link says "* With sparsity". For extremely sparse matrixes you can get more than 989 TFLOPS on CPU, if we're counting elided operations in TFLOPS.
I am counting FP16/BF16 without sparsity, which is used in majority of AI.
That change checks out then. They didn't see much need for FP16 outside of that so no longer run it at double FP32 rate outside of tensor cores (unless I'm mixing that up with AMD).
Other forms of sparsity are heavily used at training time now, like block compression in Deepseek.
Other forms of sparsity are heavily used at training time now, like block compression in Deepseek.
lol, I haven't thought about it like that, true. though of course, I mean compared to CPUs :P
I try and use tensor cores for non-obvious things every now and then. The most promising so far seems to be for linear arithmetic in Datalog, but that's just matrix-vector/gemv
I try and use tensor cores for non-obvious things every now and then. The most promising so far seems to be for linear arithmetic in Datalog, but that's just matrix-vector/gemv
Could you expand the Datalog example? I'm quite interested
This was just a brief moment of thought over a year ago, but I can try to summarize.
I was thinking about how to unify variables in certain simple Datalog settings. If we think of a clause as a vector of variables, then simple unifications can look like just a gather operation. A gather can be thought of as a matrix-vector multiplication, but that's not really useful (performance wise). But if those variables are also in a linear equation, then it becomes possibly-useful, e.g. for something like `P(x, y) :- E(x, 3x+4y)`
I didn't get that at all - to me this looks like a very smart investigation into instruction latency and the precise mechanics of out-of-order execution (no reference made to speculative or branch prediction, though?) without looking at what the instructions do in detail.
GPUs can certainly do bulk integer arithmetic but most use cases prefer FP. Maybe for DSP fixed-point is ideal.
GPUs can certainly do bulk integer arithmetic but most use cases prefer FP. Maybe for DSP fixed-point is ideal.
From your comment I've learned that you never did GPU graphical programming :)
"uniform registers" exist for about 20 years now.
"uniform registers" exist for about 20 years now.
I think you are confusing uniform registers with the uniform keyword in RSL / GLSL / HLSL?
maybe some vendors have had an equivalent to uniform registers for 20 years, but per the articles’ references they are new in nvidia GPUs in turing (2018)
maybe some vendors have had an equivalent to uniform registers for 20 years, but per the articles’ references they are new in nvidia GPUs in turing (2018)
They are the same thing. The uniform keyword in shading languages is implemented using the uniform registers.
I don't know what Nvidia did in 2018, maybe they opened up access to the uniform registers to CUDA code.
I made Grok research this topic:
> In conclusion, research strongly suggests that the "uniform" keyword in GLSL is implemented in hardware using NVIDIA's "uniform registers," as evidenced by NVIDIA's own documentation on the Turing architecture and historical practices of mapping uniforms to constant registers. While explicit links can be limited due to proprietary details, the combination of technical presentations, community discussions, and historical context supports this connection. The uniform register file, with its capacity and usage in shader instructions, aligns with GLSL's uniform functionality, ensuring efficient data access during shader execution.
https://grok.com/share/c2hhcmQtMg%3D%3D_358362f3-21e2-4fe0-a...
I don't know what Nvidia did in 2018, maybe they opened up access to the uniform registers to CUDA code.
I made Grok research this topic:
> In conclusion, research strongly suggests that the "uniform" keyword in GLSL is implemented in hardware using NVIDIA's "uniform registers," as evidenced by NVIDIA's own documentation on the Turing architecture and historical practices of mapping uniforms to constant registers. While explicit links can be limited due to proprietary details, the combination of technical presentations, community discussions, and historical context supports this connection. The uniform register file, with its capacity and usage in shader instructions, aligns with GLSL's uniform functionality, ensuring efficient data access during shader execution.
https://grok.com/share/c2hhcmQtMg%3D%3D_358362f3-21e2-4fe0-a...
While you can use uniform registers to implement the uniform keyword from shading languages, the two are not the same. Uniform registers are not constants, and they are only uniform/shared across one warp. Nvidia architectures before Turing did not have uniform registers.
[deleted]
> NVIDIA RTX A6000
Unfortunately that's already behind the latest GPU by two generations. You'd have these after A6000: 6000 Ada, Pro 6000.
Unfortunately that's already behind the latest GPU by two generations. You'd have these after A6000: 6000 Ada, Pro 6000.
Still better than most folks have access to.
I bet I can do more CUDA with my lame GeForce MX 150 from 2017, than what most people can reach for to do ROCm, and that is how NVidia keeps being ahead.
I bet I can do more CUDA with my lame GeForce MX 150 from 2017, than what most people can reach for to do ROCm, and that is how NVidia keeps being ahead.
Yeah, kind of. I have an 6000 Ada and 5090 here.
On a laptop?
Because that is part of my point, that is a laptop GPU.
Because that is part of my point, that is a laptop GPU.
Oh no, there are high-end desktops. You're right, laptops completely different profiles for these things.
It's a major step forward compared to 2006.
A6000 was released in 2020: https://www.techpowerup.com/gpu-specs/rtx-a6000.c3686
A6000 was released in 2020: https://www.techpowerup.com/gpu-specs/rtx-a6000.c3686
Nvidia's Quadro naming scheme really is bad these days, isn't it?
I bet there are plenty of papers out there claiming to have used a RTX 6000 instead of a RTX 6000 Ada gen.
I bet there are plenty of papers out there claiming to have used a RTX 6000 instead of a RTX 6000 Ada gen.
The naming scheme is horrible, to be quite frank.
To understand this, consider these names in the order of release time: Quadro RTX 6000, RTX A6000, RTX 6000 Ada, RTX Pro 6000, RTX Pro 6000 Max-Q.
To understand this, consider these names in the order of release time: Quadro RTX 6000, RTX A6000, RTX 6000 Ada, RTX Pro 6000, RTX Pro 6000 Max-Q.
>Overall, we can conclude that GPUs are hardware-compiler codesign where the compiler guides the hardware in handling dependencies and introduces hints that can improve performance and energy.
New architectures rely on the compiler to handle register data dependencies, and controlling register file cache allocation policy.
New architectures rely on the compiler to handle register data dependencies, and controlling register file cache allocation policy.
This is an age-old idea, RISC compilers were supposed to do this too, the mythical "sufficiently smart compiler"
https://wiki.c2.com/?SufficientlySmartCompiler
https://wiki.c2.com/?SufficientlySmartCompiler
It's not so much about having a "sufficiently smart compiler" in the case of GPUs doing compiler assisted scheduling. It's about not having to implement that logic in hardware at all. The more smarts they push into the core hardware, the more silicon each core needs, the less cores you can fit, and more power you spend on figuring out what to run rather than crunching numbers.
Doing the work in the compiler may produce less optimal scheduling than what is theoretically possible, but with the number of "cores" in a GPU you would spend a lot of power doing it in hardware for each one.
Doing the work in the compiler may produce less optimal scheduling than what is theoretically possible, but with the number of "cores" in a GPU you would spend a lot of power doing it in hardware for each one.
RISC woks well with compilers (ARM, RISC-V), they don't require mythical compilers, just standard good ones.
You are probably thinking VLIW like Intels Itanium, and Transmeta. Those architectures required really smart compiler for scheduling and it was a bust.
Nvidia GPU's need smart compiler and it works because the task is limited to optimizing numerical pipelines that are 99% matrix multiplication, dot products. The data movement is more predicable. Compilers know how the data will be used and know how to schedule.
You are probably thinking VLIW like Intels Itanium, and Transmeta. Those architectures required really smart compiler for scheduling and it was a bust.
Nvidia GPU's need smart compiler and it works because the task is limited to optimizing numerical pipelines that are 99% matrix multiplication, dot products. The data movement is more predicable. Compilers know how the data will be used and know how to schedule.
MIPS used to not have interlocked pipeline stages. It was the compiler's job to work around that. MIPS -- and many other RISCs -- had a branch delay slot. It was the compiler's job to try to do something useful with that. RISCs stayed in-order for a long time -- it was the compiler's job to try do schedule the instructions in a way that compensated as well as possible for that.
GPUs rely on fairly smart compilers -- but they also hide latency (memory access) by switching hw threads (a bit like barrel processors of yore).
GPUs rely on fairly smart compilers -- but they also hide latency (memory access) by switching hw threads (a bit like barrel processors of yore).
The special sauce:
> "GPUs leverage hardware-compiler techniques where the compiler guides hardware during execution."
> "GPUs leverage hardware-compiler techniques where the compiler guides hardware during execution."
Haha honestly I always thought GPUs were mostly number crunchers, but there's way more under the hood than I realized. Wondering now if anyone really gets the full potential of these cores, or if we're all just scratching the surface most days?
There are very good performance tools for GPUs.
I don't think GPU utilization is a real bottleneck in most cases.
I don't think GPU utilization is a real bottleneck in most cases.
Yet DeepSeek managed to get huge improvement by optimizing GPU code.
Edit: learned a bunch, but the "uniform" registers and 64-bit (memory) performance are some easy standouts.