CUDA Ontology(jamesakl.com)
jamesakl.com
CUDA Ontology
https://jamesakl.com/posts/cuda-ontology/
43 comments
> nvcc from the CUDA toolkit has a compatibility range with the underlying host compilers like gcc. If you install a newer CUDA toolkit on an older machine, likely you'll need to upgrade your compiler toolchain as well, and fix the paths.
Conversely, nvcc often stops working with major upgrades of gcc/clang. Fun times, indeed.
This is why a lot of people just use NVIDIA's containers even for local solo dev. It's a hassle to set up initially (docker/podman hell) but all the tools are there and they work fine.
Conversely, nvcc often stops working with major upgrades of gcc/clang. Fun times, indeed.
This is why a lot of people just use NVIDIA's containers even for local solo dev. It's a hassle to set up initially (docker/podman hell) but all the tools are there and they work fine.
> This is why a lot of people just use NVIDIA's containers even for local solo dev. It's a hassle to set up initially (docker/podman hell) but all the tools are there and they work fine.
Yeah, which I feel like is fine for one project, or one-offs, but once you've accumulated projects, having individual 30GB images for each of them quickly adds up.
I found that most of my issues went away as I started migrating everything to `ux` for the python stuff, and nix for everything system related. Now I can finally go back to a 1 year old ML project, and be sure it'll run like before, and projects share a bit more data.
Yeah, which I feel like is fine for one project, or one-offs, but once you've accumulated projects, having individual 30GB images for each of them quickly adds up.
I found that most of my issues went away as I started migrating everything to `ux` for the python stuff, and nix for everything system related. Now I can finally go back to a 1 year old ML project, and be sure it'll run like before, and projects share a bit more data.
What trouble have you had specifically? On both Win and Linux, installing the CUDA toolkit (e.g. v13) just works for me. My use case is compiling kernels (or cuFFT FFI) using nvcc for FFI in rust programs and libs.
Yep, right now nvidia libs are broken with clang-21 and recent glibc due to stuff like rsqrt() having throw() in the declaration and not in the definition
> Also, code that compiles with older CUDA toolkit versions may not compile with newer CUDA toolkit versions. Newer hardware may require a CUDA toolkit version that is newer than what the project maintainer intended.
This is the part I find confusing, especially as NVIDIA doesn't make it easy to find and download the old toolkits. Is this effectively saying that just choosing the right --arch and --code flags isn't enough to support older versions? But that as it statically links in the runtime library (by default) that newer toolkits may produce code that just won't run on older drivers? In other words, is it true that to support old hardware you need to download and use old CUDA Toolkits, regardless of nvcc flags? (And to support newer hardware you may need to compile with newer toolkits).
That's how I read it, which seems unfortunate.
This is the part I find confusing, especially as NVIDIA doesn't make it easy to find and download the old toolkits. Is this effectively saying that just choosing the right --arch and --code flags isn't enough to support older versions? But that as it statically links in the runtime library (by default) that newer toolkits may produce code that just won't run on older drivers? In other words, is it true that to support old hardware you need to download and use old CUDA Toolkits, regardless of nvcc flags? (And to support newer hardware you may need to compile with newer toolkits).
That's how I read it, which seems unfortunate.
Yes, this is the actual lived reality. Thank you for outlining it so well.
Sounds like most of these problems come from using Python.
You imply these problems would go away (or wouldn't be replaced by new ones) with another language.
Removing layers usually improves stability.
Wondering why a $4T company can't afford a smart installation assistant that can auto-detect problems and apply fixes as needed. I wasted too many days chasing driver and torch versions. It's probably the worst part of working in ML. Combine this with Python's horrible package management and you got a perfect combo - like the cough and the stitch.
I'm wondering how a $4T company got away with shipping the absolute state of the toolchain to begin with. They have total and complete sovereignty on everything on the outside of the OS and PCIe boundaries with a bottomless pool of top class labor. There's no reason it has to be cruftier or more fragile than any other low latency networked computation... and yet here we are. AMD isn't any better. I'm almost interested to see if Intel has done any better with L0, but I highly suspect it suffers from the exact same ecosystem hell problems that plague the other two.
The idea that getting a PCIe FPGA board to crunch numbers is less headache prone than a GPU is laughable, but that's the absurd reality we live in.
The idea that getting a PCIe FPGA board to crunch numbers is less headache prone than a GPU is laughable, but that's the absurd reality we live in.
They provide containers to cater to those needs: https://catalog.ngc.nvidia.com/search
After being once again frustrated by the CUDA installation experience, I thought that I should give those containers a try. Unfortunately, my computer did not boot anymore after following the installation instructions for the NVIDIA container toolkit as outlined on the NVIDIA website. Reinstalling everything and following the instructions from some random blog post made it work, but I then found that the container with the CUDA version that I needed had been deprecated.
There were other problems, such as the research cluster of my university not having Docker, but that is a different issue.
There were other problems, such as the research cluster of my university not having Docker, but that is a different issue.
Containers don't include drivers which is the primary reason for issues.
Containers afair rely on the exact driver version matching between the host system and the container itself.
We were on AWS when we used this so setting up seemed easy enough - AWS gave you the driver, and a matching docker image was easy enough to find.
We were on AWS when we used this so setting up seemed easy enough - AWS gave you the driver, and a matching docker image was easy enough to find.
That's not the case, CUDA containers user space does not have to match the host drivers CUDA capability. The container needs to be the same major version or lower. So a system with a CUDA 13 capable driver should be able to run all previous versions.
For some versions there's even sometimes compat layers built into the container to allow forward version compatibility.
For some versions there's even sometimes compat layers built into the container to allow forward version compatibility.
Just have claude code fix it
> CUDA Runtime: The runtime library (libcudart) that applications link against.
That library is actually a rather poor idea. If you're writing a CUDA application, I strongly recommend avoiding the "runtime API". It provides partial access to the actual CUDA driver and its API, which is 'simpler' in the sense that you don't explicitly create "contexts", but:
* It hides or limits a lot of the functionality.
* Its actual behavior vis-a-vis contexts is not at all simple and is likely to make your life more difficult down the road.
* It's not some clean interface that's much more convenient to use.
So, either go with the driver, or consider my CUDA API wrappers library [1], which _does_ offer a clean, unified, modern (well, C++11'ish) RAII/CADRe interface. And it covers much more than the runtime API, to boot: JIT compilation of CUDA (nvrtc) and PTX (nvptx_compiler), profiling (nvtx), etc.
> Driver API ... provides direct access to GPU functionality.
Well, I wouldn't go that far, it's not that direct. Let's call it: "Less indirect"...
[1] : https://github.com/eyalroz/cuda-api-wrappers/
That library is actually a rather poor idea. If you're writing a CUDA application, I strongly recommend avoiding the "runtime API". It provides partial access to the actual CUDA driver and its API, which is 'simpler' in the sense that you don't explicitly create "contexts", but:
* It hides or limits a lot of the functionality.
* Its actual behavior vis-a-vis contexts is not at all simple and is likely to make your life more difficult down the road.
* It's not some clean interface that's much more convenient to use.
So, either go with the driver, or consider my CUDA API wrappers library [1], which _does_ offer a clean, unified, modern (well, C++11'ish) RAII/CADRe interface. And it covers much more than the runtime API, to boot: JIT compilation of CUDA (nvrtc) and PTX (nvptx_compiler), profiling (nvtx), etc.
> Driver API ... provides direct access to GPU functionality.
Well, I wouldn't go that far, it's not that direct. Let's call it: "Less indirect"...
[1] : https://github.com/eyalroz/cuda-api-wrappers/
If you do this, you forego both backwards and forwards compatibility. You must follow the driver release cadence exactly, and rebuild all of your code for every driver you want to support when a new release happens, or you risk subtle breakage. NVIDIA guarantees nothing in terms of breakage for you.
Probably the worst part of this: for the most part, in practice, it will work just fine. Until it doesn’t. You will have lots of fun debugging subtle bugs in a closed-source black box, which reproduces only against certain driver API header versions, which potentially does not match the version of the actual driver API DSO you’ve dlopened, and which only produces problems when mixed with certain Linux kernel versions.
(I have the exact opposite opinion; people reach too eagerly for the driver API when they don’t need it. Almost everything that can be done with the driver api can be done with the runtime API. If you absolutely must use the driver API, which I doubt, you should at least resolve the function pointers through cudaGetDriverEntrypointByVersion.)
Probably the worst part of this: for the most part, in practice, it will work just fine. Until it doesn’t. You will have lots of fun debugging subtle bugs in a closed-source black box, which reproduces only against certain driver API header versions, which potentially does not match the version of the actual driver API DSO you’ve dlopened, and which only produces problems when mixed with certain Linux kernel versions.
(I have the exact opposite opinion; people reach too eagerly for the driver API when they don’t need it. Almost everything that can be done with the driver api can be done with the runtime API. If you absolutely must use the driver API, which I doubt, you should at least resolve the function pointers through cudaGetDriverEntrypointByVersion.)
Disagree, because:
* The Runtime API is also a black box - it's just differently shaped.
* CUDA runtime APIs are also incompatible with CUDA drivers which are significantly older. Although TBH I have not checked that compatibility range recently.
* C++ is a compiled language. So, yes, in some cases, you need to recompile. But - less than your might think. Specifically, the driver API headers use macros to direct your API function names to versioned names. For example:
* Finally, if you want something more stable, more portable, that doesn't change frequently - OpenCL can also be considered rather than CUDA.
* The Runtime API is also a black box - it's just differently shaped.
* CUDA runtime APIs are also incompatible with CUDA drivers which are significantly older. Although TBH I have not checked that compatibility range recently.
* C++ is a compiled language. So, yes, in some cases, you need to recompile. But - less than your might think. Specifically, the driver API headers use macros to direct your API function names to versioned names. For example:
#define cuStreamGetCaptureInfo __CUDA_API_PTSZ(cuStreamGetCaptureInfo_v3)
and this versioned function will typically be available also when the signature changes to v4 (in this example, it seems two versions backwards are available in CUDA 13.0).
* Finally, if you want something more stable, more portable, that doesn't change frequently - OpenCL can also be considered rather than CUDA.
This article has good info, but is the overloading premise slightly contrived? Maybe I don’t talk to enough CUDA beginners. I work with CUDA a lot but I’m not exactly a CUDA expert, and from my perspective, in practice there are default assumptions one can safely make for the base terms, and people do qualify the alternatives almost always. For example, if someone says “CUDA version”, they always mean the toolkit, and never mean compute capability, runtime, or language. The term “driver” when used without qualification always means the display driver, and never means the driver API, there really is no overload there.
I actually find it is pretty easy to get confused between the different kinds of versions. For example:
"The CUDA "driver version" looks like the CUDA runtime version - so what's the difference?" https://stackoverflow.com/q/40589814/1593077
or consider the version you get when you run nvidia-smi, versus the version you get when you run nvcc --version. Those are very different numbers...
The compatibility between different versions of the driver and the toolkit is also a cause for some headaches in my experience.
"The CUDA "driver version" looks like the CUDA runtime version - so what's the difference?" https://stackoverflow.com/q/40589814/1593077
or consider the version you get when you run nvidia-smi, versus the version you get when you run nvcc --version. Those are very different numbers...
The compatibility between different versions of the driver and the toolkit is also a cause for some headaches in my experience.
Oh yeah, knowing the difference between the runtime API and driver API is definitely an issue, and there is common confusion around that. But that’s not an overloaded word problem, right? I wasn’t trying to say there’s no confusion, and I do think understanding the terms in the article is super helpful. To your point, I think there’s justification for needing a codex like this article has without framing it as an overloaded terminology problem.
I suspect you're colored by your experience, despite your modesty about it. To you or I, "CUDA version" probably means something like 'v13' or w/e of the "CUDA toolkit", which you know means the user running your code needs an "nvidia driver" = "580" or higher.
I wouldn't have been able to tell you this a few months ago, and it was confusing! Machine that compiles vs machine that runs, CUDA toolkit which includes both vs nvidia driver which just includes one part of it etc... The article explicitly describes this.
I wouldn't have been able to tell you this a few months ago, and it was confusing! Machine that compiles vs machine that runs, CUDA toolkit which includes both vs nvidia driver which just includes one part of it etc... The article explicitly describes this.
For reference: CUDA means "Compute Unified Device Architecture".
I wish GPU vendors would stick to a standard terminology, at least for common parts. It's really confusing having to deal with warps vs wavefronts vs simd groups, thread block vs workgroup, streaming multiprocessor vs compute unit vs execution unit, etc...
Great overview, with lots of effort place into it.
However, it misses the polyglot part (Fortran, Python GPU JIT, all the backends that support PTX), the library ecosystem (writing CUDA kernels should be the exception not the rule), the graphical debugging tools and IDE integration.
However, it misses the polyglot part (Fortran, Python GPU JIT, all the backends that support PTX), the library ecosystem (writing CUDA kernels should be the exception not the rule), the graphical debugging tools and IDE integration.
> This article provides a rigorous ontology of CUDA components: a systematic description of what exists in the CUDA ecosystem, how components relate to each other, their versioning semantics, compatibility rules, and failure modes.
That's the first instance in my life when somebody coherently described what the word 'ontology' means. I'm sure this explanation is wrong, but still...
That's the first instance in my life when somebody coherently described what the word 'ontology' means. I'm sure this explanation is wrong, but still...
That is a great reference, explains a lot of small inaccuracies between various tutorials when you're trying to debug some of these issues. Saved and printed, thanks a lot!
Interesting, does this approach change with out-of-order cores? In fact maybe I misunderstand lol
What is the cheapest CUDA-enabled VM providers one can use to learn CUDA?
Lightning.ai
This is a classic lesson. You can write almost the same article for Java: language vs bytecode vs JVM vs JDK vs libs ...
thanks for the kernel nomenclatures
this is fantastic
Great explanation!
It should probably also add that everything CUDA is owned by NVIDIA, and "CUDA" itself is a registered trademark. The official way to refer to it is that the first time you spell it out as "NVIDIA® CUDA®" and then subsequently refer to just CUDA.
It should probably also add that everything CUDA is owned by NVIDIA, and "CUDA" itself is a registered trademark. The official way to refer to it is that the first time you spell it out as "NVIDIA® CUDA®" and then subsequently refer to just CUDA.
Why should the author use the registered trademark symbol?
I am not a layer (IANAL), but here is what Gemini 3 Pro says: "You generally do not need to use the trademark symbol for CUDA in a blog post, unless you have a specific commercial relationship with NVIDIA."
Now direct from actual sources... From [1]
> Intended users of this Brand Guideline are members of the NVIDIA Partner Network (NPN), including original equipment manufacturers (OEMs), solution advisors, cloud partners, solution providers, distributors, solutions integrators, and service delivery partners.
From [2]:
> Always include the correct trademark (™ vs ®) by referring to the content documents provided or using the list of common NVIDIA products and technologies. After the first mention of the NVIDIA product or technology, which includes the appropriate trademarks, the trademark does not need to be included in future mentions within the same document, article, etc.
> CUDA®
[1]: https://brand.nvidia.com/d/wGtgoY2mtYYM/nvidia-partner-netwo...
[2]: https://brand.nvidia.com/d/wGtgoY2mtYYM/nvidia-partner-netwo...
Now direct from actual sources... From [1]
> Intended users of this Brand Guideline are members of the NVIDIA Partner Network (NPN), including original equipment manufacturers (OEMs), solution advisors, cloud partners, solution providers, distributors, solutions integrators, and service delivery partners.
From [2]:
> Always include the correct trademark (™ vs ®) by referring to the content documents provided or using the list of common NVIDIA products and technologies. After the first mention of the NVIDIA product or technology, which includes the appropriate trademarks, the trademark does not need to be included in future mentions within the same document, article, etc.
> CUDA®
[1]: https://brand.nvidia.com/d/wGtgoY2mtYYM/nvidia-partner-netwo...
[2]: https://brand.nvidia.com/d/wGtgoY2mtYYM/nvidia-partner-netwo...
Ah, my point (which I failed to make, obviously) was not regarding the post.
It was more like "the whole CUDA stuff is by a single company".
It was more like "the whole CUDA stuff is by a single company".
nvcc from the CUDA toolkit has a compatibility range with the underlying host compilers like gcc. If you install a newer CUDA toolkit on an older machine, likely you'll need to upgrade your compiler toolchain as well, and fix the paths.
While orchestration in many (research) projects happens from Python, some depend on building CUDA extensions. An innocently looking Python project may not ship the compiled kernels and may require a CUDA toolkit to work correctly. Some package management solutions provide the ability to install CUDA toolkits (conda/mamba, pixi), the pure-Python ones do not (pip, uv). This leaves you to match the correct CUDA toolkit to your Python environment for a project. conda specifically provides different channels (default/nvidia/pytorch/conda-forge), from conda 4.6 defaulting to a strict channel priority, meaning "if a name exists in a higher-priority channel, lower ones aren't considered". The default strict priority can make your requirements unsatisfiable, even though there would be a version of each required package in the collection of channels. uv is neat and fast and awesome, but leaves you alone in dealing with the CUDA toolkit.
Also, code that compiles with older CUDA toolkit versions may not compile with newer CUDA toolkit versions. Newer hardware may require a CUDA toolkit version that is newer than what the project maintainer intended. PyTorch ships with a specific CUDA runtime version. If you have additional code in your project that also is using CUDA extensions, you need to match the CUDA runtime version of your installed PyTorch for it to work. Trying to bring up a project from a couple of years ago to run on latest hardware may thus blow up on you on multiple fronts.