Nvidia gets 20% weighting, more investor demand; Apple demoted in major techfund(cnbc.com)
cnbc.com
Nvidia gets 20% weighting, more investor demand; Apple demoted in major techfund
https://www.cnbc.com/2024/06/17/nvidia-to-get-20percent-weighting-and-billions-in-investor-demand-while-apple-demoted-in-major-tech-fund.html
79 comments
Even if today's big tech players landed their own hardware in 2-3 years, there's no guarantee that they could match the software side of it. For AMD, the hardware is already there. All they have had standing in their way of owning a free money printer for the past 5 years is competitive software and that still somehow manages to elude them.
That's not how NVIDIA is making money. They're making money by selling metric shittons of A100's to Microsoft and Meta. Like ~$4 billion/year to each company for the past 2.5 years, with total CapEx outlays of around ~$10-14 billion/year each.
Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware. Much of it is being consumed by R&D without any products or profits involved directly. At some point, the pressures to monetize will come calling and turn off the blank check spigots.
Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware. Much of it is being consumed by R&D without any products or profits involved directly. At some point, the pressures to monetize will come calling and turn off the blank check spigots.
You say "that’s not how they’re making money", but the only reason anyone wants those A100s is because CUDA is pretty good. The hardware power is secondary in the sense that AMD is competitive with them — one could imagine a dominant AMD. But their stack is missing an API that works well.
How good A100/H200 are aren't the point, it's how essential they are to customers' profitability that matters. Zuck in particular still seems to be doubling-down on "build it, and they will come" while under continuing growing pressure to implement org-wide cost controls and wear big-boy business pants.
maybe because there's actual value in the systems being built and it's not simply blockchain 2.0 just because you keep repeating it?
there's a fairly incredible amount of commercial value in a system that can provide a workable "fuzzy match" or a system that can produce an approximately-optimal outcome without an intractable optimization step, for example. what is the commercial value of making the entirety of US logistics even 5-10% more efficient, or work-scheduling optimizations in a server farm?
like people just keep blindly repeating that there's no profitability there regardless of all the places it's quietly being deployed to great effect.
there's a fairly incredible amount of commercial value in a system that can provide a workable "fuzzy match" or a system that can produce an approximately-optimal outcome without an intractable optimization step, for example. what is the commercial value of making the entirety of US logistics even 5-10% more efficient, or work-scheduling optimizations in a server farm?
like people just keep blindly repeating that there's no profitability there regardless of all the places it's quietly being deployed to great effect.
> Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware.
IIUC, OpenAI leased the model to Microsoft to power LLM search on Bing. OpenAI doesn't run the inference on their machines. Microsoft also added LLM features into Office (and Windows, too, right?)
What do you think is powering all that inference?
IIUC, OpenAI leased the model to Microsoft to power LLM search on Bing. OpenAI doesn't run the inference on their machines. Microsoft also added LLM features into Office (and Windows, too, right?)
What do you think is powering all that inference?
Those features are mostly being added to existing products and therefore aren't supporting new revenue streams. It remains to be seen if these features are popular enough to become table stakes for the existing OS and productivity software businesses and justify continued investment.
And why aren't they buying MI250Xs instead?
Prior vendor relationships and standardization. It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible. Meta's Reality Labs is sort-of a special case in that it does more "throwaway"-type stuff, but it still aims to standardize and not deviate that much because it slows things down.
>It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible.
Right, and they are standardizing on CUDA. Something no other company has any chance of matching for the forseeable future.
Right, and they are standardizing on CUDA. Something no other company has any chance of matching for the forseeable future.
> Right, and they are standardizing on CUDA.
How many people are writing to CUDA directly, and how many folks are using something like PyTorch front-end where the back-end can potentially be changed?
* https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-a...
* https://rocm.docs.amd.com/projects/install-on-linux/en/devel...
* https://dev-discuss.pytorch.org/t/opencl-backend-important-u...
How many people are writing to CUDA directly, and how many folks are using something like PyTorch front-end where the back-end can potentially be changed?
* https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-a...
* https://rocm.docs.amd.com/projects/install-on-linux/en/devel...
* https://dev-discuss.pytorch.org/t/opencl-backend-important-u...
There are tons of use cases for GPGPU, the compute world isn't only about Pytorch and Tensorflow.
And even with middleware, turns out CUDA has the best debugging tooling, followed by DirectCompute/DirectML and Metal.
And even with middleware, turns out CUDA has the best debugging tooling, followed by DirectCompute/DirectML and Metal.
sure, it's things like oneapi that have pluggable backends. spir-v/vulkan compute too.
now, for how many of those things does AMD have a pluggable backend that actually works? none of them, unfortunately. Octane implemented on SPIR-V - on NVIDIA and Apple, because those were the runtimes that worked. AMD and Intel couldn't even compile it successfully (intel probably would today, I'd guess). Blender tried to support AMD for a long time, and the runtime was so buggy and defective they eventually pulled support from OpenCL entirely, since the OpenCL only ever ran properly on NVIDIA and Apple, and both of those have alternative interfaces that are cleaner and faster. The only point was supporting AMD, and AMD's OpenCL implementation is so buggy that it ended up being a de-facto AMD-specific codepath anyway.
That's the problem, AMD's software stacks are fractally broken. Not just HIP and their oneAPI adapter, but the ROCm it runs on. Not just ROCm, but the OpenCL it runs on. Not just openCL, but the kernel scheduler it runs on. Everything is broken at every level even on supported hardware.
Saying "it's just software" is a handwave. There are literally entire stacks here that are not fit-for-purpose and need to be substantially rebuilt before you can start building the layers above them. There isn't a stable enough basis to just implement an adapter for these pluggable frameworks, because the runtime and the kernel code are also broken.
now, for how many of those things does AMD have a pluggable backend that actually works? none of them, unfortunately. Octane implemented on SPIR-V - on NVIDIA and Apple, because those were the runtimes that worked. AMD and Intel couldn't even compile it successfully (intel probably would today, I'd guess). Blender tried to support AMD for a long time, and the runtime was so buggy and defective they eventually pulled support from OpenCL entirely, since the OpenCL only ever ran properly on NVIDIA and Apple, and both of those have alternative interfaces that are cleaner and faster. The only point was supporting AMD, and AMD's OpenCL implementation is so buggy that it ended up being a de-facto AMD-specific codepath anyway.
That's the problem, AMD's software stacks are fractally broken. Not just HIP and their oneAPI adapter, but the ROCm it runs on. Not just ROCm, but the OpenCL it runs on. Not just openCL, but the kernel scheduler it runs on. Everything is broken at every level even on supported hardware.
Saying "it's just software" is a handwave. There are literally entire stacks here that are not fit-for-purpose and need to be substantially rebuilt before you can start building the layers above them. There isn't a stable enough basis to just implement an adapter for these pluggable frameworks, because the runtime and the kernel code are also broken.
MAANG are willing to invest in ASIC engineering like Apple does but are willing to buy and rent things from others in the short-term. They will eventually wise-up, and design their own custom AI accelerators. Meta already designs and builds their own servers.
>there's no guarantee that they could match the software side of it.
How many times in history has this ever happened? That a product was so good no competitors could match it, forever?
How many times in history has this ever happened? That a product was so good no competitors could match it, forever?
Apple has been doing this for decades with multiple product lines.
Apple controls the customer interface and has a huge network effect. Consumers of AI services (when they exist in financially significant numbers) don’t care which hardware processes their inference (or trains their network), they only care about speed and price.
A better example might be Intel, but Intel at least had a unique instruction set. And of course Intel is more of a cautionary tale than a story of eternal market dominance.
A better example might be Intel, but Intel at least had a unique instruction set. And of course Intel is more of a cautionary tale than a story of eternal market dominance.
Is Nvidia's technology so advanced that others can't copy them? Is their hardware (GPU+CPU chips) [1], software (CUDA) or its combination their moat?
Analogy: Apple's hardware + software combination as competitive advantage
Are they just the beneficiaries of inside (R&D) or outside (AI) market forces?
History has shown that the latter is not sustainable (dot-com, nanotech, covid, crypto, AI)
TLDR: Shovels eventually become commodities
[1] https://www.youtube.com/watch?v=Z1RrVgI_HQE
Analogy: Apple's hardware + software combination as competitive advantage
Are they just the beneficiaries of inside (R&D) or outside (AI) market forces?
History has shown that the latter is not sustainable (dot-com, nanotech, covid, crypto, AI)
TLDR: Shovels eventually become commodities
[1] https://www.youtube.com/watch?v=Z1RrVgI_HQE
They snatched Infiniband a couple of years ago. The deep integration of low latency fabric with their GPUs is basically Nvidia owning the stack that makes GPU compute scalable beyond the single chassis. There’s a long road to commoditization.
AMD has spent years developing MCM/multi-chip packaging to solve exactly this same problem. And they have Xilinx too, and Infinity Link (which is different from infinity fabric), and all these other things that help them build systems too. They aren't helpless babes unaware of the problem either.
(although I agree with the point that yes, NVIDIA was thinking about systems engineering much sooner and more comprehensively than AMD was. And not just multi-node scaling, but also things like the programming model (CUDA vs OpenCL/AMD APP/HSA Framework), and they also seem to have correctly identified that ethernet allows higher bandwidth-density than pcie as a transport for this interconnect, giving them an implementation advantage. Similarly, despite all the praise for AMD's packaging, it's often come with downsides and caveats, and NVIDIA usually isn't far behind with one that doesn't. Eg Fury X/Vega/Radeon VII vs GP100/GV100/GA100, or MI300X vs B200. And NVIDIA has explicitly been much earlier-to-market on the systems-scaling stuff like NVSwitch or pre-specced high-performance systems modules like DGX, which AMD still doesn't really have an answer for despite NVSwitch being over a decade old at this point etc.)
Tesla built a whole interconnect system for Dojo too, iirc, that was an explicit focus in their design. Not surprising for Jim Keller. But Jensen Huang nailed that requirement too.
(although I agree with the point that yes, NVIDIA was thinking about systems engineering much sooner and more comprehensively than AMD was. And not just multi-node scaling, but also things like the programming model (CUDA vs OpenCL/AMD APP/HSA Framework), and they also seem to have correctly identified that ethernet allows higher bandwidth-density than pcie as a transport for this interconnect, giving them an implementation advantage. Similarly, despite all the praise for AMD's packaging, it's often come with downsides and caveats, and NVIDIA usually isn't far behind with one that doesn't. Eg Fury X/Vega/Radeon VII vs GP100/GV100/GA100, or MI300X vs B200. And NVIDIA has explicitly been much earlier-to-market on the systems-scaling stuff like NVSwitch or pre-specced high-performance systems modules like DGX, which AMD still doesn't really have an answer for despite NVSwitch being over a decade old at this point etc.)
Tesla built a whole interconnect system for Dojo too, iirc, that was an explicit focus in their design. Not surprising for Jim Keller. But Jensen Huang nailed that requirement too.
> and that still somehow manages to elude them.
AMD is definitely behind and slower than I would think they should be, but they are making progress. I suspect they are working pretty seriously on it, just doing so quietly. At least, I hope that's what's going on because if not, it's a tragedy.
AMD is definitely behind and slower than I would think they should be, but they are making progress. I suspect they are working pretty seriously on it, just doing so quietly. At least, I hope that's what's going on because if not, it's a tragedy.
That interview with their CEO that's still on the front page says to me they aren't.
https://news.ycombinator.com/item?id=40697341
https://news.ycombinator.com/item?id=40697341
> All they have had standing in their way of owning a free money printer for the past 5 years is competitive software and that still somehow manages to elude them.
I'm pretty sure it's been more than 5 years. People have been griping since before ROCm was released which was around 7 years ago.
I'm pretty sure it's been more than 5 years. People have been griping since before ROCm was released which was around 7 years ago.
3 and 4nm Fabs are sold out to Nvidia and Apple.
[deleted]
[deleted]
What I don't fully understand is why TSMC's stock isn't skyrocketing as well. Isn't nvidia totally dependent on them, how can they scale to meet their sky-high evaluation without TSMC expanding massively as well.
Emotional investing != value investing. TSMC is an absolutely critical business to the Western world, and continues to be. Even with the CHIPS and Science Act, America still lacks silicon full-stack self-reliance.
TSMC is a beneficiary but not as much as you'd think because NVIDIA charges a very huge markup and HMB DRAM is about half the BOM cost.
Why shouldn't Micron get a pop then though?
Micron stock has done very well.
DRAM glut occurs periodically, cutting into their bottom lines, and DRAM is a pretty low-margin product due to fierce competition. So Micron is a riskier investment.
TSMC's stock is skyrocketing +70% in a year.
TSMC revenue is smaller than Nvidia revenue.
TSMC profit margin is great, but not as great as Nvidia.
TSMC revenue is smaller than Nvidia revenue.
TSMC profit margin is great, but not as great as Nvidia.
[deleted]
Nvidia probably has a better chance of adding new suppliers than TSMC finding a new Nvidia to sell to.
Factually incorrect. If Nvidia dropped TSMC, TSMC could still sell to apple, AMD and others. If TSMC dropped Nvidia, Nvidia would literally have no other options for cutting edge nodes.
TSMC already sells to those companies, so I don't think that addressed the scenario. I would imagine that other chip fabs would be able to pick up the slack, including many that are purchasing from ASML already and are on track to begin matching TSMC from a node perspective in a year or 2.
It's up 70% YoY.
Meta is already working on this [1], not sure it can replace NVIDIA for training large models within that time frame however. The ecosystem around their chips is what gives a huge competitive advantage, not having to build entire libraries and optimize a ton of code goes a long way in adoptions and retention.
N.B.: there are other 3rd-party competitors like Cerebras [2] who offer all-in-one solutions for their giant wafers along with libraries and data centers but I’m not sure behemoths would migrate to these offerings either
[1] https://ai.meta.com/blog/next-generation-meta-training-infer...
[2] https://www.cerebras.net/
N.B.: there are other 3rd-party competitors like Cerebras [2] who offer all-in-one solutions for their giant wafers along with libraries and data centers but I’m not sure behemoths would migrate to these offerings either
[1] https://ai.meta.com/blog/next-generation-meta-training-infer...
[2] https://www.cerebras.net/
if it was easy to design gear microsoft would already be doing it. NVIDIA is where they are because they are currently the best at designing that kind of gear. AMD has been chasing them for many years. I don't know that MSFT or Meta will suddenly be competitive.
That's with the assumption that AI progress doesn't just stall out.
I doubt they'll continue to buy a significant amount of GPUs if they stop seeing significant enough performance/quality gains.
I doubt they'll continue to buy a significant amount of GPUs if they stop seeing significant enough performance/quality gains.
They are already starting this work - and there is huge incentive to commoditize it and make fit into their datacenter designs.
[deleted]
Where do Microsoft/OpenAI and Meta plan to get these NVDA-killing chips fabbed...?
What I want to know is how complex is the inference on these models? My understanding is that the generation of responses (the inference) is what most of the compute is spent upon (as opposed to the training). But is inference much more complicated than a very very very big matrix matrix operation? Does it require many of the capabilities that these fancy GPU's have?
I'm also not clear how much of the cost of running models is really because the models are too large to fit on a single device? If we start making devices with very large (terabytes) of memory, how much compute do we then need?
tldr; Would it not be cheaper to build more specialized hardware that has massive amounts of memory (enough to hold the entire model on a single device)? I'm wondering if this sort of hardware (large banks of memory + relatively simple compute) is not a market that many of NVIDIA's competitors can easily enter.
I'm also not clear how much of the cost of running models is really because the models are too large to fit on a single device? If we start making devices with very large (terabytes) of memory, how much compute do we then need?
tldr; Would it not be cheaper to build more specialized hardware that has massive amounts of memory (enough to hold the entire model on a single device)? I'm wondering if this sort of hardware (large banks of memory + relatively simple compute) is not a market that many of NVIDIA's competitors can easily enter.
> inference more expensive than training
That's a gamble. It's the whole idea behind a pre-trained model (GPT). If you make a really good model then you can use it a lot of times. That seems to be playing out as intended at the moment.
> very very very big matrix operation
To do efficiently, it's many slightly smaller very very big matrix operations.
> does it require advanced GPU capabilities
No, but GPUs don't have a lot of overhead for those very very big matrix operations.
> high costs because models are too large to fit on a single device
Engineer costs are high, and communication costs are also high. Those are mostly just barriers to entry though. Compute still dominates.
> competitors with more RAM
It's a little more complicated than that because extra RAM has speed-of-light delays and other such problems. If you don't expose a CUDA interface then you have a long hill to climb to get even a couple customers. A couple extra caches or other more esoteric memory APIs won't be easy for the industry to adopt (maybe a clever electrical engineer can fix that; I don't have hard proofs of those bounds).
That's a gamble. It's the whole idea behind a pre-trained model (GPT). If you make a really good model then you can use it a lot of times. That seems to be playing out as intended at the moment.
> very very very big matrix operation
To do efficiently, it's many slightly smaller very very big matrix operations.
> does it require advanced GPU capabilities
No, but GPUs don't have a lot of overhead for those very very big matrix operations.
> high costs because models are too large to fit on a single device
Engineer costs are high, and communication costs are also high. Those are mostly just barriers to entry though. Compute still dominates.
> competitors with more RAM
It's a little more complicated than that because extra RAM has speed-of-light delays and other such problems. If you don't expose a CUDA interface then you have a long hill to climb to get even a couple customers. A couple extra caches or other more esoteric memory APIs won't be easy for the industry to adopt (maybe a clever electrical engineer can fix that; I don't have hard proofs of those bounds).
> My understanding is that the generation of responses (the inference) is what most of the compute is spent upon (as opposed to the training).
no, training takes hundreds/thousands of high-end HPC gpus for days/weeks/months/years. Inference runs in seconds.
The argument around "inference is more important/will be a larger market" revolves around the total volume of inference performed being larger than the amount to train. A million consumers perform inference, but you only train it once.
The problem is the training really never stops, it's not like you train a model and you're done. Even if there were no new innovations it would be worth continuing to train models for years even just fitting the models we've got to the data better. LORAs and QLORAs to train. Etc. As you adopt it more widely, the amount of things we need to train will continue to increase anyway. There is no particular reason to assume the growth of training needs will ever stop, or even decrease below current growth rates (TOPS, not $). It will certainly not continue at 300% margins forever, of course. But I just don't think the assertion that training needs will decrease is supportable - this seems like a Jevon's Paradox moment, the increased utility and accessibility of this will drive an enormous increase in usage, both training and inference.
It's also a misleading statement around revenue. Yes, there will be a lot of inference that's done... and it's trivial to implement and will be in almost every single device etc. Training is really the only thing it's feasible to build an ecosystem/moat around, because inference is going to be done by everyone, and will have almost zero margin for either standalone devices or the IP. Training is the interesting and profitable part of the market, and new discoveries go from a "research -> training/scale-out -> inference" pipeline that tends to favor NVIDIA because they have the ecosystem for the research etc. It has been the flexibility and programmability of the GPGPU model that has won so far - you can certainly get a lot of TOPS on Trainium or whatever, but if they can't train the techniques currently used then it doesn't matter.
They have spent an unfathomable sum of money (tens of billions) fostering that ecosystem, sponsoring academics and researchers and conferences and paying for people to write libraries and tuning their software etc. There are millions of man-hours of work that need to be done even if you fully understand exactly the minimal combination of pieces to get there, and that doesn't even begin to replicate the mindshare factors etc. Intel increasingly has a perfectly viable software ecosystem, but that doesn't matter if nobody is using it. NVIDIA and Apple are really the only 2 ecosystems anyone uses, in the sense of being a place where work takes place on the platform for the sake of being used by people on that platform. Everything else is just a port-of-convenience for access to hardware and nobody cares a whit about building a long-term community around SyCL vs ROCm as a thing in itself. That's table stakes to build an ecosystem, not the ecosystem itself. Having a complete, viable software ecosystem was the ask 10 years ago, to be competitive in 2024 you need the people using it because we're no longer in the blue-sky/academic-research stage here.
It's like thinking you can be a reddit competitor just because you implemented lemmy - the thing that makes reddit reddit is not the source code, that's table stakes, not a platform. Building a mapping layer over the top might help, but it also removes any incentive for the meta-layer to care about you as anything more than a "utility". Literally the players that are going to succeed are the ones like NVIDIA, Apple, and Sony where there is a first-party culture and an ecosystem that is intellectually self-sustaining as a target in itself (and not a utility/build target for someone else). Maybe microsoft to a lesser extent (they are in a good place given their control of the dominant client OS).
The "iphone bubble" never popped - yearly revenue increased more or less continuously for a decade, even during the Great Recession it went up. Even though the smartphone market did see many other viable players emerge, Apple retained a plurality control of the market and remains the single largest and most influential player in design as well as direct market influence. Saying something is a "bubble" is begging the question, maybe NVIDIA just popularized a whole new device/field (GPGPU) in the same way Apple did and the market grows underneath them. People are blindly making the assertions that the revenue MUST be unsustainable because "well I don't see any value!" and then building this whole little chain of logic on some things that are ultimately propositions/assertions and not fait-accompli.
no, training takes hundreds/thousands of high-end HPC gpus for days/weeks/months/years. Inference runs in seconds.
The argument around "inference is more important/will be a larger market" revolves around the total volume of inference performed being larger than the amount to train. A million consumers perform inference, but you only train it once.
The problem is the training really never stops, it's not like you train a model and you're done. Even if there were no new innovations it would be worth continuing to train models for years even just fitting the models we've got to the data better. LORAs and QLORAs to train. Etc. As you adopt it more widely, the amount of things we need to train will continue to increase anyway. There is no particular reason to assume the growth of training needs will ever stop, or even decrease below current growth rates (TOPS, not $). It will certainly not continue at 300% margins forever, of course. But I just don't think the assertion that training needs will decrease is supportable - this seems like a Jevon's Paradox moment, the increased utility and accessibility of this will drive an enormous increase in usage, both training and inference.
It's also a misleading statement around revenue. Yes, there will be a lot of inference that's done... and it's trivial to implement and will be in almost every single device etc. Training is really the only thing it's feasible to build an ecosystem/moat around, because inference is going to be done by everyone, and will have almost zero margin for either standalone devices or the IP. Training is the interesting and profitable part of the market, and new discoveries go from a "research -> training/scale-out -> inference" pipeline that tends to favor NVIDIA because they have the ecosystem for the research etc. It has been the flexibility and programmability of the GPGPU model that has won so far - you can certainly get a lot of TOPS on Trainium or whatever, but if they can't train the techniques currently used then it doesn't matter.
They have spent an unfathomable sum of money (tens of billions) fostering that ecosystem, sponsoring academics and researchers and conferences and paying for people to write libraries and tuning their software etc. There are millions of man-hours of work that need to be done even if you fully understand exactly the minimal combination of pieces to get there, and that doesn't even begin to replicate the mindshare factors etc. Intel increasingly has a perfectly viable software ecosystem, but that doesn't matter if nobody is using it. NVIDIA and Apple are really the only 2 ecosystems anyone uses, in the sense of being a place where work takes place on the platform for the sake of being used by people on that platform. Everything else is just a port-of-convenience for access to hardware and nobody cares a whit about building a long-term community around SyCL vs ROCm as a thing in itself. That's table stakes to build an ecosystem, not the ecosystem itself. Having a complete, viable software ecosystem was the ask 10 years ago, to be competitive in 2024 you need the people using it because we're no longer in the blue-sky/academic-research stage here.
It's like thinking you can be a reddit competitor just because you implemented lemmy - the thing that makes reddit reddit is not the source code, that's table stakes, not a platform. Building a mapping layer over the top might help, but it also removes any incentive for the meta-layer to care about you as anything more than a "utility". Literally the players that are going to succeed are the ones like NVIDIA, Apple, and Sony where there is a first-party culture and an ecosystem that is intellectually self-sustaining as a target in itself (and not a utility/build target for someone else). Maybe microsoft to a lesser extent (they are in a good place given their control of the dominant client OS).
The "iphone bubble" never popped - yearly revenue increased more or less continuously for a decade, even during the Great Recession it went up. Even though the smartphone market did see many other viable players emerge, Apple retained a plurality control of the market and remains the single largest and most influential player in design as well as direct market influence. Saying something is a "bubble" is begging the question, maybe NVIDIA just popularized a whole new device/field (GPGPU) in the same way Apple did and the market grows underneath them. People are blindly making the assertions that the revenue MUST be unsustainable because "well I don't see any value!" and then building this whole little chain of logic on some things that are ultimately propositions/assertions and not fait-accompli.
I mean essentially it boils down to the fact that none of the huge cloud providers - for now at least - produce their own hardware enmasse.
The moment it happens, Nvidia will stop being that profitable from hardware sales there. For now they should be able to break records YoY. And indeed it feels like 2-3 years it will last.
The moment it happens, Nvidia will stop being that profitable from hardware sales there. For now they should be able to break records YoY. And indeed it feels like 2-3 years it will last.
Amazon makes Trainium and Inferentia, and have far more of them available than NVIDIA GPUs. But the software doesn't work as well, so they aren't popular. NVIDIA has a huge software advantage that few people realize.
Agree. But they also have a lot of hardware features. Mixed precision GEMM (fp8 and soon fp3), async copies and now TMA, now global shared memory. And with Blackwell they claim 50% less power, too. They are 10 years ahead of the competition and with a massive budget. I think they reached escape velocity.
Intel and Microsoft had the problem of competing with their previously sold products. Most of their users only needed a basic PC for email, Word, and Excel. Nvidia doesn't have this problem because previous hardware generations quickly become obsolete as the demand for compute keeps growing and it's easier to manage newer clusters.
Intel and Microsoft had the problem of competing with their previously sold products. Most of their users only needed a basic PC for email, Word, and Excel. Nvidia doesn't have this problem because previous hardware generations quickly become obsolete as the demand for compute keeps growing and it's easier to manage newer clusters.
s/fp3/fp4
I know that the subtext to this is that GPUs will possibly be affordable again, and to that, all I can say is, don't play with my emotions.
> This particular hype train has about 1-2 years more of gas until people stop being interested in GenAI.
Fixed that for you.
Fixed that for you.
Well, I regret buying XLK compared to other tech ETFs. Bad weighing and potentially high tax costs going forward.
There's an artificial rule that a company can't be more than 4.5% of the index except the top 2.
Nvidia was ahead of Apple last week, so they need to sell huge amounts of Apple (distributing capital gains) to buy a bunch of Nvidia.
Worse, Apple is ahead of Nvidia at the moment. More likely than not, it will be in one quarter. So we get yet another huge rotation.
There's an artificial rule that a company can't be more than 4.5% of the index except the top 2.
Nvidia was ahead of Apple last week, so they need to sell huge amounts of Apple (distributing capital gains) to buy a bunch of Nvidia.
Worse, Apple is ahead of Nvidia at the moment. More likely than not, it will be in one quarter. So we get yet another huge rotation.
Folks, from someone that has made all of these errors over twenty years, the best thing you can do is just buy a total stock market index fund (eg. VTI).
Don't try to overweight or underweight companies or guess winners or losers.
Twenty five years ago doing this sort of guessing would have had you underweighting Apple for example.
And frankly, I think the same mistake today would be overweighting NVIDIA.
Don't try to overweight or underweight companies or guess winners or losers.
Twenty five years ago doing this sort of guessing would have had you underweighting Apple for example.
And frankly, I think the same mistake today would be overweighting NVIDIA.
this was all true 20 years ago. today trading is so universal and available that only buy VTI if you hate money :-)
if you have the time then start trading or leverage up and you'll beat the index.
if you have the time then start trading or leverage up and you'll beat the index.
Why does everyone think that NVDA currently simply sits on its laurels? I bet they are heavily investing and innovating, perhaps creating another CUDA-level software stack. I don't understand why people tend to think META, MSFT et al. are closing the gap while NVDA just sits pretty.
Does anyone think that there will be enough GPUs in data centers soon? I honestly don't know. But it seems like there should be.
Like I (kinda) get the scaling laws with LLMs. But at what point do we run out of training data? Is there that much "meat on the bone" that will carry Nvidia a few more years? And at what point does spending $x million more on training the base models no longer provide that much more value?
Like I (kinda) get the scaling laws with LLMs. But at what point do we run out of training data? Is there that much "meat on the bone" that will carry Nvidia a few more years? And at what point does spending $x million more on training the base models no longer provide that much more value?
> But at what point do we run out of training data?
We have at least another order of magnitude of additional data via audio-video. There also is synthetic data, such as was likely done for SORA. And we likely could increase that another few orders of magnitude if we had more universal sensors. Windows Recall is crazy for privacy reasons, but gathering feeds of data from everyone as they go about their lives, physical as well as digital provides much more room for training. You also need to take into account data-growth. How much more text data is generated daily, relative to how much has ever existed?
There also is probably another 1-2 orders of magnitude more data in private locations and also growing fast. Some of that can be captured via RAG. But even if a small fraction is used for fine-tuning, in aggregate that is a lot of GPU.
We have at least another order of magnitude of additional data via audio-video. There also is synthetic data, such as was likely done for SORA. And we likely could increase that another few orders of magnitude if we had more universal sensors. Windows Recall is crazy for privacy reasons, but gathering feeds of data from everyone as they go about their lives, physical as well as digital provides much more room for training. You also need to take into account data-growth. How much more text data is generated daily, relative to how much has ever existed?
There also is probably another 1-2 orders of magnitude more data in private locations and also growing fast. Some of that can be captured via RAG. But even if a small fraction is used for fine-tuning, in aggregate that is a lot of GPU.
> But at what point do we run out of training data?
Isn't the training itself very dependent on each algorithm. Even if you somehow gathered all the plausible training data, you'd have to run it through again for most improvements of your system.
But there are trends to look at in this direction: changes in the training that make it much, much faster; And processors which become more effective (fewer of them needed).
So anyway, very long term, sure. For now, doesn't seem to be a problem.
For that matter, for now, there is not only building new data centers, but also upgrading existing ones to the more efficient processors.
Isn't the training itself very dependent on each algorithm. Even if you somehow gathered all the plausible training data, you'd have to run it through again for most improvements of your system.
But there are trends to look at in this direction: changes in the training that make it much, much faster; And processors which become more effective (fewer of them needed).
So anyway, very long term, sure. For now, doesn't seem to be a problem.
For that matter, for now, there is not only building new data centers, but also upgrading existing ones to the more efficient processors.
> But at what point do we run out of training data?
Multi-modal LLMs trained on video / YouTube won't run out of data any time soon. And LLMs can always train on an endless supply of synthetic data like code.
> And at what point does spending $x million more on training the base models no longer provide that much more value?
My understanding is we're still waiting to see where on the S-curve LLMs are in regards to scaling. So far it seems like scaling hasn't hit any diminishing returns.
I haven't read many sources talking about the practicality of that though. Even if LLMs continue scaling performance as we throw more compute + data at them, at some point it may just become uneconomical to continue. Capital expenditures also have their own utility curve. How far away are we from that point?
Multi-modal LLMs trained on video / YouTube won't run out of data any time soon. And LLMs can always train on an endless supply of synthetic data like code.
> And at what point does spending $x million more on training the base models no longer provide that much more value?
My understanding is we're still waiting to see where on the S-curve LLMs are in regards to scaling. So far it seems like scaling hasn't hit any diminishing returns.
I haven't read many sources talking about the practicality of that though. Even if LLMs continue scaling performance as we throw more compute + data at them, at some point it may just become uneconomical to continue. Capital expenditures also have their own utility curve. How far away are we from that point?
> Capital expenditures also have their own utility curve
This a function of the use-cases. For simple use cases like document summarization, or speech-to-text, we might already be there. There are already applications where using something like gpt-3-turbo or a local-llm vs gpt-4 make more economic sense. The marginal improvement in reasoning or extra knowledge don't provide benefits relative to the inference costs. And you can see this playing out with Mistral, Anthropic and Google also providing models at different cost/complexity tradeoffs.
But .... if we get _new_ use cases, emergent use cases, from bigger models, there will be continued value in chasing that frontier. Reasoning is still poor in many areas. The ability to learn outside of application specific memory patterns that store state is largely unexplored. We are still far from AGI and there is a ton of value to be minded between current systems and a superhuman capabilities
This a function of the use-cases. For simple use cases like document summarization, or speech-to-text, we might already be there. There are already applications where using something like gpt-3-turbo or a local-llm vs gpt-4 make more economic sense. The marginal improvement in reasoning or extra knowledge don't provide benefits relative to the inference costs. And you can see this playing out with Mistral, Anthropic and Google also providing models at different cost/complexity tradeoffs.
But .... if we get _new_ use cases, emergent use cases, from bigger models, there will be continued value in chasing that frontier. Reasoning is still poor in many areas. The ability to learn outside of application specific memory patterns that store state is largely unexplored. We are still far from AGI and there is a ton of value to be minded between current systems and a superhuman capabilities
>> Does anyone think that there will be enough GPUs in data centers soon? I honestly don't know. But it seems like there should be.
Not sure about that, but given the TAM, associated market value, and obvious juicy opportunity -- what prevents driven alternatives (even Apple Silicon) from biting off portions of this TAM?
Not sure about that, but given the TAM, associated market value, and obvious juicy opportunity -- what prevents driven alternatives (even Apple Silicon) from biting off portions of this TAM?
Anyone knows why this transition didn't or won't happen slowly (-ish)?
The fund managers are aware of the problem, and normally use funds influx, dividends and redemptions to keep things more or less balanced (and have some leaway.)
The fund managers are aware of the problem, and normally use funds influx, dividends and redemptions to keep things more or less balanced (and have some leaway.)
So you have a gold field.
The miners are digging up the gold and selling it. Say they manage to sell $100 worth of gold.
Meanwhile a guy sets up shop selling shovels. For the sake of argument he made $100 selling shovels to the miners.
How much money did the miners make?
How long can that continue until something changes?
TLDR: For those that don't care for analogies a toolmaker can't be making more money then the companies using those tools to make money... or some very obvious things happen.
The miners are digging up the gold and selling it. Say they manage to sell $100 worth of gold.
Meanwhile a guy sets up shop selling shovels. For the sake of argument he made $100 selling shovels to the miners.
How much money did the miners make?
How long can that continue until something changes?
TLDR: For those that don't care for analogies a toolmaker can't be making more money then the companies using those tools to make money... or some very obvious things happen.
The problem with this analogy is the gold miners are not just trying to mine gold. They are simultaneously doing a bunch of other economic activities that generally make money, as well as trying to mine gold. Some are selling yearbooks back at camp, or putting on plays, or selling food, etc.
nvidia can't make more money than the client companies make altogether, but they certainly can make more money than any(/all) of the client companies make using nvidia's products. It could be the case(and currently is the case) that other non-AI revenue streams are just being handed over to nvidia in the hopes the AI revenue streams take off eventually.
nvidia can't make more money than the client companies make altogether, but they certainly can make more money than any(/all) of the client companies make using nvidia's products. It could be the case(and currently is the case) that other non-AI revenue streams are just being handed over to nvidia in the hopes the AI revenue streams take off eventually.
The analogy assumes you already know where the gold is.
Right now companies are spending a ton of money trying to find gold, and that costs capital. That's why they're slapping the word "AI" onto everything to see if it sticks. And re:
> a toolmaker can't be making more money then the companies using those tools to make money.
Every historical gold rush has had exactly that happen which is the point of the analogy. You have a bunch of broke pioneers, a few people that struck it rich on their stakes, and then a saloon owner that rakes in steady income from alcoholic prospectors.
Right now companies are spending a ton of money trying to find gold, and that costs capital. That's why they're slapping the word "AI" onto everything to see if it sticks. And re:
> a toolmaker can't be making more money then the companies using those tools to make money.
Every historical gold rush has had exactly that happen which is the point of the analogy. You have a bunch of broke pioneers, a few people that struck it rich on their stakes, and then a saloon owner that rakes in steady income from alcoholic prospectors.
> a toolmaker can't be making more money then the companies using those tools to make money.
In the historical gold rush, you also had speculators funding prospectors. That might just be personal savings, friends and family or investors. Similar to the current AI rush, you have VCs and corporate profits from companies pivoting into AI that, theoretically, could all go bust. But there still is a whole lot of capital to expend even if the investments are never profitable.
In the historical gold rush, you also had speculators funding prospectors. That might just be personal savings, friends and family or investors. Similar to the current AI rush, you have VCs and corporate profits from companies pivoting into AI that, theoretically, could all go bust. But there still is a whole lot of capital to expend even if the investments are never profitable.
Not quite how it works, even in your analogy. The miners are using capital to buy tools, not profits. They or may not make a profit but there is no way they will ever make a profit without first buying shovels. Buying shovels is not optional. Eventually something changes, yes - and eventually can be a very long time in finance. Often much longer than your useful investing outlook or patience.
I know. The idea was to make people think what happens after the rush but apparently that fell flat.
All of the major cloud providers are building their own chips. Hell, Microsoft's hardware dev org pretty much all former intel employees.
Yeah so that is one of the possibilities, people start making their own shovels.
There are a bunch of others:
People start getting shovels from somewhere else even if they aren't quite as good (AMD).
They realise there is actually only a small group of them buying literally all the shovels and they use that position to force more favourable pricing.
Another company works out how to make a better shovel, or worse something that isn't even a shovel anymore and is much more efficient at mining. (These don't exist for training yet but there are inference chips that make NVidia look terrible in comparison)
etc.
The fact of the matter is that it's not a stable situation, something has to give. Hype can allow these sorts of massive imbalances to exist for short periods but there is always balance in the end.
There are a bunch of others:
People start getting shovels from somewhere else even if they aren't quite as good (AMD).
They realise there is actually only a small group of them buying literally all the shovels and they use that position to force more favourable pricing.
Another company works out how to make a better shovel, or worse something that isn't even a shovel anymore and is much more efficient at mining. (These don't exist for training yet but there are inference chips that make NVidia look terrible in comparison)
etc.
The fact of the matter is that it's not a stable situation, something has to give. Hype can allow these sorts of massive imbalances to exist for short periods but there is always balance in the end.
I'd equate Nvidia to being the seller of materials needed to make a specific shovel shown to be very good at digging gold. Microsoft and their kiln are buying all the materials to make their own shovel shop. They're also buying their own shovels. But that's probably being too pedantic here.
This particular hype train has about 2-3 years more of gas until Microsoft/OpenAI and Meta stop buying absurd amounts of other peoples' gear and design their own.