Ask HN: What is everyone doing with all of these GPUs?
I know accelerator demand is blowing up. However, there are only a few players training foundation models. What are the core use cases everyone else has for all of these accelerators? Fine tuning, smaller transformer models, general growth in deep learning?
8 comments
Different scenarios have varying demands for GPU types. For tasks like model inference or basic operations, a CPU or even on-device solutions (mobile, web) might suffice.
When a GPU is necessary, common choices include T4, 3090, P10, V100, etc., selected based on factors like price, required computing power, and memory capacity.
Model training also has diverse needs based on the specific task. For basic, general-purpose vision tasks, 1 to 50 cards like the 3090 might be enough. However, cutting-edge areas like visual generation and LLMs often require A100s or A800s, scaling from 1 to even thousands of cards.
When a GPU is necessary, common choices include T4, 3090, P10, V100, etc., selected based on factors like price, required computing power, and memory capacity.
Model training also has diverse needs based on the specific task. For basic, general-purpose vision tasks, 1 to 50 cards like the 3090 might be enough. However, cutting-edge areas like visual generation and LLMs often require A100s or A800s, scaling from 1 to even thousands of cards.
Inference. 99% of the customers that aren't buying GPUs to train on are either using it for inference or putting it in a datacenter where inference is the intended use-case.
Absolutely.
For some reason inference seems to be overlooked. A lot of “ink” has been spilled over GPUs for training tasks but at the end of the day if you can’t do inference you can’t serve users and you can’t make money.
For some reason inference seems to be overlooked. A lot of “ink” has been spilled over GPUs for training tasks but at the end of the day if you can’t do inference you can’t serve users and you can’t make money.
Think the massive increase in demand is due to mass inference of open source LLMs? Or is the transformer architecture driving mass inference of other models too?
You might be thinking too far into this. The biggest customers are bulk buyers that are either training on a private cluster (eg. Meta or OpenAI), or selling their rack-space to other businesses. These are the people that are paying money and increasing the demand for GPU hardware; what happens to the businesses they provide for almost doesn't even matter as long as they pay for the compute. The "driver" for this demand is the hype. If people were laser-focused on the best-value solution, then everyone would pay for OpenAI's compute since it's cheaper than GPU hosting.
The real root of the problem is that GPU compute is not a competitive market. The demand is less for GPUs and more for Nvidia hardware, because nobody else is shipping CUDA or CUDA-equivalent software. Thus the demand is artificially raised beyond whatever is reasonable since buyers aren't shopping in a reactive market. Basically the same story as what happened to Nvidia's hardware during the crypto mining rush.
The real root of the problem is that GPU compute is not a competitive market. The demand is less for GPUs and more for Nvidia hardware, because nobody else is shipping CUDA or CUDA-equivalent software. Thus the demand is artificially raised beyond whatever is reasonable since buyers aren't shopping in a reactive market. Basically the same story as what happened to Nvidia's hardware during the crypto mining rush.
> then everyone would pay for OpenAI's compute since it's cheaper than GPU hosting.
This is absolutely not true. The gap is narrowing as providers (Google, Anthropic, Deepseek) introduce cross request KV caching but it’s definitely not true for OAI (yet).
This is absolutely not true. The gap is narrowing as providers (Google, Anthropic, Deepseek) introduce cross request KV caching but it’s definitely not true for OAI (yet).
I'm playing UE5 games, and doing some computational chem with CUDA.