The state of silicon and the GPU poors(latent.space)
latent.space
The state of silicon and the GPU poors
https://www.latent.space/p/semianalysis
40 comments
In the early days of computing this was the same situation with CPUs.
However, I do want to complain about the poor tooling around GPUs, particularly package managers, and closedness of drivers. Getting stuff to work on your GPU is often a shitty and time-consuming experience.
However, I do want to complain about the poor tooling around GPUs, particularly package managers, and closedness of drivers. Getting stuff to work on your GPU is often a shitty and time-consuming experience.
I guess it just depends on what you want to do. Training AI? Probably better off renting computer at this point. Everyday use? I still have a 1070 in my main workstation that has survived three computer upgrades. I am looking to upgrade, but I’m not in a hurry.
The 1070/1080 are absolute champions of longevity if using a GPU... for graphics/gaming. I still have a gaming machine with my old 1080ti from 2017 in the house, for the most part it will play most of the latest games like Cyberpunk (albeit with no raytracing, but heyho) at 1080p/1440p at good frame rates. Every year that passes now I genuinely marvel at the useful lifespans of those parts, which has been completely unprecedented for me as someone who has been building gaming PCs since the 90s.
The MacBook Pro I bought in the same year is officially classed as "obsolete" by Apple.
The MacBook Pro I bought in the same year is officially classed as "obsolete" by Apple.
I ran a used GTX570 until it died in 2017. Then a used GTX780 until it died in 2021. Now I'm on a used Radeon 5600. I paid less than $200 for each of them.
I've also been running the same i5-3470 CPU since it was new.
Computers can last a long, long time if you can bear the thought of not being on the bleeding edge. Otherwise, yeah, you're going to be paying a premium for brand new tech. That's the trade you make chasing the newest and shiniest thing.
I've also been running the same i5-3470 CPU since it was new.
Computers can last a long, long time if you can bear the thought of not being on the bleeding edge. Otherwise, yeah, you're going to be paying a premium for brand new tech. That's the trade you make chasing the newest and shiniest thing.
>Now I'm on a used Radeon 5600.
Well AMD just ended support for Vega and Polaris architectures so you're shit out of luck there, proving the point of this thread.
And AMD was selling Vega APUs all the way in 2022. So imagine a year later you get the news that the chip in your system is not getting updates because it's architectures is too old. Lol.
I wish I couldn't forget AMD for this.
Well AMD just ended support for Vega and Polaris architectures so you're shit out of luck there, proving the point of this thread.
And AMD was selling Vega APUs all the way in 2022. So imagine a year later you get the news that the chip in your system is not getting updates because it's architectures is too old. Lol.
I wish I couldn't forget AMD for this.
Ah, maybe you misunderstood. I use GPUs several generations old explicitly because I don't care for the latest shiny features.
My main computer runs a nvidia NVS5400m. It's been out of support for a very long time. I genuinely wasn't even aware that my AMD GPU was still in support at all.
Support affects me exactly none. I'll just use whatever default driver my Linux distro provides and never think of it. I'll get whatever kind of performance it provides. As long as it's above 30fps and textures aren't obviously broken, I don't care.
My main computer runs a nvidia NVS5400m. It's been out of support for a very long time. I genuinely wasn't even aware that my AMD GPU was still in support at all.
Support affects me exactly none. I'll just use whatever default driver my Linux distro provides and never think of it. I'll get whatever kind of performance it provides. As long as it's above 30fps and textures aren't obviously broken, I don't care.
Games dont magically stop working the moment after support ends, you know.
Old games not, but new games do need these things known as optimisations to run well, or eve at all, you know.
We give shit to Microsoft for W11 needing quite modern hardware even though the 10 year old W10 will still keep working after support stops, so why don't we hold HW manufacturers to a similar standard for supporting their shit longer than a year?
So forgive me for feeling stabbed in the back by being dumped on only a year after purchase.
We give shit to Microsoft for W11 needing quite modern hardware even though the 10 year old W10 will still keep working after support stops, so why don't we hold HW manufacturers to a similar standard for supporting their shit longer than a year?
So forgive me for feeling stabbed in the back by being dumped on only a year after purchase.
Ever since the 3000 gen of Nvidia GPUs have simply gone out of being a possibility for anyone in the 3rd world, they are way too expensive.
Add in the increase in energy costs... I don't know, really.
Add in the increase in energy costs... I don't know, really.
Yeah, CPUs too. Power use is unacceptable, desktops and big laptops turbo like mad to achieve really tiny performance boosts.
I've been experimenting with using fixed-point weights using the integer vector hardware (AVX) with pretty good results, though haven't built enormous models yet. The GPU may turn out to be a spur line.
It only matters if you give a fuck. You don't have to give a fuck to make an impact in the technology market.
Can I theoretically buy a 4090 and rent it out to some cloud?
Yes - https://vast.ai/ is one option.
Interesting, although the host requirements are a bit stringent for a casual. https://cloud.vast.ai/host/setup
Huh, if your electricity prices are low enough, it can quite worth it
We are working on it. If you have Windows Pro machine, see https://borg.games/setup
It is currently tailored to gaming, but ML workloads are coming shortly. Everything runs in an invisible ephemeral VM.
It is currently tailored to gaming, but ML workloads are coming shortly. Everything runs in an invisible ephemeral VM.
I’m not a cloud but I’d be happy to rent a 4090 from you now and then :D
A flood of giant H200s and MI300s is great and all, but what about personal accelerators?
The current state of local inference/finetuning is insane, where the hardware that makes any financial sense is ancient Nvidia GPUs, like the 3090 (2020) or the RTX 8000 (2018). Or maybe the rare used AMD Radeon Pro, if you can actually find one.
The only hope seems to be the rumored large APUs Intel/AMD APUs and maybe Intel Battlemage, since AMD is seemingly complicit in preserving the low VRAM status quo with Nvidia.
The current state of local inference/finetuning is insane, where the hardware that makes any financial sense is ancient Nvidia GPUs, like the 3090 (2020) or the RTX 8000 (2018). Or maybe the rare used AMD Radeon Pro, if you can actually find one.
The only hope seems to be the rumored large APUs Intel/AMD APUs and maybe Intel Battlemage, since AMD is seemingly complicit in preserving the low VRAM status quo with Nvidia.
VRAM is not the main constraint, is it? The computational power of any of the new graphical cards (beside from the highest end models, where the VRAM is the actual constraint, like RTX 4090s) is absurdly low on the stuff that actually matters (tensor cores, cuda cores, etc.). They are graphical cards, equipped with consumer grade VRAM (instead of HBM) and IMHO it will take a very big shift before we see them being used as real AI accelerators.
VRAM is everything.
The more VRAM you have, less aggressively you have to quantize models for inference, which in turn has huge speed/quality implications. You can run higher batch sizes, or draft models, or more caching, which increases efficiency. For LLMs specifically, you can load bigger models into VRAM in the first place. You can load more of a multimodal pipeline in VRAM without having to constantly swap everything out.
This is all 10x true for finetuning. Quality and speed is essentially determined by VRAM capacity, as long as you are not on truly ancient GPU like a P40 than't can't even do fp16.
As for architecture... TBH, many operations are heavily bandwidth bound these days. Sometimes a 3090 and a 4090 are essentially the same speed. And waiting a little longer for a finetune is no big deal vs not being able to do it at all, or doing it at low quality.
> They are graphical cards, equipped with consumer grade VRAM (instead of HBM) and IMHO it will take a very big shift before we see them being used as real AI accelerators
I think its important for users to break away from the cloud and APIs, and try to run stuff themself, lest we get locked into an OpenAI monopoly.
But setting that aside, running local is also extremely useful for prototyping and testing. You can see if something works without burning dollars every second you spend debugging on a big cloud instance. Even if that's affordable, just feeling like I am under the clock when debugging/optimizing is stressful to me.
The more VRAM you have, less aggressively you have to quantize models for inference, which in turn has huge speed/quality implications. You can run higher batch sizes, or draft models, or more caching, which increases efficiency. For LLMs specifically, you can load bigger models into VRAM in the first place. You can load more of a multimodal pipeline in VRAM without having to constantly swap everything out.
This is all 10x true for finetuning. Quality and speed is essentially determined by VRAM capacity, as long as you are not on truly ancient GPU like a P40 than't can't even do fp16.
As for architecture... TBH, many operations are heavily bandwidth bound these days. Sometimes a 3090 and a 4090 are essentially the same speed. And waiting a little longer for a finetune is no big deal vs not being able to do it at all, or doing it at low quality.
> They are graphical cards, equipped with consumer grade VRAM (instead of HBM) and IMHO it will take a very big shift before we see them being used as real AI accelerators
I think its important for users to break away from the cloud and APIs, and try to run stuff themself, lest we get locked into an OpenAI monopoly.
But setting that aside, running local is also extremely useful for prototyping and testing. You can see if something works without burning dollars every second you spend debugging on a big cloud instance. Even if that's affordable, just feeling like I am under the clock when debugging/optimizing is stressful to me.
Nah, vram is definitely the main constraint for most people trying to do local inference of LLMs. If you look at a lot of the local LLM communities, for people who aren't super interested in training, many people suggest the M2 Ultra or M3 Max with > 128GB of unified RAM just because it has so much memory. Again, you might not be able to train or fine tune as well, but the first step is just being able to keep the weights in memory. Inference isn't that computationally intensive relative to just how much RAM you need.
Yeah. I didn't mention them because the 64GB+ configs are very expensive, and I don't think you can even finetune on them.
They're not cheap, but they're not _that_ expensive compared to buying four NVLink'd Nvidia cards with a combined similar amount of VRAM. Plus you get a whole computer with it and you don't have to worry about casings and power supplies, etc. But yeah, you will be more compute constrained if you go down that route.
The price is similar to 2x RTX 8000s, or even A6000s, but yeah your point stands. Power efficiency is something too.
You run into immense pain the moment you venture outside of llama inference though.
You run into immense pain the moment you venture outside of llama inference though.
Well if you don't have enough VRAM to hold the whole model, you have to swap out to main RAM for every iteration, which for most home/local setups means once for every single token generated. So having enough is kind of a minimum.
Looks like you can get an MI60 with 32gb vram for ~$500 on ebay - if you're vram limited might be a good option for less than a 3090 seems to be going for, and seems supported by rocm (being an mi50 just with more hbm).
Not sure how well it will perform as it lacks some of the smaller data type acceleration available on newer cores, though.
Not sure how well it will perform as it lacks some of the smaller data type acceleration available on newer cores, though.
Its kinda funky, and AMD dropped the MI50 already. See this very interesting reddit discussion:
https://old.reddit.com/r/LocalLLaMA/comments/17vcsf9/somethi...
Another thing that jumps out:
> I also have a couple W6800's and they are actually as fast or faster than the MI100s with the same software...
That's insane. The MI100 should be so much faster than the W6800 (a ~6900XT) that its not even funny.
https://old.reddit.com/r/LocalLLaMA/comments/17vcsf9/somethi...
Another thing that jumps out:
> I also have a couple W6800's and they are actually as fast or faster than the MI100s with the same software...
That's insane. The MI100 should be so much faster than the W6800 (a ~6900XT) that its not even funny.
I grabbed a 32gb mi60 on a lark, because at the time it was ~$300. It works fine with recent ROCm etc, but it has the following potentially severe drawbacks:
1. It is not a recent GPU, so don't expect blazing inference speeds
2. It is designed for external forced air, so you will need to push air through it somehow
3. The one I got is actually not a normal card, I think it was a firmware testbed, and has no VBIOS and reports its product name as "TBD"
It does work and have 32gb of VRAM though.
1. It is not a recent GPU, so don't expect blazing inference speeds
2. It is designed for external forced air, so you will need to push air through it somehow
3. The one I got is actually not a normal card, I think it was a firmware testbed, and has no VBIOS and reports its product name as "TBD"
It does work and have 32gb of VRAM though.
AMD has a bad habit of dropping ROCm support for their devices after 4 years. Trying to work with an old dependency chain is something I wouldn't wish on an enemy. I don't know about the MI60 but much of the MI* line has already had support for compute under ROCm dropped.
It's probably not surprising that cheaper options sacrifice some level of software or hardware support - that's why they're cheaper. Software support is why the "pro" versions exist as separate (more expensive) SKUs, after all.
Of course you'll get a better service if you pay more. Expecting a high performance, easy, turnkey solution for a low price is expecting to have your cake and eat it. It's never happened on any other technology with this level of demand, and expecting it now seems naive at best. There's too much money (and hype) sloshing around in the sector, and limited supply.
Enthusiasts and experimenters have always traded their time for cost - if only because their time investment and learning (and maybe fun) is the whole point, not the end results of their experiments. If you expect financial returns due to your incredible idea, you should be looking for investors not old hardware.
Of course you'll get a better service if you pay more. Expecting a high performance, easy, turnkey solution for a low price is expecting to have your cake and eat it. It's never happened on any other technology with this level of demand, and expecting it now seems naive at best. There's too much money (and hype) sloshing around in the sector, and limited supply.
Enthusiasts and experimenters have always traded their time for cost - if only because their time investment and learning (and maybe fun) is the whole point, not the end results of their experiments. If you expect financial returns due to your incredible idea, you should be looking for investors not old hardware.
Your arguments are technically valid, and explain why AMD is doing this, but the eventual outcome is that for the (many!) non-corporate users of would-be-ML-hwardware you get better service by avoiding AMD/ROCm and buying nVidia hardware instead, because there you do get appropriate software support also for old cards, gaming cards, and old gaming cards And afterwards the AMD community is wondering why researchers and toolmakers have limited support their ecosystem - this is why.
AMD Instinct cards are very expensive Pro products.
The thing is, they are more geared towards HPC/Scientific computing. The only thing that really makes them appealing for ML is that they have somewhat more price depreciation than Nvidia cards and M1 Maxes, for the moment.
> Expecting a high performance, easy, turnkey solution for a low price is expecting to have your cake and eat it.
But I don't agree with this. This is not a normal "premium" hardware market, its a greedy pseudo monopoly that also dramatically supply constrained at the moment, until maybe next year (as the article points out). Business can pay a premium, but that's different than a lack of competition and supply.
The thing is, they are more geared towards HPC/Scientific computing. The only thing that really makes them appealing for ML is that they have somewhat more price depreciation than Nvidia cards and M1 Maxes, for the moment.
> Expecting a high performance, easy, turnkey solution for a low price is expecting to have your cake and eat it.
But I don't agree with this. This is not a normal "premium" hardware market, its a greedy pseudo monopoly that also dramatically supply constrained at the moment, until maybe next year (as the article points out). Business can pay a premium, but that's different than a lack of competition and supply.
Probably officially dropped, but gfx900-series works fine on ROCm 5.7 here, which actually goes back to vega10 (IIRC mi50/mi60 are vega20).
Apple Silicon
I wonder if we can afford the energy cost of the ever-increasing need to compile more and more information into ever larger AI baskets. Is there a point when the cost exceeds even what a Zuckerberg can pay?
I am hopeful developments like Mistral (a 7B model with the performance of ChatGPT for most of the tasks I've used it for) keeps happening. Seems like we can still compress a lot of knowledge.
Debian runs like a champ on my 8 year old CPU. When will gpus last that long?