The AMD tinybox is on hold until we can build and run the firmware on our GPUs(twitter.com)
twitter.com
The AMD tinybox is on hold until we can build and run the firmware on our GPUs
https://twitter.com/__tinygrad__/status/1770151484363354195
66 comments
I genuinely can't tell if George Hotz is genius or, mid and totally full of himself.
If anything his weakness is being too optimistic. Oh no, he started a self-driving company and didn't achieve level 5. Oh no, he set out to fix AMD GPUs and ran into roadblocks that aren't his fault. So mid!
Thats not really a scale that applies to people. People like Hotz are really fascinating. There is a lot of similarity between him and Musk, and the question of "how can a person be so smart in one way but so absolutely dumb in other ways" is super interesting to me.
Both him and Musk are quite intelligent in their respective areas, there shouldn't be any doubt about that. But there is a big issue with intelligent people where they often end up on completely wrong tracks. There are some studies that show its actually easier to convince smarter people of wrong information than ordinary people.
Generally, the way someone gets very smart in a certain area is through repeated exposure to material. The only way to have that sort of dedication is that you have intrinsic desire to do certain things because it fulfills some purpose in your life, which basically means that you have an ideology attached to that desire. That ideology can be any number of things, from wanting to help humanity, to a more selfish one of just viewing yourself as better than other people because you can accomplish stuff they cant.
The thing is, for a very smart person who has a personality like that, you can take a narrative that fits within that ideology, and they are almost guaranteed to believe it, whether its actually true or not. Because them denying it would mean that their ideology isn't correct, which means that their entire life has been wasted on wrong things - and no person would ever self destruct like that.
Its for this reason Musk believes in all the right wing conspiracy theories despite him having multiple successful startups. To him, democrats are bad (because of the exclusion of Tesla from environmental fund under Biden, which admittedly was pretty shitty), and then everything anti democrat must be correct - you can trace the start of his descent into madness right up to that event.
Same thing with Hotz. Super smart dude, but he managed to pick a fight with Sony and basically lost, which would leave a bad taste in anyone's mouth, then he read Unqualified Reservations, which perfectly explained the world in such a way where he is the good guy, and the rest follows. As such, he things literature like Atlas Shrugged is actually very good, aligns heavily with Musk because Musk is anti leftism, e.t.c. At Comma he was heavily against WFH because Musk was against WFH (for very, very stupid reasons), prided himself on being extremely selective with hiring only people that are both smart and are driven, but now with Tiny Grad, he has come to sort realize that you sort of need to hire remote people if you want to have talent, BUT instead of actually hiring people, he only has a few employees and the rest of the work is on a bounty system (being claimed as the future of employment).
One could make an argument that when looking at successful tech companies that "won" at a sector, like Apple with consumer electronics, Amazon with online shopping, none of them did it like Comma or Tiny Grad. I personally want nothing more for Hotz companies to succeed, which would truly be disruption at its core, but I don't think they will. It seems that you have to essentially engage with all people as humans and appeal to them, but when you view humanity from a viewpoint of "From each according to his ability, to each according to his needs", and tweet about it, its pretty clear that you dislike a large portion of humanity. And of course, engaging with people like that means that you have to appeal to the "professional managerial class" which goes strictly against his ideology.
For Comma Ai, that would look like playing nice with investors, getting more money, hiring a wide range of talent local and remote, which would put them on a stage where car companies take them seriously and want to work with them, at which point instead of dedicating resources to making hardware they can be actually making the software a shitload better, which is core to their mission of solving AI. And yes, there would have to be a bunch of bullshit that they would have to sift through, but thats just unavoidable fact of life.
You can view Tinycorp in the same light.
Both him and Musk are quite intelligent in their respective areas, there shouldn't be any doubt about that. But there is a big issue with intelligent people where they often end up on completely wrong tracks. There are some studies that show its actually easier to convince smarter people of wrong information than ordinary people.
Generally, the way someone gets very smart in a certain area is through repeated exposure to material. The only way to have that sort of dedication is that you have intrinsic desire to do certain things because it fulfills some purpose in your life, which basically means that you have an ideology attached to that desire. That ideology can be any number of things, from wanting to help humanity, to a more selfish one of just viewing yourself as better than other people because you can accomplish stuff they cant.
The thing is, for a very smart person who has a personality like that, you can take a narrative that fits within that ideology, and they are almost guaranteed to believe it, whether its actually true or not. Because them denying it would mean that their ideology isn't correct, which means that their entire life has been wasted on wrong things - and no person would ever self destruct like that.
Its for this reason Musk believes in all the right wing conspiracy theories despite him having multiple successful startups. To him, democrats are bad (because of the exclusion of Tesla from environmental fund under Biden, which admittedly was pretty shitty), and then everything anti democrat must be correct - you can trace the start of his descent into madness right up to that event.
Same thing with Hotz. Super smart dude, but he managed to pick a fight with Sony and basically lost, which would leave a bad taste in anyone's mouth, then he read Unqualified Reservations, which perfectly explained the world in such a way where he is the good guy, and the rest follows. As such, he things literature like Atlas Shrugged is actually very good, aligns heavily with Musk because Musk is anti leftism, e.t.c. At Comma he was heavily against WFH because Musk was against WFH (for very, very stupid reasons), prided himself on being extremely selective with hiring only people that are both smart and are driven, but now with Tiny Grad, he has come to sort realize that you sort of need to hire remote people if you want to have talent, BUT instead of actually hiring people, he only has a few employees and the rest of the work is on a bounty system (being claimed as the future of employment).
One could make an argument that when looking at successful tech companies that "won" at a sector, like Apple with consumer electronics, Amazon with online shopping, none of them did it like Comma or Tiny Grad. I personally want nothing more for Hotz companies to succeed, which would truly be disruption at its core, but I don't think they will. It seems that you have to essentially engage with all people as humans and appeal to them, but when you view humanity from a viewpoint of "From each according to his ability, to each according to his needs", and tweet about it, its pretty clear that you dislike a large portion of humanity. And of course, engaging with people like that means that you have to appeal to the "professional managerial class" which goes strictly against his ideology.
For Comma Ai, that would look like playing nice with investors, getting more money, hiring a wide range of talent local and remote, which would put them on a stage where car companies take them seriously and want to work with them, at which point instead of dedicating resources to making hardware they can be actually making the software a shitload better, which is core to their mission of solving AI. And yes, there would have to be a bunch of bullshit that they would have to sift through, but thats just unavoidable fact of life.
You can view Tinycorp in the same light.
Comma.ai is profitable, and as long as TinyCorp is profitable all else will follow. AMD / NVidia / Intel these are all means to a commoditised petaflops. And for that too happen starting with a decent affordable hardware was a good choice, turns out they are affordable because they dont have a good handle on their hardware or software. And the way the Red organisation is structured it seems things are not going to become better, its better to move to the winning Team. (Green)
There is no compelling evidence that Musk is smart. That’s not being a “hater” it’s just reality.
In any timeframe there are people who rise by luck with timing and opportunity. Musk is that guy. He bet on EVs just before massive subsidies became available for them and then got lucky as the incumbent companies mostly did nothing.
He bet on space just before the US lost its only human capable system and again hovered by taxpayer funds and contracts.
All again in an era of free money post 2008.
As for the products? Teslas are some of them worst built cars ever made and Tesla produced vehicles for segments Americans don’t want. Minivans, sedans and limousines.
Space X meanwhile makes rockets which are a step back from the Shuttle in terms of capability and which will now require 6-8 launches what Apollo could do in 1. All built on lies about building a Mars capable craft which will carry 100 people - a feat Starship will never attain as it’s too small.
Musk has had an incredible string of luck but makes awful decisions, has no design sense and pretends to be a genius.
He’s very average indeed. His politics and attitudes are like a lot of gamers because that’s all he is.
In any timeframe there are people who rise by luck with timing and opportunity. Musk is that guy. He bet on EVs just before massive subsidies became available for them and then got lucky as the incumbent companies mostly did nothing.
He bet on space just before the US lost its only human capable system and again hovered by taxpayer funds and contracts.
All again in an era of free money post 2008.
As for the products? Teslas are some of them worst built cars ever made and Tesla produced vehicles for segments Americans don’t want. Minivans, sedans and limousines.
Space X meanwhile makes rockets which are a step back from the Shuttle in terms of capability and which will now require 6-8 launches what Apollo could do in 1. All built on lies about building a Mars capable craft which will carry 100 people - a feat Starship will never attain as it’s too small.
Musk has had an incredible string of luck but makes awful decisions, has no design sense and pretends to be a genius.
He’s very average indeed. His politics and attitudes are like a lot of gamers because that’s all he is.
Ohh definitely Elon is smart, he developed a space shooter game and published it in a computer magazine at an early age, so clearly he taught himself 6502 assembly, he developed zip2 city search, it had maps using the Navteq GIS database it was a time when google maps were not developed. All of this was written in Java by a single person at the time.
You are very much being a hater.
> Musk is that guy. He bet on EVs just before massive subsidies became available for them and then got lucky as the incumbent companies mostly did nothing.
Execution of an idea is 10000% times harder than actually having an idea. Furthermore, even if you consider just the idea of putting money into something that makes you money, your "logic" of getting lucky can be used to discredit most anyone who predicted anything and made money off of it, saying that they just got lucky. When in reality, that prediction was probably made through a bunch of research and reasoning that nobody else did.
>As for the products? Teslas are some of them worst built cars ever made and Tesla produced vehicles for segments Americans don’t want.
Model Y is literally the best selling car in the world, which is more impressive given the general anti Musk sentiment, especially in more socially progressive countries. You can't honestly think that saying "well all these people want this car, but they are all wrong because the car is horribly built", makes you sound anywhere near coherent.
>Space X meanwhile makes rockets which are a step back from the Shuttle in terms of capability and which will now require 6-8 launches what Apollo could do in 1.
And yet, it still has a shitload of contracts, including a pretty big one from NASA.
>Musk has had an incredible string of luck but makes awful decisions, has no design sense and pretends to be a genius.
And yet, Twitter is still alive and kicking, Space X is doing really good, Tesla is doing good.
The problem with people like you posting your baseless opinions like this is you make it harder to actually criticize people like Musk, because it gives credence to the idea that people/media just want to push a negative narrative against them, and to anyone on the fence, they are more likely to see all the dumb shit he does with a skeptical view, believing that the stories are vastly exaggerated.
> Musk is that guy. He bet on EVs just before massive subsidies became available for them and then got lucky as the incumbent companies mostly did nothing.
Execution of an idea is 10000% times harder than actually having an idea. Furthermore, even if you consider just the idea of putting money into something that makes you money, your "logic" of getting lucky can be used to discredit most anyone who predicted anything and made money off of it, saying that they just got lucky. When in reality, that prediction was probably made through a bunch of research and reasoning that nobody else did.
>As for the products? Teslas are some of them worst built cars ever made and Tesla produced vehicles for segments Americans don’t want.
Model Y is literally the best selling car in the world, which is more impressive given the general anti Musk sentiment, especially in more socially progressive countries. You can't honestly think that saying "well all these people want this car, but they are all wrong because the car is horribly built", makes you sound anywhere near coherent.
>Space X meanwhile makes rockets which are a step back from the Shuttle in terms of capability and which will now require 6-8 launches what Apollo could do in 1.
And yet, it still has a shitload of contracts, including a pretty big one from NASA.
>Musk has had an incredible string of luck but makes awful decisions, has no design sense and pretends to be a genius.
And yet, Twitter is still alive and kicking, Space X is doing really good, Tesla is doing good.
The problem with people like you posting your baseless opinions like this is you make it harder to actually criticize people like Musk, because it gives credence to the idea that people/media just want to push a negative narrative against them, and to anyone on the fence, they are more likely to see all the dumb shit he does with a skeptical view, believing that the stories are vastly exaggerated.
Getting from scratch to new businesses, and the most complicated ones and succeeding is no luck. Rocket science and automotive are very complicated in their own ways, and competing against businesses with 100+ years of IP and experience.... If the Space shuttle and Apollo were so good they would still be in production, the Concord as well. I guess it has something to do with the reliability and % to have a crew survive the launch and also the cost. I kinda respect his legacy more than all those social networks/search engine/marketplace gurus. But I do hate Musk's personality and I'm definitely not a fanboy,but still, respect is due where it's earned.
Are there books or literature which are related to this? Or are these all your thoughts? I'm curious because I want to learn more about it
George Hotz expressed his thoughts on this comment on his youtube channel
For those wondering:
https://www.youtube.com/watch?v=6fFUfyT-EyA&t=1160s
Hah. Thats 2 comments of mine on HN that got notices by industry people.
IMO, if he, Elon Musk, and Steven Bonnell/Destiny sat down together and had like a 4 hour long discussion with the prefaces to explore ideas, I feel like thats an easy 10 million view Youtube video. Musk has the knowledge of how industry works, George is a lot more philosophically grounded in tech, and Steven is the best representation of the humanities aspect.
IMO, if he, Elon Musk, and Steven Bonnell/Destiny sat down together and had like a 4 hour long discussion with the prefaces to explore ideas, I feel like thats an easy 10 million view Youtube video. Musk has the knowledge of how industry works, George is a lot more philosophically grounded in tech, and Steven is the best representation of the humanities aspect.
There is a texture between hard and soft
Squidgeware.
That's rough
Something that isn't really talked about is that Tiny Corp has been kind of working against AMD's interests. Tinybox is/was about replacing MI300s with much cheaper 7900 XTXs. I'm not terribly surprised to discover that AMD is not investing in ROCm on consumer cards (which are a different architecture) even though they're technically supported.
No one would be buying A100's if CUDA hadn't been supported on all desktop Nvidia cards for years. Desktop cards provide an accessible on ramp to the ecosystem. PhD grads with boxes of desktop cards turn into the purchasers of data center chips.
Nvidia was forced to do that because it was so early, they had no customers except PhDs. I see no evidence AMD wants to do that right now, and instead focusing on extracting value from deep pocket enterprise customers.
The way things go, I think the AMD consumer card experience will only get better once AMD manage to gimp consumer cards’ ML throughput or RAM.
The way things go, I think the AMD consumer card experience will only get better once AMD manage to gimp consumer cards’ ML throughput or RAM.
If you ask ML engineers now 100% will have started on desktop cards. Trying to squeeze the most money out of everyone just makes the ecosystem fail.
Look at IBM, they sell technologies that no one uses until they get hired to work on them specifically. It's profitable enough but it's not a way to grow the ecosystem/market.
Look at IBM, they sell technologies that no one uses until they get hired to work on them specifically. It's profitable enough but it's not a way to grow the ecosystem/market.
I think you are preaching to the choir, and AMD is not listening.
AMD would be selling 48GB 7900s or AI-only W7900s if they really wanted a consumer card ramp.
They don't. Not because they can't (they literally prevent OEMs from doing so, who would double up VRAM in a heartbeat without AMD lifting a finger), but because AMD doesn't want that.
AMD would be selling 48GB 7900s or AI-only W7900s if they really wanted a consumer card ramp.
They don't. Not because they can't (they literally prevent OEMs from doing so, who would double up VRAM in a heartbeat without AMD lifting a finger), but because AMD doesn't want that.
> The way things go, I think the AMD consumer card experience will only get better once AMD manage to gimp consumer cards’ ML throughput or RAM.
Que? Making things worse will make things better?
Que? Making things worse will make things better?
They'll let you do some things on a consumer card as soon as they can make sure that you can't effectively use the consumer card in place of an enterprise card.
so, artificial market segmentation?
...that's exactly how you get disruptive competition.
...that's exactly how you get disruptive competition.
It's worth noting this is a game nvidia has already been playing for a long time by intentionally limiting the performance of higher precision floating point operations on consumer cards.
But is this going to blow over in a few days? Again?
I can certainly appreciate frustration with the AMD stack, but be blunt, I was not impressed with Hotz's YouTube rant from before.[1] It didn't give the impression of a stable framework, and this doesn't either.
Also (at least from the end user llm inference side of things) ROCm is not nearly as unusable as it used to be. We would certainly be renting MI300s over A100s (or even H100s) if we could get any, and we use a number of different inference backends.
1: https://news.ycombinator.com/item?id=36193625
I can certainly appreciate frustration with the AMD stack, but be blunt, I was not impressed with Hotz's YouTube rant from before.[1] It didn't give the impression of a stable framework, and this doesn't either.
Also (at least from the end user llm inference side of things) ROCm is not nearly as unusable as it used to be. We would certainly be renting MI300s over A100s (or even H100s) if we could get any, and we use a number of different inference backends.
1: https://news.ycombinator.com/item?id=36193625
he's always been nails on a chalkboard for me. rant, whine, cry, repeat. reminds me of larry david with a CS degree.
I never followed Hotz, so perhaps I missed something cool. But I never understood the hype myself.
[deleted]
The post PC era of big expensive hardware you can't even buy if you do have the money is upon us, and mercy this is a scary scary time in computing for me/us.
I was talking about renting!
There are some boutique hosts like Hot Aisle serving MI300s (who I really should reach out to), but for the immediate future our little startup is stuck with the big cloud providers. No MI300s for us mere mortals, not even to rent.
There are some boutique hosts like Hot Aisle serving MI300s (who I really should reach out to), but for the immediate future our little startup is stuck with the big cloud providers. No MI300s for us mere mortals, not even to rent.
Besides the datacenter stuff, what exactly are people struggling to source these days? The 30/40-series prices should be fairly stable relative to the MSRP these days.
Used 3090 prices are absolutely outrageous.
And the 4090 MSRP was outrageous to begin with.
And the 4090 MSRP was outrageous to begin with.
Yet another thing to go back to 1990's ecosystem. We already have timesharing again, now we have the prices as well.
Identifying MES & CP as the barriers to an AMD tinybox and letting the community know is a huge service and is still great engineering even if they decide it's an immoveable barrier and walk away.
This was destined to be a hard problem. Surprised to see they are giving up so easily.
Unfortunate but understandable. AMD needs to move faster on software support in AI space if they want any of that money.
What advantage will remain if tensorflow and PyTorch works on AMD cards? https://www.xda-developers.com/nvidia-cuda-amd-zluda/
https://pytorch.org/ has a rocm support . This doesn’t make the outlook on this company very good …
https://pytorch.org/ has a rocm support . This doesn’t make the outlook on this company very good …
given his background in the jailbreak community, I look forward to him jailbreaking the AMD GPU firmware load process :-D
Jailbreaking is not the problem. The problem is reverse engineering a large firmware that operates on HW they have no docs for and fixing the firmware (ie. doing it better that original manufacturer that has access to everything).
The job of firmware is so close to the hw that it's nearly impossible to decode. You need to decode a custom CPU instructions for a IP block the microcode is running on (it's custom, no ARM/RISC/MIPS..).
After that you have to decode what firmware actually does. It writes something to this.... What does it mean? It's completely opaque number written to a opaque memory... cache control? Delay?
And you do that why? So you can ship tinybox (i.e. cheap consumer GPU). Let's says you succeed. Do the same thing Next gen will be similar challenge, except firmware will be better locked, because AMD will want consumer HW to be segregated from data center GPU, the same way NVidia does.
The task itself is basically impossible and waste of time. There is the reason why NVidia driver driver for Linux was used basically only to install official driver.
The job of firmware is so close to the hw that it's nearly impossible to decode. You need to decode a custom CPU instructions for a IP block the microcode is running on (it's custom, no ARM/RISC/MIPS..).
After that you have to decode what firmware actually does. It writes something to this.... What does it mean? It's completely opaque number written to a opaque memory... cache control? Delay?
And you do that why? So you can ship tinybox (i.e. cheap consumer GPU). Let's says you succeed. Do the same thing Next gen will be similar challenge, except firmware will be better locked, because AMD will want consumer HW to be segregated from data center GPU, the same way NVidia does.
The task itself is basically impossible and waste of time. There is the reason why NVidia driver driver for Linux was used basically only to install official driver.
relevant to his tenstorrent mention: https://news.ycombinator.com/item?id=39658787
and an bunch more https://hn.algolia.com/?q=tenstorrent
and an bunch more https://hn.algolia.com/?q=tenstorrent
Does Nvidia have any real competition that’s already shipped?
Why would there be? The writing was on the wall that this was important 10 years ago and no one moved on it, in fact AMD and Apple both fumbled OpenCL then proceeded to waste several more years after that.
I know some people don't like Nvidia but like their competition had their opportunity and what needed to be done spelt out to them and did nothing.
I know some people don't like Nvidia but like their competition had their opportunity and what needed to be done spelt out to them and did nothing.
That is the thing, they can allow themselves to have all that proprietary stuff, because Intel, AMD just can't get their act together.
Even the whole OpenCL versus CUDA, they had years to ship something that was great tooling alternative, instead they did everything but that.
Even the whole OpenCL versus CUDA, they had years to ship something that was great tooling alternative, instead they did everything but that.
The MI300 is the best accelerator you can buy, for many current workloads.
It's technically way more advanced. Not as outrageously priced as an H100 either.
It's technically way more advanced. Not as outrageously priced as an H100 either.
TBH I don't know what they were counting on. 4090 has almost 3x BF16 tensor ops/s vs 7900XTX. So you can just buy a regular PC with 2x4090 for half the price and have basically the same training performance with much less headache.
> We are also (sadly) exploring a 6x4090 box.
$12k for 6 4090 for 144GB GDDR vs $20k H100 PCIe 80GB HBM2 (price likely dropping later this year when B100 is released).
And H100 has a lot of features like async and loading directly to tensor cores not present in consumer cards.
I want to root for the little guy, but it seems the AI hardware landscape will be Nvidia for the next few years. And us GPU poors accessing it via cloud (shudders).
$12k for 6 4090 for 144GB GDDR vs $20k H100 PCIe 80GB HBM2 (price likely dropping later this year when B100 is released).
And H100 has a lot of features like async and loading directly to tensor cores not present in consumer cards.
I want to root for the little guy, but it seems the AI hardware landscape will be Nvidia for the next few years. And us GPU poors accessing it via cloud (shudders).
what's so tiny about a box that has 6 gpus?
It's not $150K.
Reminds me of the "super mini computer" that was still the size of two fridges.
Only makes sense if you understand the niche market it competes in.
https://en.wikipedia.org/wiki/Superminicomputer
Only makes sense if you understand the niche market it competes in.
https://en.wikipedia.org/wiki/Superminicomputer
Frequently people say on HN you should use AMD but looks like it's not going to work. I am glad to stick to straightforward Nvidia GPU / Epyc CPU stack. Don't want to innovate for this.
It's not either or, you can use different vendors for different tasks.
tinygrad isn't in the realm of production ready though, AFAIK.
tinygrad isn't in the realm of production ready though, AFAIK.
Yes but the same could be said about rocm.
there is no room for innovation in high performance tensor evaluation, cost-performance, or usability. we're just done.
There is room, but if you are not working on building the framework, it's not worth building the framework. The time cost is high.
I'm hopeful for tinygrad as a piece of software, but I'm skeptical about the future of the tinybox if they keep waffling on the hardware so much.
[1] https://geohot.github.io/blog/jekyll/update/2023/05/24/the-t...