A Really Big Computer(geohot.github.io)
geohot.github.io
A Really Big Computer
https://geohot.github.io//blog/jekyll/update/2023/08/08/a-really-big-computer.html
37 comments
Maybe I'm overly critical, but if fixing the Twitter search box is too hard, maybe buying solar (and batteries), fabbing chips, and building a giant data center to replicate the current generation of what another company already built is foolhardy.
I armchair-diagnose geohot with bipolar disorder.
Only a few weeks ago, he was writing about how he was slated to lose because of his low tolerance for bullshit.
I am not going to try to come up with an underlying reason but he's really poisoned his reputation in my book. How can you so emphatically jump into a problem space that (in hindsight) you're not an expert at and then manage to distance yourself so vigorously from your failure like a month later? What am I supposed to think when you're faced with a real world hundreds-of-millions-of-dollars project spanning tons of large, complex industries? How am I supposed to believe you'll have better success with this than a few thousand lines of JavaScript?
Have you ever met someone with actual bipolar disorder? They think that they are Gods and see symbols and signs and shit. Or sometime they go negative and thing there's a conspiracy targeting them or something.
They don't usually think they are gonna slowly over the next 5 years tape out a chip and build a solar powered datacenter with the company they raised $5M for...all part of the tiny corp master plan. I'll write it up better as it comes together better.
We have 417 preorders for tinyboxes btw. https://tinygrad.org
They don't usually think they are gonna slowly over the next 5 years tape out a chip and build a solar powered datacenter with the company they raised $5M for...all part of the tiny corp master plan. I'll write it up better as it comes together better.
We have 417 preorders for tinyboxes btw. https://tinygrad.org
I am someone with actual bipolar disorder. What you are saying is not true. Both mania and depression can involve psychosis and hallucination, but neither necessarily does. Bipolar II does not involve full-blown mania at all, merely hypomania. Hypomania can be easy to miss. People often find it benign or helpful, just a bit of extra energy and motivation.
I would never try to diagnose someone I've never met and disapprove of doing so.
I would never try to diagnose someone I've never met and disapprove of doing so.
You're not describing bipolar disorder, you're describing symptoms of schizophrenia. Schizophrenia (and other similar disorders) affects your ability to think and behave coherently. It's a detachment from reality.
Bipolar disorder is characterized by long (weeks or months) mood swings that range from very motivated and often positive to deeply depressed. When you're "up" you feel overconfident, impulsive, and restless. When you're "down," you are the opposite. There's no notion of false pattern recognition or conspiracy theories.
Not my place to pass judgement either way but don't just make things up when talking about mental health, it's not a good look.
Bipolar disorder is characterized by long (weeks or months) mood swings that range from very motivated and often positive to deeply depressed. When you're "up" you feel overconfident, impulsive, and restless. When you're "down," you are the opposite. There's no notion of false pattern recognition or conspiracy theories.
Not my place to pass judgement either way but don't just make things up when talking about mental health, it's not a good look.
You've gone too far in the other direction. Mania can be extreme, far beyond what an ordinary person could imagine "very motivated" to mean. Deeply irrational grandiose thinking is common. Either mania or depression can involve psychosis, hallucination, or catatonia.
> mania or depression can involve psychosis, hallucination, or catatonia
Sure. And diarrhea is a possible symptom of the flu, yet saying "I don't have the flu, I'm not shitting my pants" is just outright wrong. Saying that people with bipolar disorder think their a god and see symbols and signs is like saying that people having a stroke are characterized by vomiting and stumbling around.
Sure. And diarrhea is a possible symptom of the flu, yet saying "I don't have the flu, I'm not shitting my pants" is just outright wrong. Saying that people with bipolar disorder think their a god and see symbols and signs is like saying that people having a stroke are characterized by vomiting and stumbling around.
haters always gonna hate
He doesn't mention yield, which I hear is currently pretty bad on 3nm. And "just route around the bad silicon" might be easier said than done; widespread defects might create too many roadblocks to efficiently move data around your wafer-chip.
If that's not factored into his 3450 wafer estimate, it could be double that.
If that's not factored into his 3450 wafer estimate, it could be double that.
This was my first thought reading this article. I am not an expert and my knowledge is limited.
When he mentions the prices of the wafers, does it come with the bad yields or without?
When he mentions the prices of the wafers, does it come with the bad yields or without?
A knowledgable friend recently told me the wafer price is, you get what you get. This was in reference to a recent Apple story that TSMC was giving Apple a discount for bad dies, or something like that, on the 3nm node. I have no references.
Say what you want about him, but George is one of the few people I know that can actually go a mile wide and a mile deep into any field they choose. And he does most of his digging live on stream, so he makes an easy punching bag. Do I think he'll successfully raise a $400M round on a $2B valuation? No, something else will distract him before that - but I'm going to enjoy watching him (and pulling for him) regardless.
Feeling cute, may make up bullshit then change my mind.
Brainstorming can help identify limiting assumptions.
Mostly compute has piggy-backed off consumer-scale production (e.g., GPU's repurposed for crypto).
The suggestion is that an AI model can justify few-shot chip production.
His proposal is for development, i.e., to build the model, and depends mostly on such models being qualitatively better.
It seems more likely that chips would be built to offer model processing, instead of forcing users into a service (with its risk of confidentiality and IP leaks). To get GPT-100, you'd incorporate the chip into your device -- and then know for sure that nothing could leak. That eliminates the primary transaction cost for AI compute: the risk.
Which presents the question: does anyone know of research or companies working on such chip models?
Mostly compute has piggy-backed off consumer-scale production (e.g., GPU's repurposed for crypto).
The suggestion is that an AI model can justify few-shot chip production.
His proposal is for development, i.e., to build the model, and depends mostly on such models being qualitatively better.
It seems more likely that chips would be built to offer model processing, instead of forcing users into a service (with its risk of confidentiality and IP leaks). To get GPT-100, you'd incorporate the chip into your device -- and then know for sure that nothing could leak. That eliminates the primary transaction cost for AI compute: the risk.
Which presents the question: does anyone know of research or companies working on such chip models?
I will refer to this as the moment when LLM hype jumped the shark.
AI is not just LLMs.
As I keep mentioning on HN.
AI is real results + hype as opposed to crypto of only hype.
A computer of this size will be immensely useful for training Vision models very quickly, as well as other kinds of models.
Why only AI? Any branch of Science (all of them) that benefit from fast, parallel compute, will benefit from this.
It is extremely narrow view to view AI only as LLMs.
When I say "real use" I mean solving real world problems that existed before AI. And I don't even count generative AI of any shape or form.
As I keep mentioning on HN.
AI is real results + hype as opposed to crypto of only hype.
A computer of this size will be immensely useful for training Vision models very quickly, as well as other kinds of models.
Why only AI? Any branch of Science (all of them) that benefit from fast, parallel compute, will benefit from this.
It is extremely narrow view to view AI only as LLMs.
When I say "real use" I mean solving real world problems that existed before AI. And I don't even count generative AI of any shape or form.
My comment is about how this man is seriously entertaining a 2B$ valuation for this napkin math, with no experience in the 4/5 different industries his plan entails – and sharing no definite plans of what to do with it except "be able to train GPT4 in a day". It is about prevailing market forces in VC that let him believe this plan might actually work.
It doesn't have anything to do with crypto, science, or Real World Problems.
It doesn't have anything to do with crypto, science, or Real World Problems.
[deleted]
How would you distribute clock signal around that wafer-scale GPU? OR is he simply suggesting you buy the whole wafer and litho standard GPUs out of it? Apologies if this is a stupid question.
How does he cool those wafers?
“Put the servers somewhere cool.”
No joke. That’s his plan.
No joke. That’s his plan.
And...it's a good plan.
There's two cooling problems, pulling heat off the wafer (water cooled loop or immersion cooling, this is easy) and dumping the heat somewhere. The second part is why the place being cool matters, I imagine a big radiator on the roof. Could also dump the heat into a river if it's chill.
There's two cooling problems, pulling heat off the wafer (water cooled loop or immersion cooling, this is easy) and dumping the heat somewhere. The second part is why the place being cool matters, I imagine a big radiator on the roof. Could also dump the heat into a river if it's chill.
You and I have real different meters for "easy", George.
Everything is easy if you don't have to actually go through with your plan
I think the modern strategy is to have the whole thing under water.
Let's build it on the crest of a lunar crater. Wafers on the shade side and solar panels on the sunny one. “Just” add some $Bs to send the stuff up in the space.
Shaded or not, cooling is actually a problem in space because radiating heat in to a "vacuum" is hard. Even a warm atmosphere is easier to transfer heat to than almost nothing.
Now a ground source heat pump could be interesting as long as it's not too hard to drill for the plumbing.
Now a ground source heat pump could be interesting as long as it's not too hard to drill for the plumbing.
How do these goals set out the needed ratios of memory to compute to interconnect bandwidth?
An ideal machine designed to train GPT4 in a day is likely very different to the ideal machine to train 50 GPT4s as once over a few weeks, which is very different from the ideal machine to train a model 100x bigger than GPT4 (perhaps the most interesting).
An ideal machine designed to train GPT4 in a day is likely very different to the ideal machine to train 50 GPT4s as once over a few weeks, which is very different from the ideal machine to train a model 100x bigger than GPT4 (perhaps the most interesting).
Not sure about that computer plan but I do enjoy his brand of entertainment.
Also really hoping he makes more progress on amd ml
Also really hoping he makes more progress on amd ml
Where is the budget line item for defending it against bombing by crazed anti-ai doomsday cultists?
[deleted]
Wyoming seems like a good place to build such a thing.
How is the sun there for the solar part?
~220 sunny days per year apparently. Plus there's plenty of space for wind energy.
NCAR has a supercomputing center there: https://en.m.wikipedia.org/wiki/NCAR-Wyoming_Supercomputing_...
NCAR has a supercomputing center there: https://en.m.wikipedia.org/wiki/NCAR-Wyoming_Supercomputing_...
This is really cool. Godspeed!