Tesla AI Day Live Stream [video](youtube.com)
youtube.com
Tesla AI Day Live Stream [video]
https://www.youtube.com/watch?v=j0z4FweCy4M&
65 comments
Previous discussion from 13 hours ago: https://news.ycombinator.com/item?id=28240882
6d6b73(1)
Absolutely blown away by dojo and their chip design. It seems that they are on the way to building the most powerful machine learning supercomputer in the world in house.
Where do these chips get made? TSMC?
I would love to know how their new chip compares with the FSD chip in terms of power usage per teraflop.
Also, it seems that what they call "vector space" is really a 2d plane, so how do they deal with 3d scenarios such as parking garages and complex overpasses. They are rare, but they do exist.
The news will of course be all about the humanoid robot and the somewhat silly dancer, giving ammo to the detractors...
Where do these chips get made? TSMC?
I would love to know how their new chip compares with the FSD chip in terms of power usage per teraflop.
Also, it seems that what they call "vector space" is really a 2d plane, so how do they deal with 3d scenarios such as parking garages and complex overpasses. They are rare, but they do exist.
The news will of course be all about the humanoid robot and the somewhat silly dancer, giving ammo to the detractors...
ditto. Was definitely the most impressive part of the presentation (IMO).
Also, did I catch it right? One of those plates is 15kW of power? That's some crazy heat dissipation!
Also, did I catch it right? One of those plates is 15kW of power? That's some crazy heat dissipation!
That was my question too. It seems they have a deal with Samsung, atleast for the FSD computer chips[1].
[1] https://electrek.co/2021/05/03/how-tesla-pivoted-avoid-globa....
[1] https://electrek.co/2021/05/03/how-tesla-pivoted-avoid-globa....
Blown away by a prototype clamped to a bench, eh? If this thing existed they'd have MLPerf benchmarks to show.
Yup, stick around for the QA. During the presentation it sounded like it was practically ready. Someone asked about memory locality when actually putting them together and the infra guy started saying it's a really hard problem. Finally Elon said "This will be ready next year"
The whole presentation (not just dojo) felt like the kind of presentation you give internally to a VP when you haven't been making much progress. Flood them with challenges and implementation detail decisions they can't follow and don't talk about results.
The whole presentation (not just dojo) felt like the kind of presentation you give internally to a VP when you haven't been making much progress. Flood them with challenges and implementation detail decisions they can't follow and don't talk about results.
> The whole presentation felt like the kind of presentation you give internally to a VP when you haven't been making much progress. Flood them with challenges and implementation detail decisions they can't follow and don't talk about results.
That hurts me a bit, because I feel like I've done that sometimes. But that's what makes it easy for me to pick up on, when other people do that sort of stuff.
Good presentations are more direct. Ex: Here's our ResNet-50 benchmark numbers. Or at very least, "We didn't run ResNet50, because we don't think its realistic. Instead, we ran this other benchmark along with NVidia A100 GPUs and are proving that our system is faster "
----------
You need to prove your system has a benchmark score worth considering before diving into architectural details... at least if you're trying to attract engineering talent to your project.
That hurts me a bit, because I feel like I've done that sometimes. But that's what makes it easy for me to pick up on, when other people do that sort of stuff.
Good presentations are more direct. Ex: Here's our ResNet-50 benchmark numbers. Or at very least, "We didn't run ResNet50, because we don't think its realistic. Instead, we ran this other benchmark along with NVidia A100 GPUs and are proving that our system is faster "
----------
You need to prove your system has a benchmark score worth considering before diving into architectural details... at least if you're trying to attract engineering talent to your project.
Other than being hypothetical vaporware, how is it different from a cabinet full of google TPUs?
Random thoughts:
I thought the fact they have 1,000 data labelers was interesting.
I'm unsure about their dojo efforts. Seems cool, seems better than Nvidia but not 10x better (hard to tell it's not apples to apples). For example their dojo cluster us supposed to have over an exaflop vs a A100 superpod at 700pb (seems like I/O is over 2x better which I understand conceptually why it matters but not real world implications). By the time this gets production ready (next year) seems possible Nvidia will have a replacement that is about as good or better.
Network architecture was not using video before, it was using 2-3 frames, which seemed obviously not going to work so seems better now.
The updated network architecture plus the fact they need to train with a supercomputer to me means they will almost certainly need a new inference chip. They didn't announce a v3 of their FSD computer, but that is probably the physical component blocking FSD rollout.
My take on timeline for actual compete everywhere self driving (robotaxi) seems to be: -dojo isn't even in production until next year, so no earlier than that. -FSD chip v3 probably not going to be design finalized until they use some dojo models in it. That plus chip shortages means it's unlikely to happen at scale until 2023.
So I think this presentation was to make it clear the large about of progress they are making (very impressive), but IMO this was also a subtle way of making 2023 the new target FSD date.
The alternative is they don't think they need dojo to get FSD, and if that is the case why bother building a fully integrated chip manufacturing process?
I thought the fact they have 1,000 data labelers was interesting.
I'm unsure about their dojo efforts. Seems cool, seems better than Nvidia but not 10x better (hard to tell it's not apples to apples). For example their dojo cluster us supposed to have over an exaflop vs a A100 superpod at 700pb (seems like I/O is over 2x better which I understand conceptually why it matters but not real world implications). By the time this gets production ready (next year) seems possible Nvidia will have a replacement that is about as good or better.
Network architecture was not using video before, it was using 2-3 frames, which seemed obviously not going to work so seems better now.
The updated network architecture plus the fact they need to train with a supercomputer to me means they will almost certainly need a new inference chip. They didn't announce a v3 of their FSD computer, but that is probably the physical component blocking FSD rollout.
My take on timeline for actual compete everywhere self driving (robotaxi) seems to be: -dojo isn't even in production until next year, so no earlier than that. -FSD chip v3 probably not going to be design finalized until they use some dojo models in it. That plus chip shortages means it's unlikely to happen at scale until 2023.
So I think this presentation was to make it clear the large about of progress they are making (very impressive), but IMO this was also a subtle way of making 2023 the new target FSD date.
The alternative is they don't think they need dojo to get FSD, and if that is the case why bother building a fully integrated chip manufacturing process?
They are already deploying the kinds of models talked about here in their FSD beta software (along with regular FSD/AP software) and they run just fine on current FSD hardware in the cars with acceptable latency. Just because you need a monstrous supercomputer to train these models doesn't mean you cannot run inference on them on much more light-weight devices. Some reasons being:
1. Training requires them to make use of ridiculous amount of data which requires large GPU compute + RAM and super high bandwidth to keep the GPUs fed at all times. Backpropagation requires a LOT of bookkeeping that adds on a lot of RAM requirement at train-time.
2. Inference is a lot easier. You just need to worry about keeping up with your 8 cameras and running them through your network. You also aren't calculating any gradients and dealing with needing to do any of the bookkeeping for backpropagation so you need a LOT less RAM
3. There are a ton of tricks you can do to make a trained network a lot more efficient for inference such as lower precision models (int8 or in their case, CFP8), network pruning, compiler tricks, etc.
It is quite impressive that they've managed to fit all this into their current V3 hardware. That being said, they are definitely starting to really hit the limits of V3 and in the long-run will likely need to upgrade the FSD hardware on newer cars. They are probably trying to avoid having to provide a retrofit for all cars in the fleet as that would be prohibitively expensive.
1. Training requires them to make use of ridiculous amount of data which requires large GPU compute + RAM and super high bandwidth to keep the GPUs fed at all times. Backpropagation requires a LOT of bookkeeping that adds on a lot of RAM requirement at train-time.
2. Inference is a lot easier. You just need to worry about keeping up with your 8 cameras and running them through your network. You also aren't calculating any gradients and dealing with needing to do any of the bookkeeping for backpropagation so you need a LOT less RAM
3. There are a ton of tricks you can do to make a trained network a lot more efficient for inference such as lower precision models (int8 or in their case, CFP8), network pruning, compiler tricks, etc.
It is quite impressive that they've managed to fit all this into their current V3 hardware. That being said, they are definitely starting to really hit the limits of V3 and in the long-run will likely need to upgrade the FSD hardware on newer cars. They are probably trying to avoid having to provide a retrofit for all cars in the fleet as that would be prohibitively expensive.
V3 is what’s currently shipping right now. Elon called it FSD 1 and said FSD 2 might come with Cybertruck.
The newly announced Tesla humanoid bot is around 2:05:11. Expected prototype in 2022.
The demo was an obvious troll but it looks pretty exciting. I'd rather have resources put into actual innovation than learning how to increase the likelihood someone will click a button by 0.01%
https://youtu.be/j0z4FweCy4M?t=7511
The demo was an obvious troll but it looks pretty exciting. I'd rather have resources put into actual innovation than learning how to increase the likelihood someone will click a button by 0.01%
https://youtu.be/j0z4FweCy4M?t=7511
I had the tv on last night, and woke up thinking Musk made a huge advancement in robotics.
I've been watching their presentations from time to time. The progress they maid within 1-2 years is incredible. They clearly focused a lot and betting a lot on their approach. Wouldn't be surprised if they'll become a real waymo competitor in 1.5 year. Saying"competitor" I mean real application of robotaxi but with major some issues. Waymo's approach still failing a nose cones on the road for example.
How does Tesla's tech compare with Waymo?
I just can't imagine how Waymo or any other company can compete with the first-principles full stack engineering and scaling of Tesla, which touches virtually every important dimension of consideration. And that is today, not even a year from now.
The foundation Tesla has built and shown on AI day will progressively increase their lead in the years to come, I just don't see how anyone can compete with that level of forward thinking. To me it just seems obvious, so what am I missing about Waymo?
I just can't imagine how Waymo or any other company can compete with the first-principles full stack engineering and scaling of Tesla, which touches virtually every important dimension of consideration. And that is today, not even a year from now.
The foundation Tesla has built and shown on AI day will progressively increase their lead in the years to come, I just don't see how anyone can compete with that level of forward thinking. To me it just seems obvious, so what am I missing about Waymo?
I've been told the consensus on HN is that Waymo is far ahead of Tesla in terms of tech. See this post & discussion from yesterday: https://news.ycombinator.com/item?id=28236964
Waymo and everybody else are only trying to solve one AI problem. Tesla wanted to tackle two at the same time. Maybe they can do it, but its just an extra hurdle that others are stepping around. Also Waymo has had level 4 deployed for years.
In geofenced customizer non consumer cars?
Tesla/musk self driving claims require extensive independent verification, as do the other fsd wannabes (they are all wannabes right now).
I still would like primary focus on highway driving and safety. That is the big efficiency win for logistics and slot easier than urban streets
Tesla/musk self driving claims require extensive independent verification, as do the other fsd wannabes (they are all wannabes right now).
I still would like primary focus on highway driving and safety. That is the big efficiency win for logistics and slot easier than urban streets
To think that on Autonomy Day in 2019, Musk promised that Tesla will "have over a million robo-taxis on the road" in 2020 [1].
Musk is a visionary that has delivered amazing promises. That doesn't mean all of his promises are amazing. Some are quite clearly BS (for example, I don't think robotaxis are impossible in general, but announcing a million of them on a 15-month time frame absolutely was).
[1] eg: https://www.engadget.com/2019-04-22-tesla-elon-musk-self-dri...
Musk is a visionary that has delivered amazing promises. That doesn't mean all of his promises are amazing. Some are quite clearly BS (for example, I don't think robotaxis are impossible in general, but announcing a million of them on a 15-month time frame absolutely was).
[1] eg: https://www.engadget.com/2019-04-22-tesla-elon-musk-self-dri...
Yeah, I don't think it's possible with the current hardware. The best Tesla can hope for is Level 3, mainly due to the lack of redundancy.
That could have literally been the command from above - "I want over a million robo-taxis on the road" - and then that trickles down into the company.
Musk is a fraudster who has consistently lied to investors and the public about timelines and features - FSD, SolarWinds solar tiles, the robotaxis, "founding" Tesla, "founding" PayPal, the Vegas "loop", cheaper tunnels, hyperloop, NeuraLink brain-computer videogames, flying cars, the cave submarine, turbines on cars, "more efficient" batteries and on and on and on. There are a few lies that have not yet been proven lies - the 40k satellites or the Mars settlement or the $2M rocket, for example - but I have faith that they will also prove to be lies in time.
His only contributions have been early investment in Tesla and hiring the right people for SpaceX to achieve some decent cost savings and vertical landing on Earth. It's definitely not nothing, but it really shouldn't make anyone believe 99% of what he says, or consider him some kind of genius.
His only contributions have been early investment in Tesla and hiring the right people for SpaceX to achieve some decent cost savings and vertical landing on Earth. It's definitely not nothing, but it really shouldn't make anyone believe 99% of what he says, or consider him some kind of genius.
He tends to share his ideas publicly before they are "verified with reality", which leads to a lot of them being late, over-promised under-delivered, etc.
But if you take his statements as "thinking aloud", I much prefer this mode of operation. On the other end of the spectrum is Apple, with zero "thinking aloud" or sharing _anything_ until it's ready to ship. It's ok... but much less fun.
But if you take his statements as "thinking aloud", I much prefer this mode of operation. On the other end of the spectrum is Apple, with zero "thinking aloud" or sharing _anything_ until it's ready to ship. It's ok... but much less fun.
These are not random ideas that he throws around, these are all examples of products he has held massive press conferences for, announcing them with fanfare, and referencing them again and again in public speeches. He really does seem to believe in some of them, even some that they are quite obviously impossible - casting serious doubts on his intelligence or understanding of science (which is not surprising, given he only has an undergraduate degree in Physics as science credentials).
>It's ok...
>thinking aloud
He's a CEO and is legally bound to not lie or mislead the investors though. For example Skreli is in part serving a federal sentence for lying to the investors who didn't lose any money.
>thinking aloud
He's a CEO and is legally bound to not lie or mislead the investors though. For example Skreli is in part serving a federal sentence for lying to the investors who didn't lose any money.
Skrelli raising the price of that medication was legal. (I know you know that.)
We still have no laws preventing another greedy person doing the same.
They went after him for PR purposes.
Personally, I wish they left him alone, and instead changed drug law that will prevent these huge price hikes in the future.
In a weird way, lonely Hoody Boy exposed a huge flaw in the system, and we should be thanking him?
We still have no laws preventing another greedy person doing the same.
They went after him for PR purposes.
Personally, I wish they left him alone, and instead changed drug law that will prevent these huge price hikes in the future.
In a weird way, lonely Hoody Boy exposed a huge flaw in the system, and we should be thanking him?
This is a very naive take. There are several examples of him being intentionally dishonest, e.g. the funding secured thing. He's a sociopathic narcissist.
Starship seems really promising, though, making amazing progress in record time. I'm really optimistic about SpaceX, that whole company seems to be aligned toward one goal only, getting people on mars, if I had any expertise in that area I would sign up in a second.
Mars is a fools' goal, only really interesting as a point of pride. There will very obviously not be any kind of Mars "colony" in our lifetimes, and there are good reasons to believe that none will ever exist - there are too little resources on Mars, and too harsh conditions, for it to make any kind of sense.
Apart from that, there are chances that Spaceship will be a good product, probably making it cheaper to put stuff in orbit than it is today. On the other hand, this may wildly accelerate problems with space junk, especially if the irresponsible plans for putting more than twice the amount of objects in LEO are allowed to go through.
Apart from that, there are chances that Spaceship will be a good product, probably making it cheaper to put stuff in orbit than it is today. On the other hand, this may wildly accelerate problems with space junk, especially if the irresponsible plans for putting more than twice the amount of objects in LEO are allowed to go through.
> There will very obviously not be any kind of Mars "colony" in our lifetimes
Define "Mars colony". A thriving metropolis with millions of residents? Or a permanently crewed research station with a few dozen staff? The former is not going to happen in our lifetimes. The later isn't certain but I think it has some chance of happening. But Musk is going to try to set up the later and call it a "Mars colony", with the declared ambition of eventually developing it into the former. Aspirational naming.
> there are too little resources on Mars, and too harsh conditions, for it to make any kind of sense.
In 2021, Musk has a net worth of $170 billion. He's only 50, he could easily live for another 30 or 40 years. It is entirely plausible that by the time he dies, he will be a multi-trillionaire. To whom, or to what, is he going to leave his trillions? (Heck, forty years from now, quadrillionaires could even be a thing, and old man Musk could be one of them.) He might leave them to a charitable foundation dedicated to paying for the colonisation of Mars. If that happens – well, even if you are right that colonising Mars is pointless, if a multi-trillion, or even multi-quadrillion, dollar foundation is willing to pay for it, it may actually happen, no matter how pointless it may be.
Another possible scenario: sometime in the next few decades, the US successfully lands on Mars, and sets up a modestly sized US-led international research base. The US government continues its policy of excluding China, so China responds by landing on Mars independently and establishing their own competing Mars base. China and the US then become trapped in a new space race, in which each seeks to outdo the other with the size and capabilities of their respective Mars bases, not motivated by any practical economic or military benefits, but simply by national pride and prestige. A US-led Mars base could grow into thousands of people, and US Congress might become politically locked into paying for them out of the federal budget, even if their existence doesn't actually benefit the average US citizen in any tangible way.
(The above two scenarios are not mutually exclusive, it is possible both might happen at the same time.)
Define "Mars colony". A thriving metropolis with millions of residents? Or a permanently crewed research station with a few dozen staff? The former is not going to happen in our lifetimes. The later isn't certain but I think it has some chance of happening. But Musk is going to try to set up the later and call it a "Mars colony", with the declared ambition of eventually developing it into the former. Aspirational naming.
> there are too little resources on Mars, and too harsh conditions, for it to make any kind of sense.
In 2021, Musk has a net worth of $170 billion. He's only 50, he could easily live for another 30 or 40 years. It is entirely plausible that by the time he dies, he will be a multi-trillionaire. To whom, or to what, is he going to leave his trillions? (Heck, forty years from now, quadrillionaires could even be a thing, and old man Musk could be one of them.) He might leave them to a charitable foundation dedicated to paying for the colonisation of Mars. If that happens – well, even if you are right that colonising Mars is pointless, if a multi-trillion, or even multi-quadrillion, dollar foundation is willing to pay for it, it may actually happen, no matter how pointless it may be.
Another possible scenario: sometime in the next few decades, the US successfully lands on Mars, and sets up a modestly sized US-led international research base. The US government continues its policy of excluding China, so China responds by landing on Mars independently and establishing their own competing Mars base. China and the US then become trapped in a new space race, in which each seeks to outdo the other with the size and capabilities of their respective Mars bases, not motivated by any practical economic or military benefits, but simply by national pride and prestige. A US-led Mars base could grow into thousands of people, and US Congress might become politically locked into paying for them out of the federal budget, even if their existence doesn't actually benefit the average US citizen in any tangible way.
(The above two scenarios are not mutually exclusive, it is possible both might happen at the same time.)
You seem so sure of yourself.
I'm not sure of myself, I'm quite sure of our collective inability to do anything as complex (and expensive and wasteful of resources) as a Mars colony. Of course, time will tell.
First "starship" launch will end with Big Bada Boom and will end SpaceX.
So there's already been 10 or more of them that exploded, no? You mean the first crewed launch? That'll probably the 100th actual launch, with the tens of failures in the beginning and a streak of successes that's convincing enough to put people on. Had the same problem with Falcon as well, but we've already forgotten. All the initial rockets exploded, but now we've just gotten used to them landing perfectly.
Starship as defined on spacex.com: SpaceX’s Starship spacecraft and Super Heavy rocket (collectively referred to as Starship). So far there were no flights with both SN and SH as one ship.
From what I've seen they think its like a 50/50 chance of the first starship blowing up, but they are already building the next couple ones, so they are ready for that. They are not just building one rocket, they are building a rocket assembly line and it seems like they have cracked the code on that. Not a chance one explosion will end spaceX, I'm not sure how you could seriously think that if you have been following the development.
Starship explosion could damage/destroy large part of the "starbase" and cause significant damage in Bocca Chica. It will also have terrible impact on the wildlife refuge that's there.
In addition to that Musk's Starlink satellite will cause Kessler Syndrome. Both of these events could not only finish Spacex but will affect space exploration for decades.
"It's not a lie if you believe it." - George Costanza
I'm assuming your Twitter feed has become full of information supporting the idea the Elon is a fraud. You are falling into the same feedback loop that people who think Trump really won and the people who think that White people should apologize for existing.
You need to create new social media accounts and try to look for information supporting the opposite idea. I'm not suggesting that Elon is totally honest about everything he says, but there is more to him than what you seem to have heard.
I'm saying this to help you, because, you have been misinformed by these feedback loop algorithms. If you keep your identity tied to this idea that Elon is a total fraud you are going to become more and more upset overtime as the evidence to the contrary accumulates.
You need to create new social media accounts and try to look for information supporting the opposite idea. I'm not suggesting that Elon is totally honest about everything he says, but there is more to him than what you seem to have heard.
I'm saying this to help you, because, you have been misinformed by these feedback loop algorithms. If you keep your identity tied to this idea that Elon is a total fraud you are going to become more and more upset overtime as the evidence to the contrary accumulates.
I thought shorter astroturfing was a 2020 thing
There is a wow factor in the presentation, especially how many of the things they are making from ground-up and in-house, at times making me feel like, may be they want to be AI company now.
Definitely the TSLA stock price increase is being reflected into the massive investments into AI.
Definitely the TSLA stock price increase is being reflected into the massive investments into AI.
> Definitely the TSLA stock price increase is being reflected into the massive investments into AI.
Actually the best indicator of stock price increase has been if an unknown buyer comes in an buys huge amounts of OTM calls expiring that week with the hope of causing a gamma squeeze.
Watch the option flow.
You'll note also that it just hasn't been the same since Archegos imploded for example, but there is definitely another smaller buyer doing the same. It's almost certainly a group effort given the amount of money at play.
Actually the best indicator of stock price increase has been if an unknown buyer comes in an buys huge amounts of OTM calls expiring that week with the hope of causing a gamma squeeze.
Watch the option flow.
You'll note also that it just hasn't been the same since Archegos imploded for example, but there is definitely another smaller buyer doing the same. It's almost certainly a group effort given the amount of money at play.
Stock price will likely not go up the day after. Analysts have to break down the implication in terms of potential revenue models that Tesla could gain for it. From there we can establish a future price target based on potential revenue. For now the market wants to focus on revenue today, that is mostly sold products today and near term products soon to be available.
What I meant was, given the meteroic rise in the past years increase in valuation - Tesla took a lot of capital and invested in this division.
"... the car can be thought of as an animal..."