Flash Memory Wear Killing Older Teslas Due to Excessive Data Logging: Report(tomshardware.com)
tomshardware.com
Flash Memory Wear Killing Older Teslas Due to Excessive Data Logging: Report
https://www.tomshardware.com/news/flash-memory-wear-killing-older-teslas-due-to-excessive-data-logging-report
181 comments
> - Why were they not using automotive grade flash?
Cool. I did not realize there were such products.
Cool. I did not realize there were such products.
Well, for a given chip, it usually just refers to a wider temperature range.
But there are other types of flash that are better suited for this environment. I see parallel and SPI NOR flash a lot in ruggedized embedded devices (usually with a minimum of 100,000 erase cycles), but they're generally a lower capacity.
I don't know the exact cause of this failure, but there are ways to design around unreliable flash too. They could partition it such that the firmware image is in a read-only partition, and it runs out of a ramdisk, with only user settings and log messages stored in a R/W partition. That may be what they're doing, and perhaps the issue is due to the software crashing when the write fails. I'm not sure.
But there are other types of flash that are better suited for this environment. I see parallel and SPI NOR flash a lot in ruggedized embedded devices (usually with a minimum of 100,000 erase cycles), but they're generally a lower capacity.
I don't know the exact cause of this failure, but there are ways to design around unreliable flash too. They could partition it such that the firmware image is in a read-only partition, and it runs out of a ramdisk, with only user settings and log messages stored in a R/W partition. That may be what they're doing, and perhaps the issue is due to the software crashing when the write fails. I'm not sure.
>I don't know the exact cause of this failure, but there are ways to design around unreliable flash too. They could partition it such that the firmware image is in a read-only partition, and it runs out of a ramdisk, with only user settings and log messages stored in a R/W partition. That may be what they're doing, and perhaps the issue is due to the software crashing when the write fails. I'm not sure.
Not in this field so I might be wrong, but I'm guessing they could have also:
* Have redundant storage, failover when the first flash chip dies, and log a fault telling the user to get it looked at. Flash is cheap anyway
* Use removable storage. PCIe, M.2, proprietary, whatever. If it dies then at least you can replace it relatively cheap.
I'm interested in what other problems will surface with Teslas in the next few years. This seems like a huge design failure, I doubt it's the only one.
Not in this field so I might be wrong, but I'm guessing they could have also:
* Have redundant storage, failover when the first flash chip dies, and log a fault telling the user to get it looked at. Flash is cheap anyway
* Use removable storage. PCIe, M.2, proprietary, whatever. If it dies then at least you can replace it relatively cheap.
I'm interested in what other problems will surface with Teslas in the next few years. This seems like a huge design failure, I doubt it's the only one.
[deleted]
Reminds me of a comment I’ve seen a couple of years ago: “Tesla is not an automotive company, it’s a tech company”
In all fairness, if you look at cars preserved from the 80s, when electronics started becoming pervasive, there are a lot of failures there too.
For instance, the multi-segment displays tend to fail in little bits. I have a car that doesn't have that problem, but the main ECU tends to fail and have to be resoldered.
Of course, mostly people don't care because after 20 years you expect the vast majority to be scrapped, but in any case, you still design for a certain lifespan.
For instance, the multi-segment displays tend to fail in little bits. I have a car that doesn't have that problem, but the main ECU tends to fail and have to be resoldered.
Of course, mostly people don't care because after 20 years you expect the vast majority to be scrapped, but in any case, you still design for a certain lifespan.
This is giving me deja vu...I'm not alone right? This same comment is posted on the same story but from the past, no?
It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this could easily happen as a BOM-reducing or size-reducing measure for something like a cloud thermostat, but for a luxury car?
They probably used a reference design and then failed to perform the step of analyzing the design for weaknesses like this one. Or they did but decided it wasn't their problem.
I assume the decision to leave datalogging turned on to this degree was not made by the same folks who decided to put datalogging and firmware in the same flash IC.
In mass production every tenth of a penny pays. That is the thing in car mass production.
In mass production every tenth of a penny pays if it pays. This is a cost benefit analysis and not just a cost analysis. If in the lifetime of the car, this part will fail then it will have to be replaced at great expense, $1,800 to $3,000. If that happens under warranty, that will cost Tesla a lot. If that happens out of warranty, that will cost the customer a lot and will cost Tesla a lot in customer good will.
I sure hope Tesla has figured this out for 3 and the Y. (It has. According to the FA, this has to do with MCUv1 Tesla Model S and Model X units up to 2018.)
I sure hope Tesla has figured this out for 3 and the Y. (It has. According to the FA, this has to do with MCUv1 Tesla Model S and Model X units up to 2018.)
Yeah but there's Toyota Corolla style mass production and Tesla Model S style mass production. In the former? Sure, save $5 here and there and it adds up. In the latter? Spend the money. Any small saving from corner-cutting will be far outweighed by the loss of reputation if your $100k car bricks itself for want of a $5 part.
And the $100s it will cost to send out a service guy.
Maybe the Toyota Corolla isn't the best comparison?
Are you saying this because of the "stuck pedal" thing? Just pretend he means 90s Corollas then. They were properly robust in spite of cost minimization.
There was no "stuck pedal" thing. Not a single incident of "unintended acceleration" with Toyota was ever proven. Something like 70-80% of the incidents turned out to be intoxicated drivers, and the rest were driver error.
Uhhh.
https://abcnews.go.com/Blotter/toyota-pay-12b-hiding-deadly-...
https://mashable.com/2014/03/19/toyota-lied-aceleration-reca...
https://www.rs-law.com/unintended-acceleration/
And here's an article by one of the experts who reviewed Toyota's code and found the smoking gun: https://embeddedgurus.com/barr-code/2013/10/an-update-on-toy...
https://abcnews.go.com/Blotter/toyota-pay-12b-hiding-deadly-...
https://mashable.com/2014/03/19/toyota-lied-aceleration-reca...
https://www.rs-law.com/unintended-acceleration/
And here's an article by one of the experts who reviewed Toyota's code and found the smoking gun: https://embeddedgurus.com/barr-code/2013/10/an-update-on-toy...
I was under the same impression for some time, but as you can see from replies, there's some disagreement about that.
You can't exactly prove it though, the design meant that we can't say for sure it didn't happen. However there isn't enough information logged to say it did either.
You're right that the government did a big investigation and ultimately didn't find a major problem with the software/electronic throttle control system, but mistaken about the other causes of unintended acceleration.
Ultimately Toyota recalled about 7 million cars (some twice). Dealers were installing the wrong floor mats, and these were trapping the accelerator pedal. This problem was well-documented as the cause of "unintended acceleration" incidents, including at least five deaths. Additionally, there was a lack of "brake override" system to idle the engine when the brake and accelerator were both applied (Toyota added one at the government's request), and some pedals manufactured by a supplier were well-documented as "sticky" (they recalled these cars and fixed this).
See the NHTSA report, pages 4-10 (PDF pages 15-21): https://one.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-UA_report.pd...
In summary, there were real "stuck pedal" problems with these cars that led to unintended acceleration and fatalities and, eventually, recalls and fixes. And, yes, some or many incidents were probably related to "pedal misapplication."
Ultimately Toyota recalled about 7 million cars (some twice). Dealers were installing the wrong floor mats, and these were trapping the accelerator pedal. This problem was well-documented as the cause of "unintended acceleration" incidents, including at least five deaths. Additionally, there was a lack of "brake override" system to idle the engine when the brake and accelerator were both applied (Toyota added one at the government's request), and some pedals manufactured by a supplier were well-documented as "sticky" (they recalled these cars and fixed this).
See the NHTSA report, pages 4-10 (PDF pages 15-21): https://one.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-UA_report.pd...
In summary, there were real "stuck pedal" problems with these cars that led to unintended acceleration and fatalities and, eventually, recalls and fixes. And, yes, some or many incidents were probably related to "pedal misapplication."
"Dealers were installing the wrong floor mats, and these were trapping the accelerator pedal. This problem was well-documented as the cause of "unintended acceleration" incidents, including at least five deaths."
This sounds like an illusory problem to me, not because I doubt the facts, but because it's all about framing. I got floor mats for an old car, with anchors to hook to the seat rails, and I couldn't figure out how to fasten them. So I took them to a dealer and asked them to do it. After a while, they came loose and the mat shifted several inches in the direction of the gas pedal. Now, it didn't especially matter, and I didn't crash, and eventually I repositioned it. But this is really about multiplying large numbers by small probabilities. There are probably only a couple hundred cars like mine on the road. If there were millions, perhaps it would be a major problem. But why should we all worry about Toyotas just because the floormat issues are mentally grouped together?
Non-floormat issues are different, but I think I would have to experience it to believe it when it comes to the brakes not being able to override the engine on a typical Toyota.
This sounds like an illusory problem to me, not because I doubt the facts, but because it's all about framing. I got floor mats for an old car, with anchors to hook to the seat rails, and I couldn't figure out how to fasten them. So I took them to a dealer and asked them to do it. After a while, they came loose and the mat shifted several inches in the direction of the gas pedal. Now, it didn't especially matter, and I didn't crash, and eventually I repositioned it. But this is really about multiplying large numbers by small probabilities. There are probably only a couple hundred cars like mine on the road. If there were millions, perhaps it would be a major problem. But why should we all worry about Toyotas just because the floormat issues are mentally grouped together?
Non-floormat issues are different, but I think I would have to experience it to believe it when it comes to the brakes not being able to override the engine on a typical Toyota.
This twitter thread from last year might enlighten you:
https://twitter.com/atomicthumbs/status/1032939617404645376
https://twitter.com/atomicthumbs/status/1032939617404645376
Discussed heavily on HN at the time: https://news.ycombinator.com/item?id=17835760
[deleted]
That thread is a trip.
Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars. This is especially evident when looking at comparable cases of part failure, like the infamous touchscreen failures: https://www.thedrive.com/tech/27989/teslas-screen-saga-shows...
You are absolutely right in that this failure should not have happened, because they should use some other medium to store their logs that is not getting cramped by increasingly bigger firmware versions and that is appropriately sized and engineered to withstand at least a century or two of heavy usage. But since cloud thermostat and media player box vendors also get away with just stuffing some embedded Linux distro on a single chip of the cheapest Flash they can find on the spot market, Tesla probably didn't put much thought into why they shouldn't just do the same. After all, this system is just a media player box mounted in a car, right?
(Except that for some totally absurd reason, the car ceases to function entirely if its media system breaks, which is the next big WTF in this story...)
You are absolutely right in that this failure should not have happened, because they should use some other medium to store their logs that is not getting cramped by increasingly bigger firmware versions and that is appropriately sized and engineered to withstand at least a century or two of heavy usage. But since cloud thermostat and media player box vendors also get away with just stuffing some embedded Linux distro on a single chip of the cheapest Flash they can find on the spot market, Tesla probably didn't put much thought into why they shouldn't just do the same. After all, this system is just a media player box mounted in a car, right?
(Except that for some totally absurd reason, the car ceases to function entirely if its media system breaks, which is the next big WTF in this story...)
"Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars."
Who didn't assume this to be the case? Teslas are automotive iPhones, and I've always assumed they would be scrap in 3-5 years rather than 15-20. This is why I don't think it was ever valid to compare the purchase price to a regular car.
Who didn't assume this to be the case? Teslas are automotive iPhones, and I've always assumed they would be scrap in 3-5 years rather than 15-20. This is why I don't think it was ever valid to compare the purchase price to a regular car.
[deleted]
> It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this could easily happen as a BOM-reducing or size-reducing measure for something like a cloud thermostat, but for a luxury car?
It is strange no one performed back of the envelope calculations on flash life vs logging size.
I've worked in embedded, we had a time series based rolling log. Flash life calculation was done at regular intervals.
IIRC with what we shipped, the flash chip would outlive the battery by 3x or 4x. And that was without wear leveling!
We were smart enough to continue the rolling log at the last point on flash we uploaded from, avoiding the mistake I've seen some products do of starting over from the very "beginning" after every log sync. Do that and you end up with 10% of your flash worn down to nothing and 90% of your flash going unused.
Also, unless needed, please stop using JSON. Use a binary format instead. Protobufs is trivial to get up and running almost anywhere. Other binary encoding formats may serve a product's needs better. JSON is fine if everything is low bandwidth web services talking now and then, or low friction HTTP APIs, but for other use cases... use something else.
Worst I ever saw was XML being used to RPC invoke every single function call, even local ones. All params got packed up in XML, sent over to the destination, who'd then unpack the XML, execute, and package up the return value. A double digit % of the CPU was being used on XML serialization and de-serialization. (If your choice is that or JSON... probably go with JSON!)
Logging flash being same as storage flash is super weird though. Though if their MCU only supports hanging a single flash chip off of it, I could see it happening.
It is strange no one performed back of the envelope calculations on flash life vs logging size.
I've worked in embedded, we had a time series based rolling log. Flash life calculation was done at regular intervals.
IIRC with what we shipped, the flash chip would outlive the battery by 3x or 4x. And that was without wear leveling!
We were smart enough to continue the rolling log at the last point on flash we uploaded from, avoiding the mistake I've seen some products do of starting over from the very "beginning" after every log sync. Do that and you end up with 10% of your flash worn down to nothing and 90% of your flash going unused.
Also, unless needed, please stop using JSON. Use a binary format instead. Protobufs is trivial to get up and running almost anywhere. Other binary encoding formats may serve a product's needs better. JSON is fine if everything is low bandwidth web services talking now and then, or low friction HTTP APIs, but for other use cases... use something else.
Worst I ever saw was XML being used to RPC invoke every single function call, even local ones. All params got packed up in XML, sent over to the destination, who'd then unpack the XML, execute, and package up the return value. A double digit % of the CPU was being used on XML serialization and de-serialization. (If your choice is that or JSON... probably go with JSON!)
Logging flash being same as storage flash is super weird though. Though if their MCU only supports hanging a single flash chip off of it, I could see it happening.
Estimating log growth rate is troublesome. It might be easy enough to measure under normal operating conditions, but sometimes something breaks, maybe a sensor goes bad, and all of a sudden you're spamming your logs at 10,000x the normal rate.
Assuring yourself that under no circumstance is that possible can sometimes be tricky.
Assuring yourself that under no circumstance is that possible can sometimes be tricky.
> maybe a sensor goes bad, and all of a sudden you're spamming your logs at 10,000x the normal rate.
Sure, but sensor error reporting frequency should also be documented and calculated somewhere. Embedded requires more planning, this stuff isn't like AWS instances where the worst case is a larger bill.
And from the sound of it just general usage is causing problems.
Sure, but sensor error reporting frequency should also be documented and calculated somewhere. Embedded requires more planning, this stuff isn't like AWS instances where the worst case is a larger bill.
And from the sound of it just general usage is causing problems.
If you have a log for something with an indefinite lifespan, and no maintenance of it possible, and a finite storage medium, why wouldn't you just make it circular?
Reminds me of some people's aversion to auto-killing processes when the OS runs out of memory, as though it could ever be better to lock up completely.
Reminds me of some people's aversion to auto-killing processes when the OS runs out of memory, as though it could ever be better to lock up completely.
FRU sizes are optimized to minimize warranty cost and must be balanced against things like design complexity, reliability and service procedure count/complexity.
This was a no-brainer IMO: Tesla wanted their cars to be superior and time to market was key. Why do handwringing about the cars that might be still on the road in five or ten years when that's beyond the warranty and beyond Tesla's life in business if they don't deliver a Model S with the target specs? They made the right tradeoff. This is probably one of dozens of design weaknesses that Tesla cars have, ones that allowed them to get way ahead of the competition.
Now that we have arrived at five or ten years later there is some nonzero risk to their brand. Do they have unreliable cars? How many Tesla owners still own the original car and how many of these were purchased second-hand? Folks who bought the original car and they encounter this failure mode might think about this expensive out-of-warranty repair when they go car shopping next. If there are a lot of them, Tesla could consider the financial tradeoff between extending the warranty to cover this failure mode and the impact on future sales.
This was a no-brainer IMO: Tesla wanted their cars to be superior and time to market was key. Why do handwringing about the cars that might be still on the road in five or ten years when that's beyond the warranty and beyond Tesla's life in business if they don't deliver a Model S with the target specs? They made the right tradeoff. This is probably one of dozens of design weaknesses that Tesla cars have, ones that allowed them to get way ahead of the competition.
Now that we have arrived at five or ten years later there is some nonzero risk to their brand. Do they have unreliable cars? How many Tesla owners still own the original car and how many of these were purchased second-hand? Folks who bought the original car and they encounter this failure mode might think about this expensive out-of-warranty repair when they go car shopping next. If there are a lot of them, Tesla could consider the financial tradeoff between extending the warranty to cover this failure mode and the impact on future sales.
Which auto maker's electronics are likely to be reliable for the 25-year nominal life of the vehicle (US fleet average age is 11.8 years and rising)?
And which will have spare parts available for that long?
I'm thinking about buying a new car, but I'm worried.
And which will have spare parts available for that long?
I'm thinking about buying a new car, but I'm worried.
My family’s ownership outcomes strongly suggest getting a Toyota. I’m driving my Dad’s old 2001 Tundra pickup with 350k miles on the same engine. I was amazed when I bought it from him how every switch still worked, the original AC still blew cold. This is not my normal experience buying used cars from other manufacturers. Usually at about 150k miles they turn into a clown car of various systems failing. Someone put thought into each component on the truck and how it would age.
Generally speaking Toyota is conservative in terms of rolling out new technologies and sticks with what's proven. For example the base Camry has a 2.5L engine and 8 speed auto transmission and the base Accord a 1.5 turbo and 10 speed auto. Toyota also goes more in depth terms of QA and testing on individual parts than others.
From an article on the Toyota - BMW Supra/M4 collaboration,
"...BMW couldn’t believe how extensive some of our quality and efficiency studies were as parts came into shape one by one. We would take every bit down to a fastener or rivet, and put it through our stringent quality control and a dozen other testing, we’d ship thousands of parts back to Japan for analysis. That is normal to us."
http://club4ag.com/chief-engineer-tetsuya-tada-reveals-the-a...
From an article on the Toyota - BMW Supra/M4 collaboration,
"...BMW couldn’t believe how extensive some of our quality and efficiency studies were as parts came into shape one by one. We would take every bit down to a fastener or rivet, and put it through our stringent quality control and a dozen other testing, we’d ship thousands of parts back to Japan for analysis. That is normal to us."
http://club4ag.com/chief-engineer-tetsuya-tada-reveals-the-a...
> From an article on the Toyota - BMW Supra/M4 collaboration
I think you mean Z4
I think you mean Z4
That article seems to be more balanced shall we say, in terms of what was normal for which team. Toyota's engineering seems to be closer to production while BMW's seems to be closer to design.
* I almost started to think if they had an infinite budget funding to the task of design. *
And it seemed like the opinion from Toyota was that Toyota has a body style in mind at first but BMW lets the body style be influenced its contents.
BMW's fundamental difference in approach was that they wanted to design a package, and from there they would naturally evolve a shape and size of the body from that packaging, a functionally oriented goal. ... Our company (Toyota) with my tenure and experience, the focus was always design elements being the priority. We would first spend a lot of time on the shape and appeal of the car from visual perspective ...
But let the article speak for itself. I kept my quotes short.
* I almost started to think if they had an infinite budget funding to the task of design. *
And it seemed like the opinion from Toyota was that Toyota has a body style in mind at first but BMW lets the body style be influenced its contents.
BMW's fundamental difference in approach was that they wanted to design a package, and from there they would naturally evolve a shape and size of the body from that packaging, a functionally oriented goal. ... Our company (Toyota) with my tenure and experience, the focus was always design elements being the priority. We would first spend a lot of time on the shape and appeal of the car from visual perspective ...
But let the article speak for itself. I kept my quotes short.
It might of changed directions recently. 2020 BMW 2 Series is a series of cut&paste design cues from 2019 Toyota Supra.
You know how rigorous it is when even the Germans are surprised.
Toyota has been known for their reliability for a long time now. I can attest that with my Lexus, it's 13 years old now and in the 7 years that I have owned it, it has had zero defects. Last year it even got recalled (12 years after production) because the fuel filter might not have been installed correctly in the factory, so they replaced it for free.
That said: I still do every scheduled maintenance, which helps tremendously. The main problem with older vehicles is that people stop following the recommended services as the car gets older. Which is of course the opposite of what you should do with an old car.
That said: I still do every scheduled maintenance, which helps tremendously. The main problem with older vehicles is that people stop following the recommended services as the car gets older. Which is of course the opposite of what you should do with an old car.
> That said: I still do every scheduled maintenance, which helps tremendously. The main problem with older vehicles is that people stop following the recommended services as the car gets older. Which is of course the opposite of what you should do with an old car.
exactly this. I've been driving my Toyota for 15 years, never had a problem with it. But I still, to this day, have it maintained. The cost of that maintenance is pennies compared to purchasing a new car. If you can afford it, it's foolish not to.
exactly this. I've been driving my Toyota for 15 years, never had a problem with it. But I still, to this day, have it maintained. The cost of that maintenance is pennies compared to purchasing a new car. If you can afford it, it's foolish not to.
> Last year it even got recalled (12 years after production)
Toyota is great about that. There was something in the steering column that needed replaced on my '95 T100 and and 15 years later they sent a notice offering to fix it for free. I never did get it replaced, it was a 'we gotta keep the vehicle for a couple of days' and we'd been driving it since it was brand new without issue so I decided to keep on since the risk seemed incredibly minimal.
Toyota is great about that. There was something in the steering column that needed replaced on my '95 T100 and and 15 years later they sent a notice offering to fix it for free. I never did get it replaced, it was a 'we gotta keep the vehicle for a couple of days' and we'd been driving it since it was brand new without issue so I decided to keep on since the risk seemed incredibly minimal.
>> Last year it even got recalled (12 years after production)
>Toyota is great about that.
after you sue them https://www.abc.net.au/news/2019-08-01/toyota-faces-class-ac... https://www.caradvice.com.au/780759/class-action-filed-over-... https://autoweek.com/article/recalls/toyota-will-settle-truc...
>Toyota is great about that.
after you sue them https://www.abc.net.au/news/2019-08-01/toyota-faces-class-ac... https://www.caradvice.com.au/780759/class-action-filed-over-... https://autoweek.com/article/recalls/toyota-will-settle-truc...
Find me a vehicle manufacturer currently in business that hasn't been sued and sells more than 100,000 cars a year.
I see the .au domains and while I don't know how stuff is in Australia, the United States it looks like more than 10,000 class action lawsuits are filed a year [1]. Hell I just got an email about one this weekend where apparently Audible was sued for not making it more clear that credits expired if you didn't use them... I mean, as long as I can remember (and I've been a member over a decade) Audible has explicitly stated that you can only accumulate so many unused credits and that if you cancel any outstanding ones are nullified. Lawyers make BANK if they win a class action.
So did they get sued and then issue a recall, yes it looks that way but in the case of my recall it was voluntary and involved no lawsuit that I know of. They found some defect that happened under extremely specific (and unrealistic for 99.99% of drivers) conditions and issued a voluntary recall. When I got the notice the Truck was 15 or even 16 years old and we hadn't had it at a dealership in 2 address (I'd never had it at a dealership under my name) yet they found me to tell me and then sent the notice every few months for a couple of years until I notified them I no longer owned the vehicle per the instructions in the mailings.
[1] https://gettingthedealthrough.com/area/82/jurisdiction/23/cl...
I see the .au domains and while I don't know how stuff is in Australia, the United States it looks like more than 10,000 class action lawsuits are filed a year [1]. Hell I just got an email about one this weekend where apparently Audible was sued for not making it more clear that credits expired if you didn't use them... I mean, as long as I can remember (and I've been a member over a decade) Audible has explicitly stated that you can only accumulate so many unused credits and that if you cancel any outstanding ones are nullified. Lawyers make BANK if they win a class action.
So did they get sued and then issue a recall, yes it looks that way but in the case of my recall it was voluntary and involved no lawsuit that I know of. They found some defect that happened under extremely specific (and unrealistic for 99.99% of drivers) conditions and issued a voluntary recall. When I got the notice the Truck was 15 or even 16 years old and we hadn't had it at a dealership in 2 address (I'd never had it at a dealership under my name) yet they found me to tell me and then sent the notice every few months for a couple of years until I notified them I no longer owned the vehicle per the instructions in the mailings.
[1] https://gettingthedealthrough.com/area/82/jurisdiction/23/cl...
When I was in HS I had a friend whose family had this old beat up orange smallbed truck from the 70's (this was mid 90's). That thing was beat all to hell, ugly as sin. It was used as a truck.
But you know what? That damned thing ran like a beauty. It literally ran until their son started driving it and wrecked it.
I was impressed enough with it, that when it was time for me to purchase a vehicle, I went with Toyota. 15 years later and that thing has never left me on the side of the road, not even once (I do keep up maintenance on it). I'm considering purchasing another vehicle because it's time to get something larger, and when I do it will most likely be Toyota.
I've seen so many people say that at 200k miles Toyota is just getting warmed up, and there's some truth to it. You obviously have to keep up with maintenance on the vehicle, but if you do I can guarantee it will treat you right. And I'm not saying other manufacturers are any worse, but I am saying Toyota is a great vehicle to buy.
But you know what? That damned thing ran like a beauty. It literally ran until their son started driving it and wrecked it.
I was impressed enough with it, that when it was time for me to purchase a vehicle, I went with Toyota. 15 years later and that thing has never left me on the side of the road, not even once (I do keep up maintenance on it). I'm considering purchasing another vehicle because it's time to get something larger, and when I do it will most likely be Toyota.
I've seen so many people say that at 200k miles Toyota is just getting warmed up, and there's some truth to it. You obviously have to keep up with maintenance on the vehicle, but if you do I can guarantee it will treat you right. And I'm not saying other manufacturers are any worse, but I am saying Toyota is a great vehicle to buy.
I was driving a '67 C10 in 2003-2004 and it still ran perfect. It needed a frame off restoration as the bed was severely damaged from decades of use as a farm truck and removing the bed alone was more work than I wanted to do (at least a 4 man job) so I ended up selling it for about 2 grand more than I had bought it for.
Then I took my father's '95 Toyota t100 from my mother when she got a new vehicle, drove it until 2013 and gave it to my half brother. It has 130k or so when I gave it to him and aside from having a really squeaky bed was in great shape, the only remotely major work I'd had to do was replace the fuel pump bracket due to rust which involved dropping the gas tank (2 man job). The ball joints and pitmans needed replaced in the next 10k miles probably but that's just a special tool rental and an afternoon sitting on the ground.
Then take my 2013 impala... I've got 68k miles on it now. I've had a wire harness malfunction which would randomly trigger ABS and traction control while you were driving, I've had to replace the evap sensor THREE TIMES and it is now throwing codes again, the factory rotors have already had to be replaced due to severe rust, any time humidity gets above 70% or so it throws another engine code because of more fuel/emissions sensors.... pinches bridge of nose
Then I took my father's '95 Toyota t100 from my mother when she got a new vehicle, drove it until 2013 and gave it to my half brother. It has 130k or so when I gave it to him and aside from having a really squeaky bed was in great shape, the only remotely major work I'd had to do was replace the fuel pump bracket due to rust which involved dropping the gas tank (2 man job). The ball joints and pitmans needed replaced in the next 10k miles probably but that's just a special tool rental and an afternoon sitting on the ground.
Then take my 2013 impala... I've got 68k miles on it now. I've had a wire harness malfunction which would randomly trigger ABS and traction control while you were driving, I've had to replace the evap sensor THREE TIMES and it is now throwing codes again, the factory rotors have already had to be replaced due to severe rust, any time humidity gets above 70% or so it throws another engine code because of more fuel/emissions sensors.... pinches bridge of nose
> Then take my 2013 impala... [...] I've had a wire harness malfunction which would randomly trigger ABS and traction control while you were driving
I had that exact same problem in a 2002 Buick. I can't believe GM hadn't fixed that yet.
I had that exact same problem in a 2002 Buick. I can't believe GM hadn't fixed that yet.
Sounds about right, the evap sensor issue I keep having seems to span about 15 years of cars and a LOT of models. What really chapped my ass was that 'wiring isn't covered under warranty' um hi, your stupid wiring keeps triggering ABS and traction control, that's dangerous.
The other day I was visiting my sister and she asked me to scan the codes on her 2014 Silverado.
When I put the key in the ignition position for on (without starting it) so I could read the codes, it complained that the battery was low, and to start the truck soon.
Oddly enough, my dad's 2001 Silverado always had the same problem (just a red battery light back then), 2 years of ownership so far and it's battery is still fine, same warning right before you start it that goes away once it's on.
When I told this to her she told me her old Jimmy did the same thing.
The code her truck was throwing was coincidentally the same code as my dad's 2001 truck, about the evap canister.
I was flabbergasted, 13 years age difference and this truck was no different.
When I put the key in the ignition position for on (without starting it) so I could read the codes, it complained that the battery was low, and to start the truck soon.
Oddly enough, my dad's 2001 Silverado always had the same problem (just a red battery light back then), 2 years of ownership so far and it's battery is still fine, same warning right before you start it that goes away once it's on.
When I told this to her she told me her old Jimmy did the same thing.
The code her truck was throwing was coincidentally the same code as my dad's 2001 truck, about the evap canister.
I was flabbergasted, 13 years age difference and this truck was no different.
15 years does indeed sound like typical GM timeframe for a fix, as evidenced by Northstar disaster.
UK Top Gear tried to destroy a hilux a few years back. They left it in the sea, set it on top of a tower block as it was blown up, and a few other things I cant remember. It still ran (to a degree) when they gave up and put it on display in their studio.
A friend of mine used to tell a story of a Mercedes Benz car he knew from the German embassy in Cairo. When they first went there, it was their car. It had a crack on the engine head that leaked a little bit of oil, just enough to make the crack obvious.
When they went back, the car was still there. Still being used day in, day out, for 10 years. It still had the crack on the engine.
When they went back, the car was still there. Still being used day in, day out, for 10 years. It still had the crack on the engine.
It highly depends on which Toyota you pick.
RAV4 for instance is plagued with faults while my 2013 Avensis ran quite ok the first 90000 kilometres I drove it before getting a tesla.
It did have to change brakes, a spring in one of the keys failed and the open/close window button on the passenger side broke. It also rusted under the chrome strips..
But compared to other cars I think it did quite well :)
RAV4 for instance is plagued with faults while my 2013 Avensis ran quite ok the first 90000 kilometres I drove it before getting a tesla.
It did have to change brakes, a spring in one of the keys failed and the open/close window button on the passenger side broke. It also rusted under the chrome strips..
But compared to other cars I think it did quite well :)
Certain Tundra, Tacoma, and Sequoia frames have rust issues so bad, Toyota settled for billions of dollars.
https://autoweek.com/article/recalls/toyota-will-settle-truc...
https://autoweek.com/article/recalls/toyota-will-settle-truc...
Happened to my uncle and toyota put a brand new frame on a 10 year old truck for free.
[deleted]
Our '91 Tacoma (simply called SR5 back then) made it well past 200k miles and only stopped then because it got caught in a garage fire a couple years ago and melted. Loved that rig
There are plenty of 90s era car electronics that are going fine. The sensors are going to be maintenance items after 20 years, but there's no reason to think the solid state components won't normally last 25+ years.
Spare (electronic) parts shouldn't be an issue with major brands. I've seen the same sensor used on a dozen models from several manufactures produced over 10+ years. As Takata exposed, there aren't that many parts suppliers around the globe, so a lot of commodity components are shared across the industry.
For long-term part replacement, I'd be far more concerned with construction techniques like plasma-transferred wire arc sprays used for creating cylinder linings. This technique cannot be replicated outside of the factory, thus, they kind of make engine blocks disposable (they can be sleeved, but they won't perform the same and will require different piston heads).
Spare (electronic) parts shouldn't be an issue with major brands. I've seen the same sensor used on a dozen models from several manufactures produced over 10+ years. As Takata exposed, there aren't that many parts suppliers around the globe, so a lot of commodity components are shared across the industry.
For long-term part replacement, I'd be far more concerned with construction techniques like plasma-transferred wire arc sprays used for creating cylinder linings. This technique cannot be replicated outside of the factory, thus, they kind of make engine blocks disposable (they can be sleeved, but they won't perform the same and will require different piston heads).
> but there's no reason to think the solid state components won't normally last 25+ years.
Except this article is specifically about solid state components failing in a much shorter timeframe.
Except this article is specifically about solid state components failing in a much shorter timeframe.
[deleted]
Except most cars don't rewrite their flash every few seconds.
Most modules with flash in 'em are programmed once at the manufacturer, reflashed once or twice during assembly, maybe reflashed once more at the dealer before delivery, and that's it. Maybe again post-sale if there's a recall or TSB or something that can be fixed in software.
Flash quite happily lasts 25+ years if you don't abuse it far beyond any flash manufacturer's wearout specs. Go figure!
Most modules with flash in 'em are programmed once at the manufacturer, reflashed once or twice during assembly, maybe reflashed once more at the dealer before delivery, and that's it. Maybe again post-sale if there's a recall or TSB or something that can be fixed in software.
Flash quite happily lasts 25+ years if you don't abuse it far beyond any flash manufacturer's wearout specs. Go figure!
Airbag control modules do constantly write data. The last 5 seconds or so of sensor and input data is recorded for crash investigations.
is this actually written to flash?
concievably a big capacitor or watch battery could keep a 4 or 8k RAM going for quite a long time.
concievably a big capacitor or watch battery could keep a 4 or 8k RAM going for quite a long time.
If you look at airbag module teardowns there is a giant capacitor, but this is used to fire airbags in the event the vehicle's power system is rendered inoperable prior to deployment.
It's got to survive a wreck, so I suspect power reliant memory is out.
Most manufacturers’ datasheets specify 20 or less years of data retention (even when powered on). This might seem like random sufficiently large number, but there really are devices made in mid-90’s (notably early FastEthernet cards) that do not work because they lost contents of their configuration EEPROMs.
Telsa having trouble building reliable cars is not evidence of an industry-wide issue. There are tens of millions of cars on the road with decades old solid state components that are in perfect operation.
"tens of millions" and "in perfect operation" doesn't quite jive with me but the numbers are mostly there.
For Toyota ~57 million vehicles from '02-'09 https://www.toyota-global.com/company/history_of_toyota/75ye...
For Honda ~23 million from '02-09 https://www.statista.com/statistics/267276/worldwide-automob...
Volkswagen group looked to be in between the two.
Didn't look for Ford or GM.
For Toyota ~57 million vehicles from '02-'09 https://www.toyota-global.com/company/history_of_toyota/75ye...
For Honda ~23 million from '02-09 https://www.statista.com/statistics/267276/worldwide-automob...
Volkswagen group looked to be in between the two.
Didn't look for Ford or GM.
There were 16 million new vehicles are sold last year in the US and there's 260 million registered vehicles on the road with a median age of nearly 12 years old.
To say that there's "tens of millions" of perfectly operational cars on the road in the US is really an understatement.
To say that there's "tens of millions" of perfectly operational cars on the road in the US is really an understatement.
If you have a used audi and the mechanic comes back and tells you its the wiring harness, thats probably it for the car. Unless you like spending 3k to fix a 4k car.
Why is a wiring harness so expensive?
EEC-IV and EEC-V (Ford) ECUs haven't yet found the "late" stage of the bathtub curve and they're 15-35yo give or take. I'm not familiar with other brands of ECU but I suspect that they tend to be similar. If you look at all brands there's probably some that don't last because they were subject to the capacitor plague but they're clearly so few and far between that no brand has developed a reputation for ECUs crapping out. I wouldn't personally be worried about electronics failures. Physical failures of the electronic systems are far more likely.
Because this is a decently upscale crowd on HN everyone is going to tell you Toyota/Honda. I'm not going to write the mini-novel it takes to explain why but it suffices to say that is one filter bubble's opinion and other bubbles will have other opinions. As someone who has fairly extensive experience with vehicles that most here would consider far beyond end of life I assure you the badge on the grill matters far far less than what you use the vehicle for and how you treat it. Anything will "last" 25yr if you're willing to baby it there, put a bunch of money into it or just deal with it having all sorts of NVH issues in years 20-25. Trying to predict what vehicle and configuration will last 25yr with the least work (the unstated assumption in these discussions) is a complete and total fools errand because the equation is dominated by variables outside, or mostly outside your control.
Because this is a decently upscale crowd on HN everyone is going to tell you Toyota/Honda. I'm not going to write the mini-novel it takes to explain why but it suffices to say that is one filter bubble's opinion and other bubbles will have other opinions. As someone who has fairly extensive experience with vehicles that most here would consider far beyond end of life I assure you the badge on the grill matters far far less than what you use the vehicle for and how you treat it. Anything will "last" 25yr if you're willing to baby it there, put a bunch of money into it or just deal with it having all sorts of NVH issues in years 20-25. Trying to predict what vehicle and configuration will last 25yr with the least work (the unstated assumption in these discussions) is a complete and total fools errand because the equation is dominated by variables outside, or mostly outside your control.
Depends. Electronics manufactures won't promise any CPU will be available for that long (Odds are someone reading this should be thinking I'm partially responsible for that policy in my company). When spare parts are not going to be made the companies will try to predict how many will be needed in the future and order that many. This will be your biggest limit: you need a part there is a good chance the part is failing more often than expected and so the manufacture is out of stock and they cannot get more. (or sometimes they can, at a cost of $1000 each, minimum order 10,000 - the manufactures want to sell the new shiny replacement even though it isn't compatible)
Even if the above isn't an issue, storing electronics not on a circuit board is not easy, but I'm probably not supposed to talk about those issues. (You can research them)
Even if the above isn't an issue, storing electronics not on a circuit board is not easy, but I'm probably not supposed to talk about those issues. (You can research them)
Prius's routinely get to 300-400,000 miles. The used market is full of 10+ year old Priuses with 200k+. You may have to replace a transaxle and hybrid battery, but they are $1k each. Part of it is Toyota, part of it is the Atkinson cycle engine.
>hybrid battery
This isn't a trivial cost
This isn't a trivial cost
No, it's not, but all cars require maintenance and you should know what they are an amortize them over the time you expect to have a car. For the added car life usage you should get on either Prius component, it is a bargain.
I don't know how much it actually costs, but taking the numbers here at face value, $1000 over 200,000 miles is half a penny a mile - that is clearly trivial compared to running costs of any car. One way to look at it, is if you assume $3/gallon gas and 50 mpg, then the cost of the battery is equivalent to reducing the gas mileage to 46 mpg, all else being equal.
Sure it makes sense amortized,
but it's the equivalent of buying a gas engine that needs a rebuild.
It's a significant up front cost to consider against a purchase price.
It's a significant up front cost to consider against a purchase price.
its ~$1K, or less if its just one faulty module https://www.youtube.com/watch?v=Q3RCdrh666w
When the alternator in my 1983 Mercedes 300D quit, I started the car off a jump and drove it dead-electric to the store for a replacement part.
I believe the only semiconductors in the car are in the timing circuits for indicators and wipers, the tach and fuel gauge amps, and the radio.
Past results do not guarantee future performance: no new cars are this simple.
I believe the only semiconductors in the car are in the timing circuits for indicators and wipers, the tach and fuel gauge amps, and the radio.
Past results do not guarantee future performance: no new cars are this simple.
I once drove my '76 300D for more than a week with no electricity whatsoever. No generator, no battery. Hill started it or borrowed a fresh battery when I needed to start it and let it run until I was done with it.
Electricity was strictly optional on those cars.
Electricity was strictly optional on those cars.
Up until around 89 (I think) you could have even tow started it! The instructions are even in the manual. They are incredible machines.
1983 Mercedes Diesel has electronic fuel injection and ABS.
ABS yes, but it’s a Bosch mechanical pump. There’s a reason for the manual stop lever on it! There is an “electronic diesel system” that feeds into the ALDA system, but the car will run without ALDA. I don’t think the 300D had EDS though.
If there is ever a nuclear apocalypse, the giant mutant cockroaches that survive will be driving around in style in 80’s diesel Mercs.
If there is ever a nuclear apocalypse, the giant mutant cockroaches that survive will be driving around in style in 80’s diesel Mercs.
My Volvo S60 is getting 20 years in 3 month. Many premium cars from the 1990s are getting to the 30 year mark, which will mean they are getting a historic car badge. Most of them still work. Only some with failed sealings need replacements.
I have a 2005 Accord which has been quite reliable, but it is an LX and doesn't have much of an infotainment system. I'm worried about repair of things like the video cameras, LCD screens, and infotainment systems that are in the new cars once a decade or more has passed. Cars are a pretty hostile environment for those systems.
Don't forget speakers!
Old cars with four or six speakers can have all their speakers replaced with off-the-shelf models pretty easily, but I expect replacing all 15 speakers in a Model 3 in a dozen years to be fairly nightmarish.
Comparison of some old car speakers: https://www.crutchfield.com/S-FPB2k6hbguH/learn/mad-science-...
Old cars with four or six speakers can have all their speakers replaced with off-the-shelf models pretty easily, but I expect replacing all 15 speakers in a Model 3 in a dozen years to be fairly nightmarish.
Comparison of some old car speakers: https://www.crutchfield.com/S-FPB2k6hbguH/learn/mad-science-...
If you like niche cars, the Mini Cooper line has dealerships with replacement parts going back to at least the 2001 models (and probably can custom-order stuff for earlier models), and an endearingly nutty online fanbase that sells both custom aftermarket parts/upgrades and original OEM parts for both the 'modern' BMW-made line and for the classic cars.
The word online is that these cars take a lot of maintenance, but I haven't had any trouble with mine (2012 model, 90K miles, original electronics) aside from needing a new fan belt and a new battery. And of course the Countryman line is basically just a BMW SUV with the Mini branding and dealership network.
If you want a vehicle that's less 'will the parts be available' and more 'will it just last forever', though, you can't go wrong with a Toyota Hilux.
The word online is that these cars take a lot of maintenance, but I haven't had any trouble with mine (2012 model, 90K miles, original electronics) aside from needing a new fan belt and a new battery. And of course the Countryman line is basically just a BMW SUV with the Mini branding and dealership network.
If you want a vehicle that's less 'will the parts be available' and more 'will it just last forever', though, you can't go wrong with a Toyota Hilux.
I bought a 2000 Toyota 4Runner a few years ago at around 160k miles, put a few thousand dollars into required routine maintenance, and it runs great.
Sale price plus maintenance divided by time owning it currently has me at $275/month and dropping.
Sale price plus maintenance divided by time owning it currently has me at $275/month and dropping.
I feel like airbag-era cars are time bombs, so I have to reluctantly pass on a lot of cars I otherwise would enjoy.
You can argue the risk comparison between an old airbag and no airbag, but the thing that sways me is, beyond a certain point, the manufacturer just doesn't care if it works or not. When 95% of the cars have been scrapped, nobody cares if it doesn't work right anymore.
I also consider that on an old car without an airbag, I still have a seatbelt that I will wear, and on an old car with an airbag, I still don't have modern crumple zones and reinforcements.
So I'm sticking with either relatively new vehicles or pre-airbag ones.
You can argue the risk comparison between an old airbag and no airbag, but the thing that sways me is, beyond a certain point, the manufacturer just doesn't care if it works or not. When 95% of the cars have been scrapped, nobody cares if it doesn't work right anymore.
I also consider that on an old car without an airbag, I still have a seatbelt that I will wear, and on an old car with an airbag, I still don't have modern crumple zones and reinforcements.
So I'm sticking with either relatively new vehicles or pre-airbag ones.
What's "a few years"?
$275/month is $3300pa, so you're already at a minimum of $6600 for a near 20 year old car, which doesn't sound particularly cheap.
I'm on a lower monthly cost of ownership for 3 years ownership of a 6 year old car.
$275/month is $3300pa, so you're already at a minimum of $6600 for a near 20 year old car, which doesn't sound particularly cheap.
I'm on a lower monthly cost of ownership for 3 years ownership of a 6 year old car.
I've owned it for 45 months. I'd like to get another 5 years out of it as a daily driver with minimal upkeep.
Also, it's not a car - it's an SUV - so it's going to cost more than a car. I bought it because I really wanted an old 4Runner, not because I wanted the cheapest car.
Also, it's not a car - it's an SUV - so it's going to cost more than a car. I bought it because I really wanted an old 4Runner, not because I wanted the cheapest car.
Car is a general word for 4 wheeled motorised personal transport? SUV is hardly more specific. A 2 wheel drive with slightly increased ride height is classed as an SUV these days.
Your mention of costs seemed to imply that it was cheap which didn't seem to me to be the case.
Your mention of costs seemed to imply that it was cheap which didn't seem to me to be the case.
Wow, OK, you win - you got a cheaper car than me. Congrats.
However, anecdote vs data:
"Used car payments hit record $400 per month as prices top $20,000" [0]
So I still think I've done pretty good with my purchase.
[0] https://www.usatoday.com/story/money/cars/2018/11/08/used-ca...
However, anecdote vs data:
"Used car payments hit record $400 per month as prices top $20,000" [0]
So I still think I've done pretty good with my purchase.
[0] https://www.usatoday.com/story/money/cars/2018/11/08/used-ca...
You said you wanted an old 4Runner, that's fair enough. I was interpreting it with more emphasis on the monthly price, rather than specific car, than perhaps you intended :)
We have two 03 vehicles that are going strong. The electronics are simple enough that I expect them to keep going unless there's ever water intrusion :) It's a Honda and a BMW for what it's worth.
Good process ensures good results: https://en.m.wikipedia.org/wiki/The_Toyota_Way
[deleted]
Toyota commit to 20 years of parts.
(I can probably still get new mechanical parts for my 23 year old Ipsum, but I doubt the in-car electronics are replaceable except from the second-hand market)
(I can probably still get new mechanical parts for my 23 year old Ipsum, but I doubt the in-car electronics are replaceable except from the second-hand market)
That's a problem.
The design life for the Ford EEC IV, used in 1980s Ford vehicles, was 30 years. My 35 year old Ford Bronco is still using the original electronics.
The design life for the Ford EEC IV, used in 1980s Ford vehicles, was 30 years. My 35 year old Ford Bronco is still using the original electronics.
I thought there were laws and regulations requiring automakers to reserve a certain number of spare parts for everything they make and support it for a period of time.
>Because a Tesla is highly dependent on its electronics, once the flash memory in a Tesla’s infotainment unit goes bad, it essentially bricks the entire car. Yikes!
Someone that has had a few classes in electronics should be able to replace this.
However, if it's similar to how Apple does it, the chip might be bound to the processor via a serial number or some other magic that's burned into the NAND at manufacturing, or OTP on the processor, or something else.
It's not just a matter of "oh just go get a Hynix NAND part from DigiKey and reball the PCB". At minimum you need to copy the data over. At worst you need to bless the part with whatever secret sauce Tesla is using at the factory.
It's not just a matter of "oh just go get a Hynix NAND part from DigiKey and reball the PCB". At minimum you need to copy the data over. At worst you need to bless the part with whatever secret sauce Tesla is using at the factory.
Good luck finding the part and not voiding your warranty
[deleted]
Unless your car has DRM that prevents you from tampering with it's flash memory...
It takes more than that to replace a BGA flash chip.
A BGA can be replaced with a freaking heat gun from Home Depot and some solder braid and some flux.
Source: I used to repair cable boxes for CTDI. While everyone else is using their Metcal station and wasting 5 minutes, I've got the BGA off, board deballed, and a new BGA package installed and ready for PKI in less than 45 seconds with a heat gun.
Source: I used to repair cable boxes for CTDI. While everyone else is using their Metcal station and wasting 5 minutes, I've got the BGA off, board deballed, and a new BGA package installed and ready for PKI in less than 45 seconds with a heat gun.
Not that much. BGA chips aren't that hard with a hot air station. I've done plenty of de-soldering and reflow using just a hot air gun.
Not really, you can learn to handle BGA in 4-7 days and $2-3K from some very best people in the business https://www.ipadrehab.com/index.cfm?Page=Practical-Board-Rep...
[deleted]
Seems a bit more complicated than just unsoldering the old chip and putting in a new one.
https://web.archive.org/web/20190621052637/http://teslaservi...
https://web.archive.org/web/20190621052637/http://teslaservi...
[deleted]
Cars are not "bricked" : this issue is well known and some Tesla hackers have solutions to fix. Rich rebuilds YouTube channel did a report on fixing flash cards on a Tesla : https://youtu.be/o-7b1waoj9Q?t=510
I really dislike how this issue is being presented in the headline and in quotes like this, because it is fundamentally no different than a critical part going bad in an ICE vehicle. The Tesla isn't "bricked" when this flash memory goes bad anymore than a Toyota is "bricked" when its transmission dies. You fix the faulty part and get the car back on the road.
That said, if there is a systemic problem with this part failing faster than it should, which certainly seems to be the case, Tesla should be pushed to either recall the affected vehicles or waive the service costs for those owners who are impacted. But that isn't any different than if there was a design flaw in a critical part of an ICE cars manufacturing.
That said, if there is a systemic problem with this part failing faster than it should, which certainly seems to be the case, Tesla should be pushed to either recall the affected vehicles or waive the service costs for those owners who are impacted. But that isn't any different than if there was a design flaw in a critical part of an ICE cars manufacturing.
In an ICE vehicle, something like a transmission is very expensive and takes up amazing amounts of space, and so any kind of redundancy is very difficult.
But we're well aware that flash memory degrades, corrupts and gets a bit screwy. Flash memory chips are also dirt cheap and quite small - couldn't Tesla of baked in a bit more redundancy in this case?
In our transmission example, any fault in the transmission is not catastrophic, but with electronics, a single bad joint or bridged trace can take the whole thing down.
But we're well aware that flash memory degrades, corrupts and gets a bit screwy. Flash memory chips are also dirt cheap and quite small - couldn't Tesla of baked in a bit more redundancy in this case?
In our transmission example, any fault in the transmission is not catastrophic, but with electronics, a single bad joint or bridged trace can take the whole thing down.
I already implied that this is a design flaw and stated clearly that Tesla should be responsible for fixing this (and it appears they did things to mitigate if not eliminate this flaw when they redesigned the MCU).
I am not defending Tesla here. I am simply pointing out that this is being covered in a fundamentally different way than other cars. My last car was recalled twice for electrical wiring issues. One of the problems could have resulted in a fire and the other one would have resulted in the exact same "bricking" that you see here. Neither of those issues would be covered or dissected like this Tesla issue.
I am not defending Tesla here. I am simply pointing out that this is being covered in a fundamentally different way than other cars. My last car was recalled twice for electrical wiring issues. One of the problems could have resulted in a fire and the other one would have resulted in the exact same "bricking" that you see here. Neither of those issues would be covered or dissected like this Tesla issue.
> The Tesla isn't "bricked" when this flash memory goes bad anymore than a Toyota is "bricked" when its transmission dies.
That's why Toyota isn't releasing updates to transmissions every few weeks, that are changing it's behavior and using off the shelf parts without proper attestation and testing in all possible configurations.
Tesla moves fast, which is great for progress. But extremely risky for long term reliability of complex machines.
That's why Toyota isn't releasing updates to transmissions every few weeks, that are changing it's behavior and using off the shelf parts without proper attestation and testing in all possible configurations.
Tesla moves fast, which is great for progress. But extremely risky for long term reliability of complex machines.
And why should "the flash memory in a Tesla’s infotainment unit" be a "critical part"? This seems like the opposite of a critical part.
That would be like if the CD player in my ICE car broke, suddenly I can't drive my car. Absurd.
That would be like if the CD player in my ICE car broke, suddenly I can't drive my car. Absurd.
>That would be like if the CD player in my ICE car broke
This is not an apt analogy at all. In a Tesla, there is no separate dashboard or control so it's not an optional part like a CD player. It would be more like saying that your transmission failed. It just happens that, because it doesn't need all the other mechanical parts, an EV's "transmission" is the same box that runs the radio, AC, etc.
This is not an apt analogy at all. In a Tesla, there is no separate dashboard or control so it's not an optional part like a CD player. It would be more like saying that your transmission failed. It just happens that, because it doesn't need all the other mechanical parts, an EV's "transmission" is the same box that runs the radio, AC, etc.
Well that is a massive engineering fail to decide to make the "infotainment" system tightly coupled with critical car functionality.
A good engineering team would have decoupled them. In an ICE car half of your dashboard could be non functional (climate control completely broken, etc) and the car will still drive because the two systems are decoupled.
A good engineering team would have decoupled them. In an ICE car half of your dashboard could be non functional (climate control completely broken, etc) and the car will still drive because the two systems are decoupled.
>a massive engineering fail
It's not the infotainment part that causes the failure, though. It's the firmware flash chip which happens to be in the same board as the infotainment array. It's not like the radio breaking would cause the car to stop working. Again, this is more akin to the transmission going out. It's just easy to make this sound like an engineering failure because of that. It's similar to saying that the new Google Pixel has a "massive engineering failure" because the "Ok, Google" voice assistant breaks. It's disingenuous to say that because all that happens on the security module. You wouldn't want someone to have access to your phone if the security module breaks but, if that breaks, it also breaks the voice assistant functionality. This phrasing makes it sound like a fault in the voice assistant can block access to the entire phone which is intentionally misleading.
It's not the infotainment part that causes the failure, though. It's the firmware flash chip which happens to be in the same board as the infotainment array. It's not like the radio breaking would cause the car to stop working. Again, this is more akin to the transmission going out. It's just easy to make this sound like an engineering failure because of that. It's similar to saying that the new Google Pixel has a "massive engineering failure" because the "Ok, Google" voice assistant breaks. It's disingenuous to say that because all that happens on the security module. You wouldn't want someone to have access to your phone if the security module breaks but, if that breaks, it also breaks the voice assistant functionality. This phrasing makes it sound like a fault in the voice assistant can block access to the entire phone which is intentionally misleading.
It's a coupling engineering failure no matter how you spin it.
Diagnostic logging should be stored on a separate (re: decoupled) flash chip. If that flash chip wears out, the car should still drive.
Something is too tightly coupled right now, as evidenced by the existence of the article.
Diagnostic logging should be stored on a separate (re: decoupled) flash chip. If that flash chip wears out, the car should still drive.
Something is too tightly coupled right now, as evidenced by the existence of the article.
No, because Toyota does not use consumer-grade parts in cars. This eMMC (H26M42002GMR) is not even rated for automotive temp range! (flash wears out faster and retains data worse at higher temps)
> it is fundamentally no different than a critical part going bad in an ICE vehicle
The main difference is that the probability of a typical ICE ECU or TCU failing is much lower than a Linux-based infotainment system running on non automotive grade flash.
The main difference is that the probability of a typical ICE ECU or TCU failing is much lower than a Linux-based infotainment system running on non automotive grade flash.
[deleted]
A failure in the infotainment will brick the entire car?! I thought they would be made up of individual systems talking to each other in messages. Tesla needs Erlang!
This is the case for many, if not most, modern/high-tech cars, especially EVs.
The on-board systems have become very tightly coupled, if not monolithic to the point where a broken info unit will prevent the car from starting.
I had the infotainment system replaced in my e-Golf a few weeks ago, it took two technicians three days to install and configure to work with all the car's on-board systems.
Edit: If I _had_ to guess, I’d say this is because the car industry (especially electric) has been pushing to innovate features so rapidly that the time hasn't been available to engineer, test and prove decoupled systems.
The on-board systems have become very tightly coupled, if not monolithic to the point where a broken info unit will prevent the car from starting.
I had the infotainment system replaced in my e-Golf a few weeks ago, it took two technicians three days to install and configure to work with all the car's on-board systems.
Edit: If I _had_ to guess, I’d say this is because the car industry (especially electric) has been pushing to innovate features so rapidly that the time hasn't been available to engineer, test and prove decoupled systems.
There's anti-theft considerations at play with how car electronics are coupled, reducing part count for cost savings, and I think a bit of the classic right to repair vs corporate interests dynamic happening. Automakers have been known to intentionally make things difficult to work on.
Yes, electronics are highly coupled, but in the case of the e-Golf it will at least function as a car with no infotainment system (unlike the Tesla). And MIB replacement in the VW MQB platform is simple so I don’t understand what took the techs so long. It’s one unit in the glovebox and guided coding using ODIS, the VW dealer tool (which is also available to independent shops albeit at a $1000/year fee, which is stupid and should be illegal). The VW units do have Component Protection which keeps them from simply plugging in, but with the aforementioned dealer tool, so long as it is legally obtained, a subscription to VW’s anti theft coding service Geko is already included and the process is very straightforward.
Why won't they think about these things? Do they really don't care or is it that we don't understand something very important in making cars?! I want to know !
It reduces cost and makes it easier to give a nice user experience when people are test driving. It makes it easier to manage all the different aspects of the car from a single interface.
The primary reason the car is made, in the first place, is to be sold at a profitable price. That is priority #1.
The primary reason the car is made, in the first place, is to be sold at a profitable price. That is priority #1.
Not exactly new to integrate multiple systems: My 2004 Land Cruiser has the heating/cooling and a few other less urgent systems integrated into the radio. Which means I can't replace it with a radio that supports bluetooth audio, without giving up heat/AC...
Same for my 2011 Explorer, and since it was such a PoS [1] the system would reboot it self while I was driving. When rebooting it would put the A/C/heat into halfway fan on 70 degree mode, whether you wanted it or not.
[1] - there's currently a class action settlement about how terrible it was
[1] - there's currently a class action settlement about how terrible it was
You can... it just means modifying the PCB for the controls extensively. I had a friend who did this on a Subaru once. Was a huge PITA.
> I thought they would be made up of individual systems talking to each other in messages.
That's how my 2011 BMW is. It's not exactly the most reliable car, but I'd imagine things would be much worse if it weren't designed that way.
For example, my body control module (FRM) just recently died, but the engine still runs and the car can still be driven, just none of the lights work.
The non-critical systems communicate with the powertrain bus through a special CAN gateway in an attempt to isolate the systems.
That's how my 2011 BMW is. It's not exactly the most reliable car, but I'd imagine things would be much worse if it weren't designed that way.
For example, my body control module (FRM) just recently died, but the engine still runs and the car can still be driven, just none of the lights work.
The non-critical systems communicate with the powertrain bus through a special CAN gateway in an attempt to isolate the systems.
> The Tesla firmware wasn't very big at the beginning, [...] the firmware has grown, leaving very little room for the logging to take place. That means individual sectors suffer from lots of data being written in a short amount of time, further accelerating the wear.
My understanding was modern flash memory wear levelling would swap regularly-written and seldom-written sectors around, allowing for even wear across the disk even if it was 95% full all the time. Is it established that Tesla doesn't have this feature?
My understanding was modern flash memory wear levelling would swap regularly-written and seldom-written sectors around, allowing for even wear across the disk even if it was 95% full all the time. Is it established that Tesla doesn't have this feature?
I'm not sure exactly what eMMC part they are using, but the SDHC spec doesn't explicitly require wear leveling. A part could implement it, but it's not a given.
It does, but the article says that the car logs a lot of data and so consequently all sectors get wear. See here: https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issu...
It's also possible that older Tesla vehicles do not have modern flash memory.
That would be bad for this use case?
I want the firmware working, I don't much care if data logging stops working.
I want the firmware working, I don't much care if data logging stops working.
While I might expect an SSD to use such a strategy, an embedded flash chip doing so is just as likely to be a misfeature, because when the flash does fail you may lose data other than that already being written, which makes it impossible to keep the system even limping along without any logging.
In case anyone's interested, here's a TMC thread with good discussion on it.
https://teslamotorsclub.com/tmc/threads/preventive-emmc-repl...
And a doc the OP put together on replacement.
https://docs.google.com/document/d/1ZH8oP4AgdVxmCN0saKA1Dxc2...
https://teslamotorsclub.com/tmc/threads/preventive-emmc-repl...
And a doc the OP put together on replacement.
https://docs.google.com/document/d/1ZH8oP4AgdVxmCN0saKA1Dxc2...
> Today, it looks like Tesla will fix the problem if you're within the warranty, and outside-of-warranty repairs can cost $1,800 to $3,000
Any other car manufacturer would issue a recall notice and repair it for free, regardless of warranty status.
I smell a class action lawsuit -- Tesla installed software that invisibly destroys user-inaccessible hardware throughout the entire warranty process and then fails afterward, and then charged people to fix it.
Any other car manufacturer would issue a recall notice and repair it for free, regardless of warranty status.
I smell a class action lawsuit -- Tesla installed software that invisibly destroys user-inaccessible hardware throughout the entire warranty process and then fails afterward, and then charged people to fix it.
"Any other car manufacturer would issue a recall notice and repair it for free, regardless of warranty status."
I don't buy this one bit. E.g. GM ignition switches, Ford Focus transmissions, LR coolant crossover pipes. They don't do this by default, and mostly get away with it.
I don't buy this one bit. E.g. GM ignition switches, Ford Focus transmissions, LR coolant crossover pipes. They don't do this by default, and mostly get away with it.
[deleted]
I doubt any car company would recall this. There’s plenty of production cars with flaws that completely destroy the engine that were never recalled, like the north star engine Cadillac used.
The Cadillac "High Tech" V8 used in the early 80's was even worse for that...
North Star is sealed for life!
Previous discussion:
https://news.ycombinator.com/item?id=19912065
https://news.ycombinator.com/item?id=19912065
This article is unreadable due to the ads
What ads?
Adblock on the web is equalivalent to antivirus on Windows.
If you're not running it, you absolutely should be, precisely because it is a security issue.
If you're not running it, you absolutely should be, precisely because it is a security issue.
serious question: do people still run antivirus on windows other than MSFT's free one?
No, for a very long time.
Build-in Microsoft stuff will get you way further, that's also why anti-virus apps are now bloatware and spyware on their own.
Ps. Downvoted above you because it really is just a terrible example.
It's also a lie recent years.
Build-in Microsoft stuff will get you way further, that's also why anti-virus apps are now bloatware and spyware on their own.
Ps. Downvoted above you because it really is just a terrible example.
It's also a lie recent years.
At work, yes. It's a management CYA requirement.
I don't, it's bloatware anymore and always trying to upsell you some other service.
Of course there are absurd ads. As adblocking gets more mainstream and less technicalml, the users left to monetize are less skilled and less valuable. Meaning publishers are stuck trying to monetize a smaller slice of their total traffic, that's also worth less. Leaving only more advertising "mass" as a way to get the same income.
Especially annoying since this article is really a retelling of another article [1], which is itself a retelling of a forum thread [2].
[1] https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issu...
[2] https://teslamotorsclub.com/tmc/threads/likely-mcu-failure.1...
[1] https://insideevs.com/news/376037/tesla-mcu-emmc-memory-issu...
[2] https://teslamotorsclub.com/tmc/threads/likely-mcu-failure.1...
Which is why any flash drive that is regularly written to, should be exchangeable. Be it a car or a laptop. Putting things on SD or M.2 seems like a much more robust solution.
As a suggestion for a general re-design to solve this issue...
Cluster the consumable components together. Flash and battery. Attach the storage via iSCSI or something similar, have the car's main computer perform encryption/decryption (rather than trusting a component expected to be swapped).
In the case of a battery swap the bulk contents of the (encrypted) external storage can also be cloned. That can probably happen in the timespan that a battery is topped up to charge, but would be more difficult for automated swaps at refueling points. A home/cloud backup and transferring only the new data might be sufficient.
Cluster the consumable components together. Flash and battery. Attach the storage via iSCSI or something similar, have the car's main computer perform encryption/decryption (rather than trusting a component expected to be swapped).
In the case of a battery swap the bulk contents of the (encrypted) external storage can also be cloned. That can probably happen in the timespan that a battery is topped up to charge, but would be more difficult for automated swaps at refueling points. A home/cloud backup and transferring only the new data might be sufficient.
> Today, it looks like Tesla will fix the problem if you're within the warranty, and outside-of-warranty repairs can cost $1,800 to $3,000, depending on your location. Tesla’s method is to replace the entire MCU.
This is ridiculous. Can we all agree that this is ridiculous? Because that's what it is. If we can at least all agree that this is indeed ridiculous, perhaps the collapse of modern civilization can be avoided.
This is ridiculous. Can we all agree that this is ridiculous? Because that's what it is. If we can at least all agree that this is indeed ridiculous, perhaps the collapse of modern civilization can be avoided.
Which part? The price? Yes, it is ridiculous.
The method of replacing entire MCU? Maybe not, flash is likely not the only component in there that has an age limit.
The method of replacing entire MCU? Maybe not, flash is likely not the only component in there that has an age limit.
Replacing the entire MCU. The price for that isn't necessarily ridiculous.
> Maybe not, flash is likely not the only component in there that has an age limit.
The MCU itself should outlast the lifetime of the car in the vast majority of the cases.
> Maybe not, flash is likely not the only component in there that has an age limit.
The MCU itself should outlast the lifetime of the car in the vast majority of the cases.
> With electric vehicles now on the market in masses and coming of age, we’re starting to see the real culprits of aging
The issue with the flash memory resides in the MCU (media control unit), which has nothing to do with the electric drivetrain. Cars with combustion engines can have this issue too.
> once the flash memory in a Tesla’s infotainment unit goes bad, it essentially bricks the entire car.
Essentially, yes. But as far as I am aware (correct me if i'm wrong) a Tesla with a bricked MCU will still drive.
I am annoyed that articles like this make it sound that these types of faults are only happening to electric cars, which it is not. It might be a Tesla issue, as they are both ambitious and inexperienced in car component design, but it could have happened with other brands as well.
> Fortunately, Jason Hughes, known as a ‘Tesla hacker,’ can service these units at a much lower cost.
Yes, many people can, there is no secret technique to it. Just like a local garage can fix any part with any brand of car for much cheaper than the dealership. That is because dealerships have to replace parts, rather than repair them. There are many good reasons why they do this, I wont go into details here.
Sure, a conventional car mechanic probably can't do it, but anyone with electronic repair experience can. That is a shift that is happening with all car brands. As cars are shifting towards computers on wheels, we must learn to distinguish mechanical repairs from electronic repairs.
The issue with the flash memory resides in the MCU (media control unit), which has nothing to do with the electric drivetrain. Cars with combustion engines can have this issue too.
> once the flash memory in a Tesla’s infotainment unit goes bad, it essentially bricks the entire car.
Essentially, yes. But as far as I am aware (correct me if i'm wrong) a Tesla with a bricked MCU will still drive.
I am annoyed that articles like this make it sound that these types of faults are only happening to electric cars, which it is not. It might be a Tesla issue, as they are both ambitious and inexperienced in car component design, but it could have happened with other brands as well.
> Fortunately, Jason Hughes, known as a ‘Tesla hacker,’ can service these units at a much lower cost.
Yes, many people can, there is no secret technique to it. Just like a local garage can fix any part with any brand of car for much cheaper than the dealership. That is because dealerships have to replace parts, rather than repair them. There are many good reasons why they do this, I wont go into details here.
Sure, a conventional car mechanic probably can't do it, but anyone with electronic repair experience can. That is a shift that is happening with all car brands. As cars are shifting towards computers on wheels, we must learn to distinguish mechanical repairs from electronic repairs.
> It might be a Tesla issue, as they are both ambitious and inexperienced in car component design, but it could have happened with other brands as well.
For me this is a trend Tesla has been showing. (e.g. https://www.thedrive.com/tech/27989/teslas-screen-saga-shows... stuff is trying to use off the shelf but aren't designing for what their cars will actually be experiencing. It -could- happen to other brands, but it is happening with Teslas. This combined with their reported code practices (https://twitter.com/atomicthumbs/status/1032939617404645376) generally put in the camp of recommending people lease, don't buy and definitely don't buy used. These first/second gen Teslas are going to be trash is a few more years.
For me this is a trend Tesla has been showing. (e.g. https://www.thedrive.com/tech/27989/teslas-screen-saga-shows... stuff is trying to use off the shelf but aren't designing for what their cars will actually be experiencing. It -could- happen to other brands, but it is happening with Teslas. This combined with their reported code practices (https://twitter.com/atomicthumbs/status/1032939617404645376) generally put in the camp of recommending people lease, don't buy and definitely don't buy used. These first/second gen Teslas are going to be trash is a few more years.
It's common for Tesla fans to mock other automotive manufacturers for being "old and slow" - as if they're incompetent because they take so long to develop new features and adapt to changes in technology. It's like a "why does it take 5 years for BMW to update their infotainment system when I can do the same thing with a Raspberry Pi in 5 minutes?" mentality.
What I think goes unappreciated by some is the level of testing that normally goes into automotive grade development.
That's not to say that modern ICE cars are more reliable than a Tesla, but their electronics are usually designed for a higher level of robustness.
Here are my issues:
- Why is the car rendered useless if the infotainment system fails? This should be a non-critical component. If it is critical and the critical part can't be isolated, then the system as a whole should have been tested more extensively.
- Why does the infotainment system fail if a log message write fails, and why was this failure mode not tested? It is very common knowledge that flash wears out, and log messages aren't critical.
- Why were they not using automotive grade flash?
This, coupled with the recent story about the non-automotive-grade displays failing really demonstrates that maybe, just maybe, the rest of the automotive industry isn't totally clueless. Maybe they're just more scared of a failure.
Who knows - it's probably a mixture of both.