The Migration Of Engine ECU Software From Single-Core To Multi-Core(semiengineering.com)
semiengineering.com
The Migration Of Engine ECU Software From Single-Core To Multi-Core
https://semiengineering.com/the-migration-of-engine-ecu-software-from-single-core-to-multi-core/
24 comments
How do they plan to get through ISO 26262 verification? You do not need to do HARA to know ECU is ASIL D. ASIL D Item requires developer to do Modified Condition and Decision Coverage. Running through all the test cases will be a huge task. The golden rule for Automobile Software is simplicity. Always keep it simple unless it is absolutely necessary. They need to read through ISO 26262 chapter 6.
Well that's horrifying. Not where I want multithreaded code. Definite risk of it being written in C with volatile~=atomic as the working assumption.
People found the concept of computer control over an idle stabilization valve terrifying in the early 80's. Killer cars, everyone predicted. It never materialized. The lawsuits against Audi never went anywhere, and Audi even won some counter-suits.
Most cars have drive-by-wire throttles. Hundreds of millions of cars out there, and DBW throttle failures leading to crashes are unheard of.
You know what is an issue? Ford unable to make ignition switches that don't cause cars parked in house garages to catch fire. GM cheaping out on their own ignition switches, causing them to suddenly fail while driving....and thus shutting off the engine, ABS, and airbag systems, something that is known to have caused the deaths of half a dozen people and likely tens if not hundreds of other Cruze owners.
Most cars have drive-by-wire throttles. Hundreds of millions of cars out there, and DBW throttle failures leading to crashes are unheard of.
You know what is an issue? Ford unable to make ignition switches that don't cause cars parked in house garages to catch fire. GM cheaping out on their own ignition switches, causing them to suddenly fail while driving....and thus shutting off the engine, ABS, and airbag systems, something that is known to have caused the deaths of half a dozen people and likely tens if not hundreds of other Cruze owners.
> Most cars have drive-by-wire throttles.
Cars where? Many cars here in Europe have a valve directly mechanically linked to the pedal. Any car without cruise control has it that way.
Cars where? Many cars here in Europe have a valve directly mechanically linked to the pedal. Any car without cruise control has it that way.
I don’t have a source for this but I recall reading that mechanical throttle wires are actually not especially reliable so drive-by-wire doesn’t have to be super reliable in order to be better.
But perhaps it’s the same as self driving cars where people get especially disturbed by computers screwing up which sets the reliability bar higher than is otherwise logical.
But perhaps it’s the same as self driving cars where people get especially disturbed by computers screwing up which sets the reliability bar higher than is otherwise logical.
Have you ever had either fail on you? I never have. I have heard of a few failures but always on very modified or old vehicles. Some failure modes can happen whether actuated by wire or motor as well.
it isn’t a first order concern really, dozens of higher risk, more complex things going on like tires, brakes, explosives in the steering wheels.
it isn’t a first order concern really, dozens of higher risk, more complex things going on like tires, brakes, explosives in the steering wheels.
Anecdata isn't helpful here; when you are one of millions of users, thousands of people may be inconvenienced (or killed) before you have personal experience with a problem.
>DBW throttle failures leading to crashes are unheard of
I doubt Michael Hastings crash would have happened if he didn't drive by wire.
I doubt Michael Hastings crash would have happened if he didn't drive by wire.
I've been reverse engineering a modern ABS control module. It has 2 cores, and they are separate CPU's with their own memory (although one cpu can access the RAM of the other CPU) and role. One reads/writes the CAN bus and does verification on that data, the other CPU is the one that monitors wheel speed sensors and determines when to control the solenoid valves. There is a cleanly defined area where data is copied from one CPU to the other.
While I agree with you in general about being careful about the pitfalls of multi-threading, the automotive industry is fairly competent at safety and correctness. They are probably the only few industry making good use of formal verification. In short, I am less worried about cars not working due to misunderstanding of C language than any other device like my PC or iPhone
Can anyone enlighten me why you need multi thread to control something of much slower frequency than a CPU? I think there is more to be gained in the sensors than in a faster ECU. Of course I’m only speaking about Engine control, not all the other stuff in the car.
Full disclosure, I'm an insider in the industry. I'd like to comment on a few trends I've been seeing.
First, ECUs are now running a general purpose OS rather than a single embedded application. That means compartmentalization of functionality into separate, restartable processes (a program crash is considered a safe failure mode), which is a very good thing. Most synchronization is performed at the low level by the safety-certified OS kernel using very well-known well-understood and well-tested mechanisms, so you don't have to worry about non-experts writing the bad C code so often lambasted in religious screeds. A big advantage is that the software components can be developed, tested, and verified in isolation all using proven-in-use techniquies on a development host before ever finding their way to a target system. The tradeoff is much higher overhead, which with todays processors is effectively noise that disappears into the background.
Second, the CPUs are so fast that manufacturers have developed tickless mode in which most of the silicon gets put to sleep. ECUs these days do much much more nothing faster and in parallel. Things like bigLITTLE make it end up that only 1 32-bit core is actually doing anything almost all the time. So cheaper, less cost, but available when needed. Pretty much all of this is hidden by the OS, so don't worry about your storied ignorant C developer not understanding how to control their software.
Third, somebody not close to the silicon but close to the ledger books looks at all this glorious CPU and memory on an ECU and realizes there are unused resources being paid for and going to waste. They realize they can stick logos on their product marketing material if they start using those spare resources for AI technologies. Yes folks, some of those ECUs are starting to crunch numbers using Fortran libraries driven by Python. From a consumer point of view it's great because the software can now make up for bad hardware dynamically, adaptively, and in real time. From a functional safety engineering point of view, as long as all hazards are mitigated and all failure modes lead to a safe state, it's not a bad thing.
Yes, the silicon in control of your truck is getting bigger, wider, and faster is blooming multiple cores. The software development for it is taking advantage of 70 years of evolution to match. Sit back and enjoy the ride.
First, ECUs are now running a general purpose OS rather than a single embedded application. That means compartmentalization of functionality into separate, restartable processes (a program crash is considered a safe failure mode), which is a very good thing. Most synchronization is performed at the low level by the safety-certified OS kernel using very well-known well-understood and well-tested mechanisms, so you don't have to worry about non-experts writing the bad C code so often lambasted in religious screeds. A big advantage is that the software components can be developed, tested, and verified in isolation all using proven-in-use techniquies on a development host before ever finding their way to a target system. The tradeoff is much higher overhead, which with todays processors is effectively noise that disappears into the background.
Second, the CPUs are so fast that manufacturers have developed tickless mode in which most of the silicon gets put to sleep. ECUs these days do much much more nothing faster and in parallel. Things like bigLITTLE make it end up that only 1 32-bit core is actually doing anything almost all the time. So cheaper, less cost, but available when needed. Pretty much all of this is hidden by the OS, so don't worry about your storied ignorant C developer not understanding how to control their software.
Third, somebody not close to the silicon but close to the ledger books looks at all this glorious CPU and memory on an ECU and realizes there are unused resources being paid for and going to waste. They realize they can stick logos on their product marketing material if they start using those spare resources for AI technologies. Yes folks, some of those ECUs are starting to crunch numbers using Fortran libraries driven by Python. From a consumer point of view it's great because the software can now make up for bad hardware dynamically, adaptively, and in real time. From a functional safety engineering point of view, as long as all hazards are mitigated and all failure modes lead to a safe state, it's not a bad thing.
Yes, the silicon in control of your truck is getting bigger, wider, and faster is blooming multiple cores. The software development for it is taking advantage of 70 years of evolution to match. Sit back and enjoy the ride.
Many of the benefits that you list are for manufacturers, not consumers.
My 1990s car has an ECU that only has one job, and runs fully-characterized code. I'd trust it to run another 25 years with no problems, firmware updates, or security patches.
I do not trust something running a full-fledged OS to drive something safety-critical; I cannot imagine that the whole system is fully vetted, and I don't want bugs in the chip that is in charge of my engine.
And now they're planning to run ML algorithms on the ECU to use all the extra compute that they never needed in the first place? That sounds like an absolute nightmare.
Based on what you've written, I am very glad that I'll probably never buy a new ICE car ever again.
My 1990s car has an ECU that only has one job, and runs fully-characterized code. I'd trust it to run another 25 years with no problems, firmware updates, or security patches.
I do not trust something running a full-fledged OS to drive something safety-critical; I cannot imagine that the whole system is fully vetted, and I don't want bugs in the chip that is in charge of my engine.
And now they're planning to run ML algorithms on the ECU to use all the extra compute that they never needed in the first place? That sounds like an absolute nightmare.
Based on what you've written, I am very glad that I'll probably never buy a new ICE car ever again.
The "E" in ECU sometimes is eletronic and sometimes engine, depending on the sub-field and region of the automotive industry.
For engine control (the topic of the original article) a general-purpose Os and lots of idle time are unusual. In many cases the most powerful controller available on the market is is filled with software just shy of losing tasks.
For engine control (the topic of the original article) a general-purpose Os and lots of idle time are unusual. In many cases the most powerful controller available on the market is is filled with software just shy of losing tasks.
> Third, somebody not close to the silicon but close to the ledger books looks at all this glorious CPU and memory on an ECU and realizes there are unused resources being paid for and going to waste. They realize they can stick logos on their product marketing material if they start using those spare resources for AI technologies. Yes folks, some of those ECUs are starting to crunch numbers using Fortran libraries driven by Python. From a consumer point of view it's great because the software can now make up for bad hardware dynamically, adaptively, and in real time. From a functional safety engineering point of view, as long as all hazards are mitigated and all failure modes lead to a safe state, it's not a bad thing.
That's gross. Like that hardware uses no power, its purpose is to be at attention every foot of the journey. That's waiting, which in western culture means "slacking off", but in other cultures, like African cultures, is...well I don't know quite, but I think it's neither work nor rest.
It's like going to a restaurant, you want the waiter to wait on you, you do your thing and when you need him he'll be ready. Maybe you'll need him often, or not even once. Doesn't matter, he still waited on you. Doesn't have to be moving all his arms and legs every second from the minute you walk in to the minute you leave, if he's ready to help when you need him, it's much more useful.
Ultimately for tickless chips to be work there has to be a change in mentality that respects waiting as unique, that values availability without requiring effort.
I guess it goes back to the idea in the west that values effort in different ways, rather than results. Effort meaning personal cost, too. Well when I was an employee I'd say they're both valued, equally, in the following way: you get paid according to the minimum of the two. Good results? Yeah but that only took you an hour, that only cost you $20, "don't complain." Working really hard but not achieving the goals? We're going to have to fire you and rehire at a lower wage that you can actually bring in. And that's why employees are so "resentful" about working, they're not, it's just an optimization function, if they're producing good value they have to look like they're working hard or there's a big cost to them of producing it, that's the resentful part, otherwise they get cheated. And further, it explains forfeiting results for the business all the time, if they're maxed out on the effort part they won't make a single fucking cent from it. Even if it costs nothing, bad idea, no working for free because then the floodgates of exploitation open. The employee does not want to work for an even richer, and therefore more powerful, employer if that employer is exploitative.
That's gross. Like that hardware uses no power, its purpose is to be at attention every foot of the journey. That's waiting, which in western culture means "slacking off", but in other cultures, like African cultures, is...well I don't know quite, but I think it's neither work nor rest.
It's like going to a restaurant, you want the waiter to wait on you, you do your thing and when you need him he'll be ready. Maybe you'll need him often, or not even once. Doesn't matter, he still waited on you. Doesn't have to be moving all his arms and legs every second from the minute you walk in to the minute you leave, if he's ready to help when you need him, it's much more useful.
Ultimately for tickless chips to be work there has to be a change in mentality that respects waiting as unique, that values availability without requiring effort.
I guess it goes back to the idea in the west that values effort in different ways, rather than results. Effort meaning personal cost, too. Well when I was an employee I'd say they're both valued, equally, in the following way: you get paid according to the minimum of the two. Good results? Yeah but that only took you an hour, that only cost you $20, "don't complain." Working really hard but not achieving the goals? We're going to have to fire you and rehire at a lower wage that you can actually bring in. And that's why employees are so "resentful" about working, they're not, it's just an optimization function, if they're producing good value they have to look like they're working hard or there's a big cost to them of producing it, that's the resentful part, otherwise they get cheated. And further, it explains forfeiting results for the business all the time, if they're maxed out on the effort part they won't make a single fucking cent from it. Even if it costs nothing, bad idea, no working for free because then the floodgates of exploitation open. The employee does not want to work for an even richer, and therefore more powerful, employer if that employer is exploitative.
CPUs in the automotive space are primarily chosen for being cheap, and with the right number/kind of IO. If you do enough stuff¹, it's certainly possible to reach the CPU load limits of a single core CPU, and it might cheaper to upgrade to a dual core than put in another ECU (also, faster ECUs with the same IO/arch might not be available).
I've also seen arrangements where the dual-cores have separate memory spaces, and one core acts like a safety processor/monitor while the other is the main compute module.
1: examples of CPU intensive stuff include active damping, active aero (spoilers), active rear steering, steer by wire, etc. You may think these things are irrelevant w.r.t. an engine ECU, but if the engine ECU is responsible for producing torque, then systems like traction control, stability control and etc are all directly related to torque production
I've also seen arrangements where the dual-cores have separate memory spaces, and one core acts like a safety processor/monitor while the other is the main compute module.
1: examples of CPU intensive stuff include active damping, active aero (spoilers), active rear steering, steer by wire, etc. You may think these things are irrelevant w.r.t. an engine ECU, but if the engine ECU is responsible for producing torque, then systems like traction control, stability control and etc are all directly related to torque production
I've worked in the field, even with the biggest and fastest ECU's in existance, Formula 1, with 10Khz cycles on 4Ghz CPU's
we never saw the need to use the other cores. the CPU is a few million times faster than the cycles, if there is a problem it's a HW IO or blocking driver. even with normal 500Hz ECU's you can always upgrade the CPU freq to run faster, than needing more cores. even better would be to use better drivers with zero copy and MMU access.
it's nice that threading got better recently in embedded Realtime OS's, HW always gets better, but certainly drivers not and SW not. and developers even less. maybe they think rust will help, but then they should chose a proper concurrency-safe system. certainly not rust, which only lies about it.
also this thing is only simulation verified, not formally verified.
we never saw the need to use the other cores. the CPU is a few million times faster than the cycles, if there is a problem it's a HW IO or blocking driver. even with normal 500Hz ECU's you can always upgrade the CPU freq to run faster, than needing more cores. even better would be to use better drivers with zero copy and MMU access.
it's nice that threading got better recently in embedded Realtime OS's, HW always gets better, but certainly drivers not and SW not. and developers even less. maybe they think rust will help, but then they should chose a proper concurrency-safe system. certainly not rust, which only lies about it.
also this thing is only simulation verified, not formally verified.
> it's nice that threading got better recently in embedded Realtime OS's, HW always gets better, but certainly drivers not and SW not.
I hate to break it to you, but hardware is basically always broken, and it's all "fixed" in software (especially in embedded, where some vendors don't even bother with steppings to fix errata).
I hate to break it to you, but hardware is basically always broken, and it's all "fixed" in software (especially in embedded, where some vendors don't even bother with steppings to fix errata).
which systems would you consider "proper concurrency-safe" ?
Not GP, but something generated by CoPilot (https://hackage.haskell.org/package/copilot) would generate constant-time constant-memory C code from a specification written in a DSL. That is what I would start to consider "proper" safe for stuff where it is really important.
It depends on what you consider to be absolute requirements for proper concurrency safety of course, and how much of the hardware is allowed to fail before it starts impacting the software.
It depends on what you consider to be absolute requirements for proper concurrency safety of course, and how much of the hardware is allowed to fail before it starts impacting the software.
From 2014-2017 I have participated in a large European subsidiary project [0] that focused on the major industrial challenge arising from the need to face cost efficient integration of different applications with different levels of safety and security on a single computing platform. Use of multi-core fpgas and the like in aviation, automative and medical industry is an enormous challenge. Mixed-critical, multi-core programming models would mostly make sure critical tasks would run entirely on their isolated core and registers leaving the remainder for non-critical tasks. That solved being able to buy common hardware but did not allow programs to actually benefit from multi-core boards.
[0] http://www.emc2-project.eu/
[0] http://www.emc2-project.eu/
Few have demonstrated actual competency with using 1 computer. I struggle to comprehend a reality in which auto manufacturers or their vendors are able to manage more than 1 without causing net harm to themselves or society.
The only arguments for more compute power in cars boil down to distracting the driver or making them feel like they are somewhat less responsible for the laws of physics.
The only arguments for more compute power in cars boil down to distracting the driver or making them feel like they are somewhat less responsible for the laws of physics.