Toyota suspends domestic factory operations after suspected cyber attack(reuters.com)
reuters.com
Toyota suspends domestic factory operations after suspected cyber attack
https://www.reuters.com/business/autos-transportation/toyota-suspends-all-domestic-factory-operations-after-suspected-cyber-attack-2022-02-28/
79 comments
I would hope most of us have been taking it seriously well-before this incident.
We've been doing B2B business for banks and we eschew any cloud/remote infrastructure as part of our offering because no one would pay for it. Everyone demands on-prem hosting of our software and it has to exclusively flow through their network security appliances.
I had a conversation with more than one CIO who would rather suffer arbitrary DDOS attacks than allow cloudflare the ability to decrypt their application traffic.
I am more than happy to work with these kinds of customers. There is no excuse to compromise on security when the stakes are this high. Everyone is willing to take their time to get it right.
We've been doing B2B business for banks and we eschew any cloud/remote infrastructure as part of our offering because no one would pay for it. Everyone demands on-prem hosting of our software and it has to exclusively flow through their network security appliances.
I had a conversation with more than one CIO who would rather suffer arbitrary DDOS attacks than allow cloudflare the ability to decrypt their application traffic.
I am more than happy to work with these kinds of customers. There is no excuse to compromise on security when the stakes are this high. Everyone is willing to take their time to get it right.
>- Cloud-based factory automation? Unsafe now.
I feel like I must have slept through the point where that was ever a safe option rather than just a compromise for convenience sake.
I feel like I must have slept through the point where that was ever a safe option rather than just a compromise for convenience sake.
Right. Internet-configurable power grid control relays were not a good idea.[1]
"Web server interface is supported on UR over HTTP protocol. It allows sensitive information exposure without authentication." (Yes, they actually put a web server in a device which directly controls high voltage relays in power grids.)
"UR IED with “Basic” security variant does not allow the disabling of the “Factory Mode,” which is used for servicing the IED by a “Factory” user."
...
Those particular bugs were supposedly fixed.
[1] https://www.cisa.gov/uscert/ics/advisories/icsa-21-075-02
"Web server interface is supported on UR over HTTP protocol. It allows sensitive information exposure without authentication." (Yes, they actually put a web server in a device which directly controls high voltage relays in power grids.)
"UR IED with “Basic” security variant does not allow the disabling of the “Factory Mode,” which is used for servicing the IED by a “Factory” user."
...
Those particular bugs were supposedly fixed.
[1] https://www.cisa.gov/uscert/ics/advisories/icsa-21-075-02
I'm not a tech person by trade, but isn't that what regulation & compliance such as ISO27001 are there for? To ensure that best practices are followed and to prevent breaches like that?
I was also told that most Cloud-based providers have a shared responsibility model when it comes to security, wouldn't that jeopardize some of the existing relationship between the business and the Cloud Platform?
I was also told that most Cloud-based providers have a shared responsibility model when it comes to security, wouldn't that jeopardize some of the existing relationship between the business and the Cloud Platform?
There are two major problems with standards.
First, companies lie. I've been through a few companies that were audited for security compliance and simple truth was the company made great pains to keep the auditors away from but a few key people who will say what the auditors want to hear. Once, I was even told what I would need to say if I were asked (and it was blatantly untrue). Companies see security as a cost, not as an investment for on-going operations. So, the goal is to check it off the list that you're complying, not actually do it.
Second, best practices actually aren't. For instance, in my company I talked with the head of security after a presentation. I said that having passwords rotated was a horrible practice. It forces users to come up with something memorable. We should be using password managers and start using a long, random, gibberish passwords. He agreed, but "this is (such and such's) standard of best practices we contracted to support so we have to do this."
First, companies lie. I've been through a few companies that were audited for security compliance and simple truth was the company made great pains to keep the auditors away from but a few key people who will say what the auditors want to hear. Once, I was even told what I would need to say if I were asked (and it was blatantly untrue). Companies see security as a cost, not as an investment for on-going operations. So, the goal is to check it off the list that you're complying, not actually do it.
Second, best practices actually aren't. For instance, in my company I talked with the head of security after a presentation. I said that having passwords rotated was a horrible practice. It forces users to come up with something memorable. We should be using password managers and start using a long, random, gibberish passwords. He agreed, but "this is (such and such's) standard of best practices we contracted to support so we have to do this."
On the second point, thankfully we now have the NIST password guidelines recommending against password expiration, so we have an authoritative source to point at when fighting back against counterproductive password policies.
Everybody's PC is going to crawl with three layers of useless checkbox-ware.
When was the internet safe? Maybe in a sci-fi utopia.
> due to a cyberattack on one or more of the automaker's major business partners.
...and that's about the whole article.
...and that's about the whole article.
You're actually right.
How is this even considered an article? Can't find the author too, perhaps it's automated?
How is this even considered an article? Can't find the author too, perhaps it's automated?
They also use auto-increment on their author id https://www3.nhk.or.jp/nhkworld/en/news/authors/341/
Everyone is assuming this is the Russians and that could very well be true. If I were a bad actor from some place other than Russia, now would be a great time to pursue my own objectives and make as much noise as I wanted assuming that Russia would be blamed for it. Maybe even drop some misleading artifacts while I inside someone's network to re-enforce the paranoia. Just saying...
It's really regrettable that "cyberattack" can mean so many very different things, and reporters rarely clarify. Is it a random ransomware infection? Or is it a targeted attack by a national adversary?
It really could be either, and they have very different implications!
It really could be either, and they have very different implications!
Domestic doesn't mean a whole lot when reporting about a multinational. Domestic from the seat pof power? Domestic from the perspective of potential buyers? Domestic from the point of the reporter?
I know clicking through will tell me, but even without taking 'cyber' into account, meaning in 'simplified' news headlines is hard to find with the huge amount of specifics and context required with a lot of global news coverage.
I know clicking through will tell me, but even without taking 'cyber' into account, meaning in 'simplified' news headlines is hard to find with the huge amount of specifics and context required with a lot of global news coverage.
Nothing mentioned on Toyota's site: https://global.toyota/en/newsroom/
Supplier is apparently Kojima Industries, their website is being hammered: https://www.kojima-tns.co.jp/en/. Or maybe it's down because of the attack?
Supplier is apparently Kojima Industries, their website is being hammered: https://www.kojima-tns.co.jp/en/. Or maybe it's down because of the attack?
Technically true, but a bit click baitish.
Article says a plastics supplier was hit, which isn't the same as Toyota being hit.
Article says a plastics supplier was hit, which isn't the same as Toyota being hit.
Technically true, and arguably practically true too. Because Toyota do not maintain a stockpile of parts, the supplier and Toyota themselves become practically indistinguishable (except one is apparently easier to attack). When taking out a supplier shuts down 14 factories almost immediately, the supplier being a separate company for business, logistical, tax or PR reasons makes no practical difference.
The headline is accurate, and not click bait IMO.
No where did they imply Toyota was hit, however to believe it is any less serious because a Supplier was hit vs them directly shows ignorance of how manufacturing works
Supply Chain attacks are very serious and are how a lot of malware is moving because the primary targets are getting very good at preventing direct attacks however supplier are often overlooked as an attack vector, and even if they are aware of the risk, that risk is often very much understated
No where did they imply Toyota was hit, however to believe it is any less serious because a Supplier was hit vs them directly shows ignorance of how manufacturing works
Supply Chain attacks are very serious and are how a lot of malware is moving because the primary targets are getting very good at preventing direct attacks however supplier are often overlooked as an attack vector, and even if they are aware of the risk, that risk is often very much understated
I think it's clickbait. I immediately thought Toyota was hacked.
A supplier was hit, so they suspended domestic (i.e. Japanese) production. The title is actually a good encapsulation, and not at all click bait. Gotta celebrate these rare times when it's a legit title.
>The title is actually a good encapsulation, and not at all click bait.
I, too, thought it was an attack on Toyota based on the title. I see where you're coming from in saying that it's not, but because many of us have stated that we were confused by the title, I don't think it's fair to say that it's "not at all" click bait. It is, at least just a teensy bit.
I, too, thought it was an attack on Toyota based on the title. I see where you're coming from in saying that it's not, but because many of us have stated that we were confused by the title, I don't think it's fair to say that it's "not at all" click bait. It is, at least just a teensy bit.
The dividing line should be whether the headline is misleading, rather than confusing. If everything about an article could be encapsulated in a headline there would be no need for articles.
Was there a cyberattack? Yes. Did Toyota suspend domestic factory operations after it? Also yes. I'm content.
Was there a cyberattack? Yes. Did Toyota suspend domestic factory operations after it? Also yes. I'm content.
"Toyota suspends domestic factory operations after suspected cyber attack on supplier" would be far more accurate
All they had to do is add 2 words ("on supplier") but decided not too probably to make it sound like Toyota, a billion dollar company doesn't have good cyber security.
I would even feel better about adding one more word "some" as in Toyota suspends some domestic factory operations because by default it sounds like it suspends all domestic factory operations.
I agree. The title is click bait.
I agree. The title is click bait.
... after suspected cyber attack of a supplier would have been non-clickbait. As it is, it fooled me and so I say it's clickbait.
The headline is misleading. “Toyota halts Japan production after supplier hit by cyberattack” isn’t. https://www.autonews.com/automakers-suppliers/toyota-halts-j...
Moreover, geopolitically, there is a HUGE difference between random plastic company being a random extortion victim, and Japan’s largest company being the target of Putin’s retaliation.
Moreover, geopolitically, there is a HUGE difference between random plastic company being a random extortion victim, and Japan’s largest company being the target of Putin’s retaliation.
You have made a large amount of assumptions that it was a random supplier and it was a random extortion
Not saying it was not, but the idea that only Russia would attack Toyota directly, and only random attacker hi suppliers is not true
Not saying it was not, but the idea that only Russia would attack Toyota directly, and only random attacker hi suppliers is not true
I’m not making any assumptions- that’s the whole point. We don’t know if it was random or not. The original headline suggested Toyota was the direct target. If that was so, it would be some evidence that it was retaliation. But, Toyota wasn’t directly targeted, so the headline is misleading.
You two are talking past each other, at this point it’s just semantics.
> Toyota wasn’t directly targeted
GP is trying to say this is the assumption you’re making. Supply chain attacks can be a clever way to disguise a direct target as an indirect one.
I agree with you though, the headline could be more clear.
> Toyota wasn’t directly targeted
GP is trying to say this is the assumption you’re making. Supply chain attacks can be a clever way to disguise a direct target as an indirect one.
I agree with you though, the headline could be more clear.
Yeah I think this situation is more nuanced and we will probably find out more later. Supply chains are certainly very valid ways to attach an entity even if the supplier is not technically part of that entity.
If someone were to attack a major private utility in the US, for example electricity, they would not be directly attacking the US Government but I do not think most would find a title similar to this being clickbait.
If someone were to attack a major private utility in the US, for example electricity, they would not be directly attacking the US Government but I do not think most would find a title similar to this being clickbait.
If only the NSA had been spending the past decade hardening our cyber infrastructure rather than spying on its citizens and keeping found exploits secret from our industries
Your statement makes sense if you assume the best while considering the purpose of the NSA. Maybe they're doing exactly what they're supposed to do, and we are just making the wrong assumptions about their goals..
>NSA postures to prevent and eradicate threats and help the United States and its Allies defeat adversaries consistent with its authorities and with guidance from various national strategies.
From their about page [1], assuming you can take it at face value.
[1] https://www.nsa.gov/about/
From their about page [1], assuming you can take it at face value.
[1] https://www.nsa.gov/about/
>assuming you can take it at face value.
That depends, has the NSA ever given us reason not to trust them..?
That depends, has the NSA ever given us reason not to trust them..?
[deleted]
you know toyota is a japanese company right?
Good point. There are plants in the USA, but this appears to have to taken place in Tokyo.
I think the reality is if a nation state wanted to they could shut down 80%+ of infrastructure (water, power, internet etc) and any domestic manufacturing by attacking something in the supply line. There's terrible security everywhere, you literally have ransom ware groups taking down key infrastructure on accident its so bad. Our insane military budget does nothing to prevent this and it honestly seems almost as bad to me as a few nukes could be.
They actually I believe this year announced new programs in the US at least to specifically combat this - but I agree - it has been a known issue for a long time and is potentially a huge problem.
So I imagine the last 5-10 years of minor squabbling in the news about "Russian Hackers" was just training. I hope we're ready for the full onslaught now.
Why would Russia hit Toyota? They have more important stuff to hack right now.
China on the other hand, has a new auto industry to take over the world.
China on the other hand, has a new auto industry to take over the world.
Japan is supporting the sanctions and pledged to give $100M in aid to Ukraine.
I would bet that most major security services have a list of known vulnerabilities they can exploit whenever they see a good reason to.
I’m questioning motivation and priorities not ability.
Huge motive: gain access to "free" compute and storage resources. If they have GPU stacks too you can put ML to use for further penetration.
[deleted]
I don't think you have any idea what the NSA is capable of...
When I hear stuff like this, I begin to ask questions:
Why is your supply chain so vulnerable that a cyberattack on business partner (or multiple) causes you to close down *all* plants in your country? That's insane.
That sounds like your supply chain is the key problem here, and the cyberattack is just a smokescreen that is compounding the problem.
Why is your supply chain so vulnerable that a cyberattack on business partner (or multiple) causes you to close down *all* plants in your country? That's insane.
That sounds like your supply chain is the key problem here, and the cyberattack is just a smokescreen that is compounding the problem.
The Machine that Changed the World and the Toyota Way describe the theoretical advantages of having your supply chain situated in this manner. The pa democracy has been stress testing these advantages and demonstrating the downsides.
For the amount of fawning the get on HN and demographically similiar parts of the internet you'd think that a single supplier going poof would result in a partial outage or slowdown, not a stoppage of all production operations in one of their major markets.
How does just-in-time prepare for a fire at the the steering wheel supplier? How is a 30 day supply not sensible?
If a car factory is making 1000 cars a day, a 30-day supply is all the parts needed to make 30,000 cars. Managing that is a giant, complicated, land-consuming operation unto itself, and it all has to get sorted before you can build car #1.
And all the suppliers need to be able to ramp up supply after a problem to replenish the buffer, which means excess capacity normally sitting idle.
Toyota actually does stockpile some supplies it deems critical. This is despite the fact that Toyota is basically the pioneer of the JIT method of building cars.
https://www.france24.com/en/live-news/20210512-lessons-from-...
https://www.france24.com/en/live-news/20210512-lessons-from-...
I don't know about other countries but German auto manufacturers require (nearly) all suppliers to submit an FMEA [1] for their products. Some or maybe all US manufacturers do this, too, if I remember correctly.
In short, FMEA is a way to calculate risks and prepare for supply chain issues to a certain degree. When done correctly those FMEAs get very large and extensively lay out where risks are high so plans can be made to minimize them.
1: https://en.wikipedia.org/wiki/Failure_mode_and_effects_analy...
In short, FMEA is a way to calculate risks and prepare for supply chain issues to a certain degree. When done correctly those FMEAs get very large and extensively lay out where risks are high so plans can be made to minimize them.
1: https://en.wikipedia.org/wiki/Failure_mode_and_effects_analy...
Warehousing parts adds "fire at the warehouse" risk. You could have redundant warehouses, sure, but at some point you have to decide how much expense and operational complexity you're willing to tolerate to remove more 9s of risk.
Multiple JIT steering-wheel manufacturers?
I admit to not having a solid grasp of the numbers in play, but if my limited understanding is correct, a 30 day outage is significantly cheaper than the problems pre-JIT supply chains faced.
Supply chain risk management. There's no black box from which steering wheels emerge. You send people to their factories, you check their finances, you look into their history with regulatory agencies, you examine their leadership, you look at their supplier relationships. The place I work has a hundred folks that do nothing but risk and cybersecurity audits of our vendors and suppliers. It's a whole thing.
It's obviously not perfect and can get caught up in systemic issues like a pandemic, but as mentioned in a sibling comment it's still cheaper than the alternatives.
It's obviously not perfect and can get caught up in systemic issues like a pandemic, but as mentioned in a sibling comment it's still cheaper than the alternatives.
I get that at the end of the day somebody’s got to make the wheel, and the fire could be anywhere.
However, the article suggests that Toyota has to shut down all operations the minute their plastic doohickey supplier goes down - ie they have 0 buffer. That doesn’t seem optimal. Given that there are dozens or hundreds of suppliers, on any given day, isn’t at least one of them having issues?
However, the article suggests that Toyota has to shut down all operations the minute their plastic doohickey supplier goes down - ie they have 0 buffer. That doesn’t seem optimal. Given that there are dozens or hundreds of suppliers, on any given day, isn’t at least one of them having issues?
The theory, and the reason Toyota get studied, is that having no buffers means no waste and overall better production. The plastic doohickey supplier is producing plastic doohickeys at an optimal rate, and when supply stops everything stops. The problem is immediately apparent, and all eyes immediately available to fix the problem because all work has stopped. The alternative is a buffer. The plastic doohickey supplier needs to be able to produce plastic doohickeys at a faster rate than Toyota can consume them, to build up that buffer in the first place and to replenish it when problems occur. This extra capacity makes plastic doohickeys more expensive, both because of the excess production capacity and storage logistics. Many believe, including Toyota, that this thinking is the key to their success. It certainly requires good risk management though, identifying points of failure that will just shut down production for a short while vs. points of failure that risk destroying the company (say, a fire in a unique chip manufacturing plant that could take years to rebuild). See Toyota Production System or Lean Manufacturing.
I can see how these people can confirm risks. But how to they falsify risks? And more importantly: How are they managing it? Is the the size of the supply heap derived from the assumed risk and the cost of a default?
It's tricky going from an engineering mindset to risk management. I still struggle with it and have been in the industry for a long time. You can't falsify risks and many of your inputs are going to be guesses based on an amalgam of experience, historical information, trends and available risk/threat models.
Today there's all of these metrics available that give it the feel of an exact science, but its not. It's educated guesswork, but there's enough evidence showing it works that its worth substantially increasing the costs and friction of doing business to align with its outputs.
Usually.
In this case something arguably failed in the process. It's noteworthy but it's not an existential threat to Toyota. They will learn from it, someone's probably going to 'seek opportunities outside the firm' and they'll get back to making great products.
Today there's all of these metrics available that give it the feel of an exact science, but its not. It's educated guesswork, but there's enough evidence showing it works that its worth substantially increasing the costs and friction of doing business to align with its outputs.
Usually.
In this case something arguably failed in the process. It's noteworthy but it's not an existential threat to Toyota. They will learn from it, someone's probably going to 'seek opportunities outside the firm' and they'll get back to making great products.
Is this just a way to collect on insurance?
American factories are closed because of “chip shortages.” It seems way too early to say it was a cyber attack, much less for it to already impact production.
American factories are closed because of “chip shortages.” It seems way too early to say it was a cyber attack, much less for it to already impact production.
I wonder how much time they had to make this call?
The attack was on a supplier network
Expect a ramp up in security disruptions globally as a result of the Russia's war on Ukraine.
> the war on Russia
That’s a funny way to spell “Russia’s war on Ukraine”. :)
That’s a funny way to spell “Russia’s war on Ukraine”. :)
It's not Russia's fault that Ukraine's border got so close to Russian troops
Yeah was an egregious typo - fixed it. Clearly wrote the comment quickly.
[deleted]
[deleted]
[deleted]
- Cloud-based factory automation? Unsafe now.
- Mandatory remote diagnostics? Unsafe now.
- Remote updates? Questionable, and need to be blockable during crisis periods.