Trains were designed to break down after third-party repairs, hackers find(arstechnica.com)
arstechnica.com
Trains were designed to break down after third-party repairs, hackers find
https://arstechnica.com/tech-policy/2023/12/manufacturer-deliberately-bricked-trains-repaired-by-competitors-hackers-find/
244 comments
Thanks! Macroexpanded:
Polish Hackers that repaired DRM trains threatened by train company - https://news.ycombinator.com/item?id=38628635 - Dec 2023 (137 comments)
Polish train maker denies claims its software bricked competitor rolling stock - https://news.ycombinator.com/item?id=38570654 - Dec 2023 (2 comments)
Dieselgate, but for trains – some heavyweight hardware hacking - https://news.ycombinator.com/item?id=38567687 - Dec 2023 (292 comments)
Polish trains lock up when serviced in third-party workshops - https://news.ycombinator.com/item?id=38530885 - Dec 2023 (359 comments)
Polish Hackers that repaired DRM trains threatened by train company - https://news.ycombinator.com/item?id=38628635 - Dec 2023 (137 comments)
Polish train maker denies claims its software bricked competitor rolling stock - https://news.ycombinator.com/item?id=38570654 - Dec 2023 (2 comments)
Dieselgate, but for trains – some heavyweight hardware hacking - https://news.ycombinator.com/item?id=38567687 - Dec 2023 (292 comments)
Polish trains lock up when serviced in third-party workshops - https://news.ycombinator.com/item?id=38530885 - Dec 2023 (359 comments)
What really bugs me about all this: not only were they designed to break down, they messed that up to the point that the trains could have broken down while in service. The fact that a manufacturer would risk the lives of the passengers of the trains should result in personal liability for all of the execs of that company.
https://news.ycombinator.com/item?id=38641289
I'd love to see that angle researched more because I think it changes the game from something commercial to a far more important level.
https://news.ycombinator.com/item?id=38641289
I'd love to see that angle researched more because I think it changes the game from something commercial to a far more important level.
When bad things happen, the execs don't know it happened because they tell other people to go off and do stuff. When good things happen, execs take all the credit because the told people to go off and do stuff.
When my wife was in business school, the teacher of her management class used this idea as an unironic definition of management. They had to memorize the definition, which was essentially "taking credit for the work of other people." Not as a joke.
That's painful. But I totally believe it.
The issue started back when highways were being built across the US. Subsequently the US deregulated the Rail industry so that it could compete with long-haul trucking. Its been a steady spiral since then. John Oliver did an amazing and informative piece on trains.
Since then, workers get no days off, they reduced the amount of workers on a train to 2 and are trying to get it down to 1 as we speak. They increased the length of trans by literal miles, and these trains have killed multiple people by blocking roadways / ambulances and have run over children who have had to crawl underneath the trains to get to school. This happens multiple times a week where the road is blocked for almost a day.
These conductors have to walk the length of 4 miles to "inspect" the train, and deregulation has pushed the inspection time down to mere minutes instead of an entire checklist. They have taken all the power from the single regulation body and allow trains to run with brakes built in the early 1900s. We have many multiple "accidents" a year with chemical spills, all for profit seeking behavior. These workers are consistently over-worked and cannot take any sick days unless they are scheduled 2-3 months in advance.
Train accidents are inevitable at this point, the next one may be more disastrous than Palestine Ohio. This is the direct result to profit seeking by corporations (who pull in billions a year) and deregulation. It sucks because trains are f*cking awesome and could help society in a million ways.
Since then, workers get no days off, they reduced the amount of workers on a train to 2 and are trying to get it down to 1 as we speak. They increased the length of trans by literal miles, and these trains have killed multiple people by blocking roadways / ambulances and have run over children who have had to crawl underneath the trains to get to school. This happens multiple times a week where the road is blocked for almost a day.
These conductors have to walk the length of 4 miles to "inspect" the train, and deregulation has pushed the inspection time down to mere minutes instead of an entire checklist. They have taken all the power from the single regulation body and allow trains to run with brakes built in the early 1900s. We have many multiple "accidents" a year with chemical spills, all for profit seeking behavior. These workers are consistently over-worked and cannot take any sick days unless they are scheduled 2-3 months in advance.
Train accidents are inevitable at this point, the next one may be more disastrous than Palestine Ohio. This is the direct result to profit seeking by corporations (who pull in billions a year) and deregulation. It sucks because trains are f*cking awesome and could help society in a million ways.
A train breaking down or otherwise stopping causes no risk of injury or death.
Wrong. Trains generally stopping at intersections already pose a real risk by blocking emergency vehicles, resulting in cases of loss of property and life that could otherwise have been avoided. Having an unplanned shutdown that cannot be easily remedied only further exacerbates that potential.
Ok, I hadn't thought of that. In most of Europe level crossings are rare enough that this would be very unlikely.
It's also something the driver would easily avoid if they still have any control over the train, e.g. braking force.
It's also something the driver would easily avoid if they still have any control over the train, e.g. braking force.
In Poland level crossings are pretty normal.
> It's also something the driver would easily avoid if they still have any control over the train, e.g. braking force.
Yes, but: it would be much harder to test whether bricking this thing selectively does what it should do and for all we know right now you'd have a runaway on your hands. So this isn't just for shits and giggles.
And even a stopped train on a live track can under the right circumstances be extremely risky.
> It's also something the driver would easily avoid if they still have any control over the train, e.g. braking force.
Yes, but: it would be much harder to test whether bricking this thing selectively does what it should do and for all we know right now you'd have a runaway on your hands. So this isn't just for shits and giggles.
And even a stopped train on a live track can under the right circumstances be extremely risky.
You wrote that the passengers' lives were at risk.
This is simply not the case. It is a fundamental feature of every signaling system, and has been for over a century. Any collision would be caused by a seriously defective signalling system, not the stopped train.
This is simply not the case. It is a fundamental feature of every signaling system, and has been for over a century. Any collision would be caused by a seriously defective signalling system, not the stopped train.
You don't stop a train on a live track without a reason and yes, it would take multiple faults but since we have seen several such accidents in the last couple of years alone it would seem to me that it is indeed a risk. Poland has a ton of aging infrastructure, I wouldn't make any assumptions about what works and what doesn't, systems fail, and trains fail, when both fail at the same time your chances of an accident go up quite a bit so you try hard to make sure that both don't fail. Making one of them fail on purpose is a really bad idea.
Anyway, I'm sure you'll find a new reason to say why it's perfectly ok to stop trains with passengers in them willy nilly and how that isn't a safety issue but I'm just going to let it go here. I certainly hope you're not in charge of anything that involves public transport.
Anyway, I'm sure you'll find a new reason to say why it's perfectly ok to stop trains with passengers in them willy nilly and how that isn't a safety issue but I'm just going to let it go here. I certainly hope you're not in charge of anything that involves public transport.
So I’m an engineer in the passenger rail industry who often reviews and has input into the HTL (hazard tracking log). Stopping the train is assumed 100% safe for passengers and crew.
(Edit: you might be able to find some of these online, as regulators sometimes publish them for comment if there is a waiver request… not going to dox myself though)
Your point about external grade crossing hazards is valid… but also completely avoidable unless the train is super long (IE: not a passenger train) or the train was already going slow (like just leaving a station or signal).
There are separate system safety plans to ensure that timely/safe evacuation is possible regardless of where the train stops.
In the absolutely worst case, this means coupling another train and waiting the ~30 mins for people to walk through the train to the new cars, then uncoupling.
(Edit: you might be able to find some of these online, as regulators sometimes publish them for comment if there is a waiver request… not going to dox myself though)
Your point about external grade crossing hazards is valid… but also completely avoidable unless the train is super long (IE: not a passenger train) or the train was already going slow (like just leaving a station or signal).
There are separate system safety plans to ensure that timely/safe evacuation is possible regardless of where the train stops.
In the absolutely worst case, this means coupling another train and waiting the ~30 mins for people to walk through the train to the new cars, then uncoupling.
You're an engineer in the passenger rail industry in Poland? If not then you probably will want to familiarize yourself with the state of Polish rail infra before commenting further. It's a small miracle they don't have more accidents than they already do.
The most recent major incident is two months ago:
https://apnews.com/article/poland-train-accident-1db1a088c31...
And there are plenty of others to choose from:
https://en.wikipedia.org/wiki/Category:Railway_accidents_and...
There are only two countries in the EU that have worse rail infra than Poland (Romania, Bulgaria). Rolling stock is off mixed quality and vintage, maintenance spotty at best.
The most recent major incident is two months ago:
https://apnews.com/article/poland-train-accident-1db1a088c31...
And there are plenty of others to choose from:
https://en.wikipedia.org/wiki/Category:Railway_accidents_and...
There are only two countries in the EU that have worse rail infra than Poland (Romania, Bulgaria). Rolling stock is off mixed quality and vintage, maintenance spotty at best.
What makes a stopped train on a track 100% safe in terms of other trains or locomotives that might be oncoming? Are signals and means of communication redundant in a way that there‘s no chance of a collision?
The signalling system is designed so there's no possibility of an oncoming (or path-crossing) train. On a modern system this is done with formal proof of the software, on an old system with mechanical interlocks.
game_the0ry(2)
Here is a better write-up of the story:
https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
And the relevant HN discussion: https://news.ycombinator.com/item?id=38567687 5 days ago, 289 comments
Which again is a successor to: https://news.ycombinator.com/item?id=38530885 8 days ago, 357 comments
> A condition has been written in the computer code to disable the ability to run a train if it spends at least 10 days in one of these workshops.
Wow, that's just... so wrong.
Wow, that's just... so wrong.
It's like the code was designed to get my wife on my case after, "Honey, don't worry, I can fix it myself and save a bundle not having to go to the dealership."
"Told you you would make it worse" incoming.
"Told you you would make it worse" incoming.
I seem to remember from the original publication that the train would break down if it spent 10 days idle in some spot, away from a station.
I guess the logic being that it could be an unknown service location.
I guess the logic being that it could be an unknown service location.
They were actively comparing the GPS position of the train to the known location of the workshops of their competitors.
[deleted]
Uh, maybe not. These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
> These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
This makes little sense. It's pretty reasonable that any machine that is sent for repair may take longer than expected in the workshop. Parts availability for one. Also, manpower shortages, scheduling (we don't need it back until next month) and so on all make this "heuristic" more likely be a scam. This idea that the manufacturer is entitled to a lifetime stream of repair revenue has to stop.
I shudder to think what would happen in the US if an auto manufacturer tried this heuristic on the average owner of a pickup truck.
This makes little sense. It's pretty reasonable that any machine that is sent for repair may take longer than expected in the workshop. Parts availability for one. Also, manpower shortages, scheduling (we don't need it back until next month) and so on all make this "heuristic" more likely be a scam. This idea that the manufacturer is entitled to a lifetime stream of repair revenue has to stop.
I shudder to think what would happen in the US if an auto manufacturer tried this heuristic on the average owner of a pickup truck.
> I shudder to think what would happen in the US if an auto manufacturer tried this heuristic on the average owner of a pickup truck.
Yeah if you're going to try to pull a scam like that the US is not your best option:
https://nypost.com/2022/12/27/texas-mechanic-executed-over-5...
https://www.thetrucker.com/trucking-news/the-nation/pennsylv...
https://apnews.com/article/auto-shop-shooting-florida-c4d45f...
If you just search "truck owner murders mechanic" you get tons of unique results.
Yeah if you're going to try to pull a scam like that the US is not your best option:
https://nypost.com/2022/12/27/texas-mechanic-executed-over-5...
https://www.thetrucker.com/trucking-news/the-nation/pennsylv...
https://apnews.com/article/auto-shop-shooting-florida-c4d45f...
If you just search "truck owner murders mechanic" you get tons of unique results.
The top comments on that nypost article are insane, blaming the victim for probably doing "unauthorized repairs", and saying that the "southern border" is at fault.
Wow.
Wow.
> and saying that the "southern border" is at fault
There's people posting like that here on HN.
There's people posting like that here on HN.
Nah, the Canadians on HN are too nice to blame us.
I feel like blaming guns might upset some people, but I don't feel like this type of thing happens as much in the rest of the western world, where gun laws are typically a lot stricter.
Don't you ever care about upsetting people who live in countries with unreasonable gun laws. Especially third world countries like the US.
It seems weird to blame guns for the story of a customer running down the mechanic with his vehicle?
I assume most of it is access to guns. But I do wonder how much is due to people are not using giant ego-soothing trucks in the parts of the world where petrol doesn't get subsidies. So there are fewer truck owners to do the murders.
Total nonsense, car owners murder truck drivers too.
That's nuts. Who says those were scams? To me it reads as though the customers had a couple of loose wires.
I'm not saying these were scams. I was just pointing out that customers in the US that feel like they were scammed can often get deadly violent.
Ah ok, I see now. Yes, between deranged and entitled customers and the ready availability of point-and-click interfaces of the fatal variety you can have some pretty bad outcomes.
No, absolutely not.
(1) This was scheduled maintenance service, not a quick repair. Maintenance is complicated and is expected to take more than 10 days:
> Maintenance a train is a complicated affair – it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested. The SPS carries out the maintenance procedures according to the relevant maintenance manual (some 20,000 pages) provided by the manufacturer, but the train does not start after being put together.
From [1].
(2) If this was a legitimate check then there should be a legible error code instead of the train randomly locking up with no explanation why. This was clearly designed to sabotage competing maintenance service companies.
[1] https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
(1) This was scheduled maintenance service, not a quick repair. Maintenance is complicated and is expected to take more than 10 days:
> Maintenance a train is a complicated affair – it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested. The SPS carries out the maintenance procedures according to the relevant maintenance manual (some 20,000 pages) provided by the manufacturer, but the train does not start after being put together.
From [1].
(2) If this was a legitimate check then there should be a legible error code instead of the train randomly locking up with no explanation why. This was clearly designed to sabotage competing maintenance service companies.
[1] https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
The manufacture's current position is someone hacked their systems and changed the code so that it would make them a lot of money, they totally didn't do that.
>"The president of Newag contacted me," Cieszyński wrote. "He claims that Newag fell victim to cybercriminals and it was not an intentional action by the company. The analysis I saw indicated something else, but for the sake of clarity, I will write about everything.
>Newag president Zbigniew Konieczek said that "no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software."
>"The president of Newag contacted me," Cieszyński wrote. "He claims that Newag fell victim to cybercriminals and it was not an intentional action by the company. The analysis I saw indicated something else, but for the sake of clarity, I will write about everything.
>Newag president Zbigniew Konieczek said that "no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software."
Which is obviously bullshit made up by the PR agency they hired, designed to muddy the waters. For example, researchers dumped the Train's firmware (in presence of third party witnesses) before and after sending it to the Newag's service, and found that the a backdoor code was recompiled and updated there.
> These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
I know HN is a great place to practice PR-speak, but could you please stop fucking with the gullible people?
I know HN is a great place to practice PR-speak, but could you please stop fucking with the gullible people?
The words were chosen to indicate uncertainty on my part. A train parked in a repair depot for a fortnight triggering a need for more checks before putting it in service? I can totally imagine that spec being signed off in meetings. Fortunately the voting system on here does an excellent job of hiding unpopular thoughts so I doubt I've confused anyone.
You’re just pushing a pretty blatantly uninformed opinion supporting a frankly indefensible position. My car had minor collision damage and it took the manufacture’s 1st party service center 3 weeks to fix it. The assumption that manufacturers are uniquely positioned to repair their products is a poisonous tactic being used by insidious players to discredit right to repair.
This is stretching my sarcasm meter to the breaking point, either you are very good at deadpanning or you are trying really hard to muddy waters that are crystal clear. If the former, well played.
That’s not a long outage.
If someone decided to paint it, the clean, prep and paint could take that long. And now you have a dead train.
"Hey, car 1812 needs a new restroom toilet and a couple of new seats."
"Lead time on the toilet is 21 days from the manufacturer."
"Uff, 21 days? That's more than 10! We will have to junk the whole car."
"Lead time on the toilet is 21 days from the manufacturer."
"Uff, 21 days? That's more than 10! We will have to junk the whole car."
Of course, because we all know that only the manufacturer of such a complicated device can be trusted to maintain it. Oh wait, maintenance of trains is a competitive process? The manufacturer provides a 20,000 page maintenance specific manual? Maybe trains really are simpler than iPhones.
/s
Parent comment is really an indication that society has lost the plot when it comes to ownership. The trains do not belong to the manufacturer after sale. They can't introduce anti-competitive code and pretend that users want it, like phone companies can.
/s
Parent comment is really an indication that society has lost the plot when it comes to ownership. The trains do not belong to the manufacturer after sale. They can't introduce anti-competitive code and pretend that users want it, like phone companies can.
That would make sense if we were talking about something that changes dramatically over time; like a human heart for transplants. Or a nuclear fission core without its carbon bars. But trains don’t change significantly in a 10-day period. I doubt they degrade much after a full year, even. They need to be sturdy.
Ok so no sarcasm, well, there are many reasons a train can spend a long time in the workshop:
Complete checkup takes time, or one important mechanic is sick, delaying things or whatever. It is not the responsibility of the manufactor anymore. (A different company has the official service contract.) If you have other informations pls share.
Complete checkup takes time, or one important mechanic is sick, delaying things or whatever. It is not the responsibility of the manufactor anymore. (A different company has the official service contract.) If you have other informations pls share.
In at least two cases, the train was simply unused for 10 days for unrelated reasons.
After which the two specific trains this happened to "gained" GPS lockout.
After which the two specific trains this happened to "gained" GPS lockout.
Was that sarcasm?
No sarcasm, just in total disagreement with the (initial/apparent) HN consensus. I've put a top level comment which is hopefully clearer.
You are also disagreeing with the owner and operator of the train, who is responsible for its safe operation.
The OP article is worth reading for the brazen blustering statements made by Newag management, I paraphrase "you can't prove that the software found on our trains that sabotages our competition was installed by us!"
That headline was so misleading. I'm interested in the topic and saw that link on HN, but scrolled away because dieselgate is "obviously" about emissions and I assumed that nothing new will come from the article about train emissions, we all know already they are big polluters but we kinda have to deal with it.
They should have mentioned something about vendor lock or right to repair in the headline.
They should have mentioned something about vendor lock or right to repair in the headline.
> we all know already they are big polluters but we kinda have to deal with it.
There is a lot of variation in train emissions, but most modern ones (electrified) have minimal emissions - and far lower than electric cars even, per passenger mile or per tonnage transported.
Even the oldest ones in common use (diesel electric) are more efficient per ton-mile and passenger mile than a typical car.
There is a lot of variation in train emissions, but most modern ones (electrified) have minimal emissions - and far lower than electric cars even, per passenger mile or per tonnage transported.
Even the oldest ones in common use (diesel electric) are more efficient per ton-mile and passenger mile than a typical car.
I've tried to explain myself in the comment below, sorry for the confusion.
> I assumed that nothing new will come from the article about train emissions, we all know already they are big polluters but we kinda have to deal with it.
... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
In absolute values, not relative per ton of cargo or compared to other modes of transport. Maybe I wasn't clear enough, for that I'm sorry. I do support trains everywhere and think they are the best land transport we have by far. I was merely thinking that a diesel engine is a diesel, even if a train vendor fudge the emissions somehow (hence the reference to the Volkswagen dieselgate, I thought it meant) it wouldn't matter much on the planetary scale. Thus not particularly interesting.
But instead this was all about electronics and malicious locks, which wasn't apparent from the headline.
But instead this was all about electronics and malicious locks, which wasn't apparent from the headline.
IMO this is really what Dieselgate was about. Not specifically fudging the emissions, but hiding secret code to do so during specific test profiles, then lying about it afterwards. Analogously, these trains hid secret code to disable them after being taken to certain locations.
Especially since these trains in particular are electric.
> ... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
Modern diesel engines yes, but the old ones? They got barely any emission controls, only for locomotives built after the 90s there is regulations [1]. On top of that comes brake dust [2] which is only irrelevant in powered-car electric passenger trains with regenerative braking - every other train releases insane amounts of it, no surprise given that the brakes have power outputs in the megawatt range.
[1] https://www.epa.gov/regulations-emissions-vehicles-and-engin...
[2] https://www.railwaygazette.com/vehicles/reducing-brake-dust-...
Modern diesel engines yes, but the old ones? They got barely any emission controls, only for locomotives built after the 90s there is regulations [1]. On top of that comes brake dust [2] which is only irrelevant in powered-car electric passenger trains with regenerative braking - every other train releases insane amounts of it, no surprise given that the brakes have power outputs in the megawatt range.
[1] https://www.epa.gov/regulations-emissions-vehicles-and-engin...
[2] https://www.railwaygazette.com/vehicles/reducing-brake-dust-...
Many trains have rheostatic brakes, where the kinetic energy is converted to heat using a bank of resistors.
https://en.wikipedia.org/wiki/Dynamic_braking
https://en.wikipedia.org/wiki/Dynamic_braking
Right above you it explicitly states the trains are •electric• and not diesel. RTFA
As said: Electric trains are not emissions-free due to the brake dust.
By that measure nothing is emissions free, me walking down the road will generate a bit of rubber / asphalt dust.
Also regenerative braking... Trains are amazing
Also regenerative braking... Trains are amazing
What is it with this thread? You're the second person to bring up the most wild stuff.
When I walk I kick up dust. And I breathe. I guess I'm also not emissions free even when I cycle? Brake dust and all that?
When I walk I kick up dust. And I breathe. I guess I'm also not emissions free even when I cycle? Brake dust and all that?
>According to Dragon Sector, Newag entered code into the control systems of Impuls trains to stop them from operating if a GPS tracker indicated that the train was parked for several days at an independent repair shop.
Oh, that seems pretty damning. I wonder if this was a lone developer or ordered from above.
Oh, that seems pretty damning. I wonder if this was a lone developer or ordered from above.
“Newag president Zbigniew Konieczek said that ‘no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software.’”
The ridiculousness of this defence makes it clear leadership was in the know.
The ridiculousness of this defence makes it clear leadership was in the know.
> The ridiculousness of this defence makes it clear leadership was in the know.
https://kolejowyportal.pl/koleje-dolnoslaskie-odpowiadaja-ne... - this is from 2022-07-06. The train owner complains that they still didn't receive information from Newagg about what was fixed when unit had to be shipped to manufacturer after it refused to start.
So the issues with 3rd party servicing were publicly known well before the smoking gun was found in firmware.
https://kolejowyportal.pl/koleje-dolnoslaskie-odpowiadaja-ne... - this is from 2022-07-06. The train owner complains that they still didn't receive information from Newagg about what was fixed when unit had to be shipped to manufacturer after it refused to start.
So the issues with 3rd party servicing were publicly known well before the smoking gun was found in firmware.
I'm not sure that's the smoking gun.
IIRC, there was some "konami" code to re-enable the train after disablement that was removed in a first-party update after the third-party repair company found the sequence.
And not related to the third-party locations, one of the trains had some code where if the year>2021 & month>11 & day>? then the train would disable itself; ultimately doing it in the wrong year because the train was off during november / december of that year. This is a little excessive for somebody random to do.
IIRC, there was some "konami" code to re-enable the train after disablement that was removed in a first-party update after the third-party repair company found the sequence.
And not related to the third-party locations, one of the trains had some code where if the year>2021 & month>11 & day>? then the train would disable itself; ultimately doing it in the wrong year because the train was off during november / december of that year. This is a little excessive for somebody random to do.
The 2021-11-?? date was the day the train was scheduled to be serviced (so the intention of the code was to brick the code after the service). Only by random chance it didn't work, because the train was in maintenance a bit longer, and the code didn't trigger in January 2022 because of the date checking bug.
That is completely nuts and irresponsible beyond belief.
I guess they can't claim any copy write / drm violations if they also deny it was their code.
Riiight?! This is a really strong defense for the white hats. Either it’s your code and you are egregious slime balls doing illegal things, or “some third party interfered and hacked your trains like you said and we fixed it you’re welcome”.
Once this is proven false, they will blame the devs, just like VW did.
The dev. at VW was at least smart enough to keep the paper trail around. Lets hope the people working for this company did the same.
There's also a bunch of service people involved etc. (some "non-working" trains were sent back to NEWAG, and the people doing the maintenance had to know how to unlock them, etc.)
I wonder how they explain the "rogue dev" somehow clandestinely communicating the reset procedure to the manufacturer's shop.
I hope evidence will be provided in court.
What lone developer says "let me find all the GPS coordinates of all the train repair shops and make the trains not work if they've been parked there for days"?
"My cat slept on the keyboard and accidentally developed and merged working advertisement feature into a release build of the new AC game. It was totally on accident, happens all the time." (c) Ubisoft
"I come to you now with apologies. This is the third time my cat has put paid mods into Skyrim, something for which he is very very sorry you are unhappy about."
We talk on HN a lot about ethics and programming and the importance of engineers refusing to develop unethical software.
But I think we need another PSA that, if you are going to write some code that could land you in jail, make sure you get your entire reporting chain in writing telling you to do it, up to and including the CEO and the board of directors.
But I think we need another PSA that, if you are going to write some code that could land you in jail, make sure you get your entire reporting chain in writing telling you to do it, up to and including the CEO and the board of directors.
Alternatively, if you're going to break the law, don't do that (break the law).
Don't underestimate these damned rouge developers!
I've heard of Khmer Rouge but not of Dev Rouge before.
Humor is so little appreciated these days.
Humor is prevalent in the internets various forums. It's well appreciated generally - easy upvotes on Reddit and elsewhere. But that also means that a serious forum with higher signal-to-noise ratio is something uniquely valuable.
[deleted]
All of them. Dieselgate? Rogue dev. Trains breaking for independent shops? Rogue dev. Some airplane going down because of bad software? Rogue dev.
Repeat after me: The company is never at fault.
Repeat after me: The company is never at fault.
Perhaps GitHub Autopilot and ChatGPT teamed up to write the code upon noticing that the developer tripped and fell, causing a GSM modem to fall off their desk and into the assembly process.
Management: "a 6-pack of beer for the brilliant dev who figures out a solution to this problem of us ..."
Yeah, sounds like a secret 20% project :)
As a lone developer there is absolutely no motivation to do this and hide this decision from management.
Having said that a lone developer can come up with the idea, propose it to management with the expectation of some fat bonus. But why do this in secret?
There is just no possibility where upper management is not involved in this.
Having said that a lone developer can come up with the idea, propose it to management with the expectation of some fat bonus. But why do this in secret?
There is just no possibility where upper management is not involved in this.
100% agree. Doing this in secret without management's express approval means no bonus, working extra for no benefit, compromising your other tasks and exposing yourself to heavy liability for compromising the country's infrastructure. This is something that companies could get away with, but a single developers would not only lose their job, but risk going to jail.
Note that management's express approval could be verbal conversations that you cannot prove happened or not. With some work management can find some way to get a developer who has some other conflict of interest to do this: if your brother in law works for the maintenance yard situations then you have motive to ensure he has work.
They will surely try that explanation too if the others fail. Currently their stance seems to be "there is no such code on the trains, and if there is, it's not from us".
That didn't happen, and if it did then it wasn't us, and if it was then it was inadvertent, and if it wasn't then our current leaders didn't know, and if they did then it wasn't illegal, and if it was then we have a different interpretation of that law which is unreasonable anyway and should be ignored and repealed.
It’s from those darn third party developers! ;)
Blame the hacker, wait for it.
Also: I wonder if their management realizes that they probably have a nice trail in the form of a bunch of repositories and commit messages. Would be nice if that leaked.
Also: I wonder if their management realizes that they probably have a nice trail in the form of a bunch of repositories and commit messages. Would be nice if that leaked.
You're talking about industrial PLCs. They're programmed using a-bit-more-fancy Scratch snappy blocks. There is no version control. The firmware contained paths embedded as strings, so we know that firmware for each model and customer was developed in a separate folder on disk. I wouldn't be surprised if they also had .zip files with backups of previous versions.
The IDE for these PLCs actually has VCS integration! It's SVN, but it's still better than nothing.
Its on-disk representation of graphical 61131-3 languages (FBD / SFC) is text-based and somewhat human readable, so there's nothing technically preventing the developers from keeping all of this in any other VCS of their choice.
Its on-disk representation of graphical 61131-3 languages (FBD / SFC) is text-based and somewhat human readable, so there's nothing technically preventing the developers from keeping all of this in any other VCS of their choice.
There is nothing wrong with SVN, it's just that Git allows for some workflows that are better suited to larger teams and more complex projects. But for your average PLC project with a team of 10 and one binary as the output it should be more than enough.
You likely won't see any 'feature branches' or frequent merges in this kind of environment.
You likely won't see any 'feature branches' or frequent merges in this kind of environment.
> There is nothing wrong with SVN
Except merging things, and handling a lot of files...
There are lots of small things wrong with SVN. But it's indeed usable.
Except merging things, and handling a lot of files...
There are lots of small things wrong with SVN. But it's indeed usable.
I'm aware of repos with a few million files in it that have been going since 2003 and not a single issue.
Merging things is different than in Git but it works. I use both, and I'm not religious about either, some things are easier in Git, some are easier in SVN. Git provides more footguns. And loads and points them too.
Merging things is different than in Git but it works. I use both, and I'm not religious about either, some things are easier in Git, some are easier in SVN. Git provides more footguns. And loads and points them too.
I'm intimately familiar with PLC programming, yes, you can do it the 1980's way but there are also plenty of environments that allow for modern version control.
https://www.google.com/search?q=version+control+plc+programm...
You'd have to be pretty daft to do this kind of development today and not take advantage of version control and even the most visual versions of these systems eventually output (text) files. You may not be able to do an easy line-by-line comparison but you will have a commit log with helpful messages.
Look for 'engage in anti-competitive behavior' in the log message ;)
https://www.google.com/search?q=version+control+plc+programm...
You'd have to be pretty daft to do this kind of development today and not take advantage of version control and even the most visual versions of these systems eventually output (text) files. You may not be able to do an easy line-by-line comparison but you will have a commit log with helpful messages.
Look for 'engage in anti-competitive behavior' in the log message ;)
What about: rogue hackers maliciously squashed our whole repo into a single commit?
I don't doubt that's exactly what would happen, in fact I think that that rogue hacker is about to do his thing, quick, erase the backups!
Yeah.. I've just realized that while it's entertaining to watch how it unfolds or predict what can happen next, it's also sad, because pretty much everybody in the rail industry loses..
One of my business partners works for PKP it's very annoying to see this all unfold and in this particular way. Poland has so much potential, these idiots are ruining Polands image in ways that really matter.
But then again, as a Dutch person I have enough issues locally that I can't even complain...
Ugly times.
But then again, as a Dutch person I have enough issues locally that I can't even complain...
Ugly times.
I have a friend that runs a business, where he hires Polish developers to do his coding.
He absolutely raves about them. It sounds like he's got some good coders.
He absolutely raves about them. It sounds like he's got some good coders.
Poland has a very strong technology and mathematics tradition that goes back decades. It's one of the reasons Poland has some strong feelings about their role in the breaking of the Enigma, for the longest time that was played down.
Working in security on the operating side (albeit not in Poland):
No, pretty much just the manufacturer loses. Short term the operator loses, but I'm sure that the courts will award damages.
For me, this incident is a welcome argument with which I can tighten the screws on manufacturers in the next round of train buying (at minimum, they will agree to heavy contractual fines for anything like this; at best I get full source code for every train).
For too long the only priority in OT was safety (fine in the 80ies, but the second you integrate an IP stack that posture doesn't work anymore). This has been changing in the industry thanks to EU-regulation; this incident will accelerate the change.
No, pretty much just the manufacturer loses. Short term the operator loses, but I'm sure that the courts will award damages.
For me, this incident is a welcome argument with which I can tighten the screws on manufacturers in the next round of train buying (at minimum, they will agree to heavy contractual fines for anything like this; at best I get full source code for every train).
For too long the only priority in OT was safety (fine in the 80ies, but the second you integrate an IP stack that posture doesn't work anymore). This has been changing in the industry thanks to EU-regulation; this incident will accelerate the change.
What I meant is that I feel the trust among parties might go down industry-wide.
In a sense you admitted that:
> (...)I can tighten the screws on manufacturers in the next round of train buying(...)
But then I can see it might help change things for the better across the board, as you nicely described. Thanks for the illuminating comment!
> (...)I can tighten the screws on manufacturers in the next round of train buying(...)
But then I can see it might help change things for the better across the board, as you nicely described. Thanks for the illuminating comment!
That's assuming we will get to the bottom of this. And I really hope we will. But I'm kind of concerned that it will all be wiped under the carpet.
Temporarily lost ok. Better to let these manufactures do whatever they want.
Oops, rogue hackers deleted our .zip files with the backups! :)
EDIT: BTW having no version control would be pretty telling on its own. It's a critical piece of software, that controls a train..
EDIT: BTW having no version control would be pretty telling on its own. It's a critical piece of software, that controls a train..
If it was developed anytime after 1990 (probably before) you will find plenty of programmers willing to be expert witnesses and tell the court that the company not having version control is gross incompetence, the only reason a company would do that would be so they can hide evidence of illegal actions. As such the court should impose punitary damages.
Of course before going on the stand the expert witness will work with a lawyer to word smith the above into something the court will better understand. however I think the generic idea is something everyone here will agree with.
Of course before going on the stand the expert witness will work with a lawyer to word smith the above into something the court will better understand. however I think the generic idea is something everyone here will agree with.
> There is no version control.
I've used a thing that not only doesn't play nice with versioning (your local workspace is a collection of embedded db files) but doesn't play nice with multiple developers (no way to sync workspaces). I still managed to get it into version control, even if useful things like diffs didn't do anything useful.
I've used a thing that not only doesn't play nice with versioning (your local workspace is a collection of embedded db files) but doesn't play nice with multiple developers (no way to sync workspaces). I still managed to get it into version control, even if useful things like diffs didn't do anything useful.
If an employee did this without authorization, the company should still be liable because I do not think safety regulations exempt manufacture from responsibility of employee actions. I believe the opposite is true, that they must ensure safety despite rogue employees, with reviews, audits and such.
They need to show audits and that the code is obscure enough that it could be missed. I've seen IOCCC type things that do things I wouldn't catch in a code review. (IIRC there is a malicious code contest where the goal is to put backdoors in that others don't see in review, but I can't recall what it is)
You’re thinking of the Underhanded C contest: http://www.underhanded-c.org/. The goal is to write C code which looks completely innocent, but which actually performs some malicious or unwanted behaviour.
In the last year they ran the contest (2016), the goal was to write a program that would compare two sets of measurements for similarity (nominally radiation emission spectra) but fail to give a correct answer when given a particular crafted input.
In the last year they ran the contest (2016), the goal was to write a program that would compare two sets of measurements for similarity (nominally radiation emission spectra) but fail to give a correct answer when given a particular crafted input.
There's been a decent amount of those in the wild, too. If you're in an environment without unicode auditing you can do all sorts of crazy things without IOCCC levels of code tricks too.
[deleted]
I suspect this is going to end all further sales of these trains outside of Poland. What railway will ever trust them again?
His dog did it after eating the CS homework.
Is this really suggesting some developer, buried under tasks and putting extra hours, thought to implement this feature in his free time for months, slipped it through reviews and scans and pipeline deployments, for... what? Really? I rather prefer to think the comment was sarcasm.
Yes it was sarcasm, since it is common for management to shift the blame on individual engineers in these cases (see for example the Volkswagen emissions scandal and the Boeing MAX crash).
IIRC Volkswagen actually used the term “rogue engineer” when speaking to the press.
In VW's case there was some reason to believe that at first. There are reasons to test an engine in the lab without emissions controls, and so it is believable that a lazy engineer could put in code for the lab that also triggered in real world conditions.
Note that the above is only believable before the rest of the investigation. Proper investigation proved the story wrong, but it is just possible enough to believe it could be true if the proper investigation hod come up differently.
Note that the above is only believable before the rest of the investigation. Proper investigation proved the story wrong, but it is just possible enough to believe it could be true if the proper investigation hod come up differently.
npm add block-repair-shop. It can happen so fast.
[deleted]
> Oh, that seems pretty damning. I wonder if this was a lone developer or ordered from above.
Guess. (Did I miss the sarcasm again? I always miss the sarcasm.)
Guess. (Did I miss the sarcasm again? I always miss the sarcasm.)
I sure hope the dev who was made to implement it kept the paperwork that proves who made them do it.
Otherwise, they’ll find out how loyalty is rewarded nowadays.
Otherwise, they’ll find out how loyalty is rewarded nowadays.
Even if they did keep that paperwork, they should be punished alongside the manager who asked them to do it. They knew it was unethical and did it anyway. They should have refused. Risk of being fired is not as big of a problem for a programmer as it is for almost anyone else.
I've got mixed feelings. I would be interested in hearing the whole story first. The "hey, I can program the trains to break if we're not the ones to run maintenance, should I do that?" or the "yeah, I guess I could make the trains break if we're not the ones to run maintenance" are both pretty clearly unethical.
However, the "yeah, sure I can add in a GPS locator module" and the "yeah, I can add analytics that reports when the train is in a maintenance hanger" and the "the catastrophic program halt code module used in cases of extreme failure is located here, but why do you want to know that?" all seem less than unethical.
Theoretically you only need one unethical line of code, so how it got there, I think, is pretty important to know before passing ultimate judgement.
EDIT:
Of course for something like train control software, you really should have a process or at the very least responsible engineers that would notice a middle manager with limited technical skills asking suspicious questions and then pushing up a PR that is self approved.
I would be more than willing to entertain an ethical debate along those lines. Although, like I said, I think it's important to understand the whole story because the specifics really do make a difference.
However, the "yeah, sure I can add in a GPS locator module" and the "yeah, I can add analytics that reports when the train is in a maintenance hanger" and the "the catastrophic program halt code module used in cases of extreme failure is located here, but why do you want to know that?" all seem less than unethical.
Theoretically you only need one unethical line of code, so how it got there, I think, is pretty important to know before passing ultimate judgement.
EDIT:
Of course for something like train control software, you really should have a process or at the very least responsible engineers that would notice a middle manager with limited technical skills asking suspicious questions and then pushing up a PR that is self approved.
I would be more than willing to entertain an ethical debate along those lines. Although, like I said, I think it's important to understand the whole story because the specifics really do make a difference.
I don't think that's a good way of thinking about it. Just because programmers might have better job security doesn't mean other motivators for not wanting to get fired exist.
Whatever other motivation you have it should not be enough for you to do illegal/highly unethical things.
This is why we need very strong whistleblower protections.
This is why we need very strong whistleblower protections.
Which also applies to everyone else. The programmer job security just makes that easier in most cases.
Which is not really my point as to why they should be held responsible. The reason is the real world consequences of their actions and the scale and ease of introducing negative consequences by tech creators.
Which is not really my point as to why they should be held responsible. The reason is the real world consequences of their actions and the scale and ease of introducing negative consequences by tech creators.
To be honest, with or without a paper trail, society need less people who do shit like this.
Whoever implemented this, if you read this, you are a very bad person and destructive for society.
Whoever implemented this, if you read this, you are a very bad person and destructive for society.
I disagree with the framing "You are a bad person" (although I think I understand the sentiment), because it implies they can't change (or understand the error). It seems better to leave it at "You did something very harmful, destructive for society".
I agree. I understand sometimes a person is under enormous pressure to do something they know is not right.
People respond to incentives. That's the entire premise of capitalism.
If you want this to not happen, you need to incentivize refusing to do this more than the existing incentive to do this.
Otherwise, punishment only changes who's in the chair next time.
(Future possible punishment is not a very good disincentive, which is why increased prison sentences don't deter crime.)
If you want this to not happen, you need to incentivize refusing to do this more than the existing incentive to do this.
Otherwise, punishment only changes who's in the chair next time.
(Future possible punishment is not a very good disincentive, which is why increased prison sentences don't deter crime.)
I hope they will testify so higher-level managers will be fined and spend some time in prison. And company will be hit with massive contract penalties and fines.
(not expecting this, but would be nice)
(not expecting this, but would be nice)
Is there any conceivable point where people will be charged and jailed for fraud after selling sabotaged goods?
Theoretically, the recently-revealed prosecution started with notice of two crimes that include 0.5 to 8 years jail time.
[deleted]
Why would the powerful punish themselves?
Themself? No.
But there is more than one powerful person in Poland, and Newag owner is far from the most powerful.
But there is more than one powerful person in Poland, and Newag owner is far from the most powerful.
Because more powerful entities might be enraged.
> Newag has said that "any remote intervention" is "virtually impossible."
That’s a very bold statement right there! Also, from my experience, most if not all companies who build automation/robotic systems they maintain “backdoors” of some sort, not to disable them remotely, but to quickly and efficiently access the software remotely upon request to troubleshoot or fix issues, it doesn’t make sense to fly to the client and sometimes the other half of the planet just find out it was a local IP mismatch or a system service is down, so saying “impossible” is not true.
That’s a very bold statement right there! Also, from my experience, most if not all companies who build automation/robotic systems they maintain “backdoors” of some sort, not to disable them remotely, but to quickly and efficiently access the software remotely upon request to troubleshoot or fix issues, it doesn’t make sense to fly to the client and sometimes the other half of the planet just find out it was a local IP mismatch or a system service is down, so saying “impossible” is not true.
TL;DR:
* Polish train maintenance company, SPS, was getting suspicious as trains made by a company, Newag, kept on "randomly" breaking and couldn't be fixed. They was getting fined millions by Polish government as they had a contract that fined them for being too slow with repairs.
* They secretly hired literal hackers (Dragon Sector) for 2 months to dig around Newag train code.
* Hackers found out some incredible things, generally that fit under the umbrella of "late-stage capitalism", or more specifically, corporate protectionism, sabotage, ransom, etc.
Some examples of the secret code that the hackers found:
* Breaks the trains if they go into geo polygons that are right around the warehouses of 5 Polish train maintenance companies, including SPS.
* Breaks the trains after 1 million kilometers.
* Breaks the trains if they don't move for 10 days.
* Secret button press combination (basically Tekken, Street Fighter, etc.) to disable the "malfunctions".
* Polish train maintenance company, SPS, was getting suspicious as trains made by a company, Newag, kept on "randomly" breaking and couldn't be fixed. They was getting fined millions by Polish government as they had a contract that fined them for being too slow with repairs.
* They secretly hired literal hackers (Dragon Sector) for 2 months to dig around Newag train code.
* Hackers found out some incredible things, generally that fit under the umbrella of "late-stage capitalism", or more specifically, corporate protectionism, sabotage, ransom, etc.
Some examples of the secret code that the hackers found:
* Breaks the trains if they go into geo polygons that are right around the warehouses of 5 Polish train maintenance companies, including SPS.
* Breaks the trains after 1 million kilometers.
* Breaks the trains if they don't move for 10 days.
* Secret button press combination (basically Tekken, Street Fighter, etc.) to disable the "malfunctions".
Do you know if the low-level technical report is available? I love reverse engineering firmware, and this sounds like a holy grail. I mean a freaking train? Ugh someone should drop binaries.
Some info is here https://badcyber.com/dieselgate-but-for-trains-some-heavywei... , but the actual details will be presented at the Chaos Communication Congress at the end of December.
Not found one yet. The linked article contains a very small amount of detail, such as the lat-long coordinate values they found within the dissassembled code, etc., but not much else unfortunately.
I'm waiting for the stuxnet-like report on this as much as anyone.
I'm waiting for the stuxnet-like report on this as much as anyone.
What noobs. Everyone knows you've got to put that chicanery behind a web service call so they can't find the evidence directly from the client.
Let's not give anybody ideas now shall we.
https://api.newag.pl/shouldirunornot?lat=&long=&trainid=
can't wait to see the swagger docs on this...
https://api.newag.pl/shouldirunornot?lat=&long=&trainid=
can't wait to see the swagger docs on this...
Just realized: “late stage enshittification of capitalism” is essentially “verelendung” as prophesized by Marx
"Immiseration" as known in English.
[dupe]
Does no one read the site anymore?
original news story discussion just over a week ago: https://news.ycombinator.com/item?id=38530885
And the followup from the company
Polish train maker denies claims its software bricked competitor rolling stock https://news.ycombinator.com/item?id=38570654
Not to mention the 404media dupe with a ton of votes just yesterday also!
https://news.ycombinator.com/item?id=38628635
Does no one read the site anymore?
original news story discussion just over a week ago: https://news.ycombinator.com/item?id=38530885
And the followup from the company
Polish train maker denies claims its software bricked competitor rolling stock https://news.ycombinator.com/item?id=38570654
Not to mention the 404media dupe with a ton of votes just yesterday also!
https://news.ycombinator.com/item?id=38628635
>>Does no one read the site anymore?
I don't know, is it some kind of requirement to read every piece of news on the main page every single day to keep visiting? Or can I log in every few days, read a few stories, upvote the ones I like, then visit again few days later? Or is that not allowed?
I don't know, is it some kind of requirement to read every piece of news on the main page every single day to keep visiting? Or can I log in every few days, read a few stories, upvote the ones I like, then visit again few days later? Or is that not allowed?
More on the submitter in this case.
Upvote what you see/like, sure, all good.
But if you're not around, you miss some stories, that's it. (especially when it's come up so many times over weeks) There is a current events aspect to HN.
Upvote what you see/like, sure, all good.
But if you're not around, you miss some stories, that's it. (especially when it's come up so many times over weeks) There is a current events aspect to HN.
> Does no one read the site anymore?
I don't read the vast majority of sites linked here. I come here for the comments. I assume anything that's important enough to know will be directly quoted in a comment here.
I don't read the vast majority of sites linked here. I come here for the comments. I assume anything that's important enough to know will be directly quoted in a comment here.
[deleted]
I was expecting some malicious code like "knock 3 times on this door to enter, otherwise the door locks for good". No. Instead it was "if you're geographically parked at our competitor's lat/long then we'll brick the train".
There's ways to add back doors and other fun easter eggs in code, but wow they picked a really cumbersome and obvious method of exploitation.
There's ways to add back doors and other fun easter eggs in code, but wow they picked a really cumbersome and obvious method of exploitation.
You almost guessed it though. There was a (since removed) secret combination of buttons on driver console that magically unlocked the train.
"hesoyam"
Reading their official statement it's hard to not get the impression that it wasn't even consulted with their attorney - they deny specific allegations in a way that just begs further investigation.
I mean, I'm Polish and worked for such people in the past. The statement reeks of an approach to business that I wish went extinct already, but apparently you still see pockets of it here and there.
I mean, I'm Polish and worked for such people in the past. The statement reeks of an approach to business that I wish went extinct already, but apparently you still see pockets of it here and there.
The real question is whether or not we will get quality reporting from the courts! Hopefully someone goes to jail over this.
pretty sure even if they have source code version control, the "funny" patches are outside of the version control and applied on top before making a build.
That's almost certainly not the case. In general for vehicles, you're installing signed, verified images. Not only would it be a LOT harder to consistently apply OOVC patches that don't show up anywhere, having signed builds that don't match your source would be a worse red flag than just having the code in version control.
You do get deniability though. Given their line is "this is malicious code that was put on the train. Physical security is the responsibility of the train companies”, they would argue the fact their binaries are different is proof of malicious actors.
Essentially, if they're going to do this, a good route is for them to be the malicious actors. Just before shipping, someone trusted surreptitiously switches the binaries to the modded versions.
Essentially, if they're going to do this, a good route is for them to be the malicious actors. Just before shipping, someone trusted surreptitiously switches the binaries to the modded versions.
yes, they probably have a "clean" build and "a build with extra functionality" build.
So, they could even install a clean build for some trains.
Though, investigators could just compare firmware from all trains made so far. Another question if it would be even be investigated.
Right to repair for anything we buy is essential and customers need the right to access software they have purchased.
I'm slightly envious of the reverse engineers that got to dig into this.
I can imagine the initial disbelief and shock as you start to piece together what the software is doing.
So malicious, I wouldn't believe it at first.
I can imagine the initial disbelief and shock as you start to piece together what the software is doing.
So malicious, I wouldn't believe it at first.
I have seen developers lock/encrypt their code they write and then when they get fired for it, they want the company to pay a fee to get them to unlock it/release source code
That is textbook https://en.wikipedia.org/wiki/Logic_bomb and is very "jailable" - as if plain blackmail laws weren't enough.
Edit: someone already added Newag there :D
Edit: someone already added Newag there :D
They better have some code around it that stops that from happening when the train is in service.
We lament that circumventing DRM is legally perilous, but sabotaging your customer's property is illegal. That you were the one that sold that property makes no difference - you sold it, it's not yours anymore, despite corporate messaging otherwise.
DeathArrow(4)
iamleppert(7)
This is on the general topic of right to repair and freedom of fixing and so forth so naturally HN hates the train company.
On the other hand, trains that refuse to start don't immediately kill everyone on board, and trains that have fall off the tracks do tend to kill people. So when a train fails and people are dead, and people go in search of who is responsible, it seems decently likely that ambiguity between the third party repair shop and the company responsible for the train is a problem.
Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There should be an opt out system - something where the train operator contacts the train supplier and says "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Or to bring it closer to home, say one of us breaks into our tesla to exercise our God given right to change the software stack, and then it kills the driver and various people around it, to what extent is that Tesla's fault? Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
On the other hand, trains that refuse to start don't immediately kill everyone on board, and trains that have fall off the tracks do tend to kill people. So when a train fails and people are dead, and people go in search of who is responsible, it seems decently likely that ambiguity between the third party repair shop and the company responsible for the train is a problem.
Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There should be an opt out system - something where the train operator contacts the train supplier and says "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Or to bring it closer to home, say one of us breaks into our tesla to exercise our God given right to change the software stack, and then it kills the driver and various people around it, to what extent is that Tesla's fault? Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
You are putting forth some general rules that seem reasonable, but they don't really apply in this situation.
From what I've read, the train operator bid out the maintenance for these trains, and the manufacturer lost the bid to a third party. So your "opt-out" system should absolutely have been triggered here.
If there was some reason to believe that the low bidder would be incapable of safely performing the maintenance, the train manufacturer should have raised that issue publicly, not silently sabotaged the low bidder's ability to perform the maintenance.
From what I've read, the train operator bid out the maintenance for these trains, and the manufacturer lost the bid to a third party. So your "opt-out" system should absolutely have been triggered here.
If there was some reason to believe that the low bidder would be incapable of safely performing the maintenance, the train manufacturer should have raised that issue publicly, not silently sabotaged the low bidder's ability to perform the maintenance.
Maybe? I'm speculating here as it's not my field. The obvious explanation to me is very different to the obvious explanation to every other comment I've seen so far. That difference seemed interesting enough to be worth posting.
The question of where liability falls after a repair isn't covered in the articles linked unfortunately. It seems plausible that it was initially whoever wrote the software and is now someone else.
I think it's extremely likely that this disable-train-on-various-conditions behaviour is exactly as specified and documented since train software is a bit obsessed with formal methods and documentation. It's then only "silent sabotage" to the extent that said docs were ignored.
It's credible that a way to easily disable these safety checks for the case where it's now someone else's liability wasn't considered worth paying for by whoever bought the train. It's tomorrow's problem after all. Also the customer may not be totally convinced of the necessity of the software dev paranoia, especially if they're not liable for failures.
So sure, maybe this is evil/negligent software people at the train company. I don't see that conclusion well supported by the article.
The question of where liability falls after a repair isn't covered in the articles linked unfortunately. It seems plausible that it was initially whoever wrote the software and is now someone else.
I think it's extremely likely that this disable-train-on-various-conditions behaviour is exactly as specified and documented since train software is a bit obsessed with formal methods and documentation. It's then only "silent sabotage" to the extent that said docs were ignored.
It's credible that a way to easily disable these safety checks for the case where it's now someone else's liability wasn't considered worth paying for by whoever bought the train. It's tomorrow's problem after all. Also the customer may not be totally convinced of the necessity of the software dev paranoia, especially if they're not liable for failures.
So sure, maybe this is evil/negligent software people at the train company. I don't see that conclusion well supported by the article.
Your argument is based on an incorrect understanding of liability, and is also nullified by the fact that these measures, allegedly justified on safety grounds, were not just undocumented (see other sources) but clearly deliberately hidden, to the point where the manufacturer pretended not to know why the trains would not start.
Update: here's a quote from the Ars Tech. article itself, showing a) no tampering with the trains' software or hardware was necessary to get them running, and b) there's no even slightly plausible case for believing the manufacturer was unaware of this. Furthermore, if it really was somehow unaware of what was going on, then these 'features' cannot be justified as a safety measure.
Dragon Sector got the trains running after discovering "an undocumented ‘unlock code’ which you could enter from the train driver’s panel which magically fixed the issue."
Update 2: It is barely plausible that the ability to track the trains' presence in other maintenance shops was initially added to gather evidence for liability and warranty purposes, and then someone had the dumb idea of using this to covertly disable trains when this happened.
Note that the use of a third-party maintenance operation was not a secret: the train operator solicited bids for the work, and the manufacturer tendered one. Clearly, third-party maintenance was not in violation of any contract (and if somehow it was, the manufacturer did not need to gather any covertly-acquired evidence for a breach-of-contract suit.)
Update: here's a quote from the Ars Tech. article itself, showing a) no tampering with the trains' software or hardware was necessary to get them running, and b) there's no even slightly plausible case for believing the manufacturer was unaware of this. Furthermore, if it really was somehow unaware of what was going on, then these 'features' cannot be justified as a safety measure.
Dragon Sector got the trains running after discovering "an undocumented ‘unlock code’ which you could enter from the train driver’s panel which magically fixed the issue."
Update 2: It is barely plausible that the ability to track the trains' presence in other maintenance shops was initially added to gather evidence for liability and warranty purposes, and then someone had the dumb idea of using this to covertly disable trains when this happened.
Note that the use of a third-party maintenance operation was not a secret: the train operator solicited bids for the work, and the manufacturer tendered one. Clearly, third-party maintenance was not in violation of any contract (and if somehow it was, the manufacturer did not need to gather any covertly-acquired evidence for a breach-of-contract suit.)
>It's then only "silent sabotage" to the extent that said docs were ignored.
From what I've read, the maintenance company did scour all documentation and found no explanation for why the train was disabled. It was an entirely undocumented feature. Maybe they were just lying, but this idea you have in your head that it was just a fly-by-night operation who couldn't be bothered to read the documentation is contradicted by the available public record.
From what I've read, the maintenance company did scour all documentation and found no explanation for why the train was disabled. It was an entirely undocumented feature. Maybe they were just lying, but this idea you have in your head that it was just a fly-by-night operation who couldn't be bothered to read the documentation is contradicted by the available public record.
I haven't read the Ars article yet, but imo the evil/negligence of the train manufacturer is reasonably well supported by this article: https://www.404media.co/polish-hackers-repaired-trains-the-m...
So all of this obviously biased and incorrect speculation on your part is based on.... not being an expert in the field... and not having read the actual article? Thanks man, very insightful.
Sorry to be blunt, but your longwinded "speculation" in fields you don't understand is not a useful contribution to the discussion.
The difference is:
- there's a proper registry of who is responsible for maintenance of the train, including chain of liability. Crucially, the vendor wasn't in it for the affected trains - they'd only be liable if design issues were found.
- maintenance companies was appropriately certified and verified - we're talking professional MRO, not random workshop
- owners of the trains were also registered as responsible parties for maintenance and repair
- vendor was supposed to provide documentation that would allow such a workshop to perform maintenance and repairs up to P5 (largest scale) maintenance
- vendor never disclosed lockouts they implemented, thus committing fraud at the very least.
- there's a proper registry of who is responsible for maintenance of the train, including chain of liability. Crucially, the vendor wasn't in it for the affected trains - they'd only be liable if design issues were found.
- maintenance companies was appropriately certified and verified - we're talking professional MRO, not random workshop
- owners of the trains were also registered as responsible parties for maintenance and repair
- vendor was supposed to provide documentation that would allow such a workshop to perform maintenance and repairs up to P5 (largest scale) maintenance
- vendor never disclosed lockouts they implemented, thus committing fraud at the very least.
Sounds reasonable to me. Plus you've used domain specific terminology which suggests you know more about this than I do. Thanks for the context.
I definitely haven't gone looking for the documentation on the train system, partly because I assume it's unavailable to the public and mostly because I assume it's enormous in extent.
I definitely haven't gone looking for the documentation on the train system, partly because I assume it's unavailable to the public and mostly because I assume it's enormous in extent.
The same viewpoint applies to any motor vehicle too. How many road accidents are due to mechanical error introduced by a mechanic (that does not operate on behalf of the car manufacturer)? What makes an "official" mechanic employed by the manufacturer superior to a third party?
This is an interesting discussion, thank you for your nuanced opinion. I don't know where I stand on it but you are spot on about the demographics of HN. I certainly have a knee-jerk reaction in the direction of operator freedom.
Edit: Additionally, if something like this goes wrong (say the train derails) it will be a massive blow to right to repair. Even if it is later found that the third party repair is not responsible, the media will scoop up this story and it will forever warp the public's ideas about right to repair.
This is an interesting discussion, thank you for your nuanced opinion. I don't know where I stand on it but you are spot on about the demographics of HN. I certainly have a knee-jerk reaction in the direction of operator freedom.
Edit: Additionally, if something like this goes wrong (say the train derails) it will be a massive blow to right to repair. Even if it is later found that the third party repair is not responsible, the media will scoop up this story and it will forever warp the public's ideas about right to repair.
That's what we have regulations for. If the shops follow the regulations and have the appropriate certifications we know that the work they did is up to spec. Also, obviously in case of some problem the inspectors will check for the root cause and if the repair shop is established as the reason they are liable, not the train company. This is all established law since .. ages.
This is just the next episode in an age old battle. Old cars were easy to repair, so people often went to independent repair shops. Companies obviously didn't like this, so they started using proprietary screws. Then laws were made to forbid this. Next round came when cars got computers and companies didn't give the repair shops the required software and/or hardware to work with them. So, laws got made which forced them to. And so on. Each time there's some new technology companies will try to find ways to misuse it to the detriment of customers. And each time law makers will have to get involved and put the screws on them. And then people later go around "why do we have so many laws" .. because companies are shit, if they can get away with it. It's their natural impulse.
This is just the next episode in an age old battle. Old cars were easy to repair, so people often went to independent repair shops. Companies obviously didn't like this, so they started using proprietary screws. Then laws were made to forbid this. Next round came when cars got computers and companies didn't give the repair shops the required software and/or hardware to work with them. So, laws got made which forced them to. And so on. Each time there's some new technology companies will try to find ways to misuse it to the detriment of customers. And each time law makers will have to get involved and put the screws on them. And then people later go around "why do we have so many laws" .. because companies are shit, if they can get away with it. It's their natural impulse.
> That's what we have regulations for.
Regulations don't work. There needs to be legal liability for the fraud perpetrated by the vendor against the customer, not prescriptive rules that vendors influence, then follow dogmatically so they can cite compliance in order to disclaim liability for their intended bad outcomes.
Regulations don't work. There needs to be legal liability for the fraud perpetrated by the vendor against the customer, not prescriptive rules that vendors influence, then follow dogmatically so they can cite compliance in order to disclaim liability for their intended bad outcomes.
Regulations are the basis for legal liability. You cannot put someone in prison or fine them without a law that says what they did was wrong.
Statute and common law are the basis for legal liability. The modern concept of regulation, as promulgated via various administrative agencies, is distinct from, and often displaces, the traditional function of law.
In theory, I agree with your viewpoint. Yes, obviously, safety critical systems need to remain safe, and a workshop that isn't capable of performing maintenance to the degree mandated by the train software shouldn't be doing it at all.
That is not the case here.
If that was indeed the case, one would expect the train's diagnostic systems to report that it has detected certain issues that need to be resolved before the train will start up again. But that didn't happen, instead the train simply refused to start, no error codes, nada.
Not only that, but further investigation revealed that the train's firmware contained code that would disable the train if it was present in a competitors shop for a considerable time.
> There should be an opt out system
This mechanism for disabling the trains, was never mentioned in the discussions or contracts for purchasing the train (which is why a third-party could win the repair contract). There was never an option presented to "opt out", that is why people are saying that the trains were sabotaged.
> to what extent is that Tesla's fault?
It isnt. You made unsafe modifications that can be clearly shown to have caused damage, it's your liability.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Depends, did their software present an actionable error or not? If the car gave an error code, and you look in the manual and it says that too much current is going to the motors (idk), and then you fix that and the car starts? Great! But if the car just refuses to start without any indication as to what is wrong, and further even if all repairs are reversed it still no longer starts? That sounds like deliberately sabotaging third-party repair.
That is not the case here.
If that was indeed the case, one would expect the train's diagnostic systems to report that it has detected certain issues that need to be resolved before the train will start up again. But that didn't happen, instead the train simply refused to start, no error codes, nada.
Not only that, but further investigation revealed that the train's firmware contained code that would disable the train if it was present in a competitors shop for a considerable time.
> There should be an opt out system
This mechanism for disabling the trains, was never mentioned in the discussions or contracts for purchasing the train (which is why a third-party could win the repair contract). There was never an option presented to "opt out", that is why people are saying that the trains were sabotaged.
> to what extent is that Tesla's fault?
It isnt. You made unsafe modifications that can be clearly shown to have caused damage, it's your liability.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Depends, did their software present an actionable error or not? If the car gave an error code, and you look in the manual and it says that too much current is going to the motors (idk), and then you fix that and the car starts? Great! But if the car just refuses to start without any indication as to what is wrong, and further even if all repairs are reversed it still no longer starts? That sounds like deliberately sabotaging third-party repair.
> This is on the general topic of right to repair and freedom of fixing and so forth so naturally HN hates the train company.
Actually, the issue here is that the train company is on the wrong side of this issue.
> Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There are tons of cases where the repairer, not the OEM is held liable. The manufacturer is at fault for the defective parts/product. The repairer is liable for a defective repair. Courts have been sorting this kind of thing out for centuries and it's not a difficult thing to deal with.
> "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Except that really isn't the case at all, and hasn't been the case. If the repairer does not do the repair correctly, the repairer is liable, not the manufacturer. Most mechanical shops even carry insurance and offer their own warranties on their work. Most cities (and hopefully consumers) will only do business with repair shops that are insured for this exact reason.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Yes. It is not their car. It is my car.
Ford is not liable if that variable geometry camshaft I installed in my Mustang causes the engine to blow, killing six people with shrapnel. I would be liable for that. Why should Tesla be liable for someone hacking the software? The issue here is the world doesn't work the way you are describing, where manufacturers are held liable for the work of third parties. Tesla is held liable for their own warranties and laws applying to product safety.
Actually, the issue here is that the train company is on the wrong side of this issue.
> Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There are tons of cases where the repairer, not the OEM is held liable. The manufacturer is at fault for the defective parts/product. The repairer is liable for a defective repair. Courts have been sorting this kind of thing out for centuries and it's not a difficult thing to deal with.
> "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Except that really isn't the case at all, and hasn't been the case. If the repairer does not do the repair correctly, the repairer is liable, not the manufacturer. Most mechanical shops even carry insurance and offer their own warranties on their work. Most cities (and hopefully consumers) will only do business with repair shops that are insured for this exact reason.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Yes. It is not their car. It is my car.
Ford is not liable if that variable geometry camshaft I installed in my Mustang causes the engine to blow, killing six people with shrapnel. I would be liable for that. Why should Tesla be liable for someone hacking the software? The issue here is the world doesn't work the way you are describing, where manufacturers are held liable for the work of third parties. Tesla is held liable for their own warranties and laws applying to product safety.
I once saw a Boeing 727 that had been retrofitted with winglets. That was allowed, but Boeing no longer guaranteed the wings.
So yes, it works just as you describe. (Planes, not trains, I know, but it's still a datapoint on how liability works out there.)
So yes, it works just as you describe. (Planes, not trains, I know, but it's still a datapoint on how liability works out there.)
[deleted]
[deleted]
> The code also allegedly bricked the train if "certain components had been replaced without a manufacturer-approved serial number," 404 Media reported.
If only commercial airliners had this too...
https://news.sky.com/story/fraud-officers-arrest-one-in-dawn...
If only commercial airliners had this too...
https://news.sky.com/story/fraud-officers-arrest-one-in-dawn...
Commercial airliners have other procedures, including - just like with trains in europe - complete unbundling of maintenance and repair.
Because it's pretty normal for an aircraft to be in use longer than the manufacturer exists.
Because it's pretty normal for an aircraft to be in use longer than the manufacturer exists.
- https://news.ycombinator.com/item?id=38628635
- https://news.ycombinator.com/item?id=38632033