Therac-25(en.wikipedia.org)
en.wikipedia.org
Therac-25
https://en.wikipedia.org/wiki/Therac-25
15 comments
It's worth noting that we don't know for sure how many people this killed, we only have a lower bound.
The Causality episode on this is really good: https://engineered.network/causality/episode-39-therac-25/
The always-excellent Plainly Difficult ran an episode [0] on this topic a few years ago.
[0] https://www.youtube.com/watch?v=-7gVqBY52MY
[0] https://www.youtube.com/watch?v=-7gVqBY52MY
Kyle Hill also made a video on the topic. Different video styles if Plainly Difficult isn't your jam.
https://www.youtube.com/watch?v=Ap0orGCiou8
https://www.youtube.com/watch?v=Ap0orGCiou8
> 1986, the programmer left AECL. In a lawsuit, lawyers could not identify the programmer or learn about his qualification and experience.
I find this hard to believe: how? Are there no records?
Edit: apparently the identity is known, but the lawsuit was settled before a deposition was required. Also:
https://www.reddit.com/r/programming/comments/mzht1/comment/...
I find this hard to believe: how? Are there no records?
Edit: apparently the identity is known, but the lawsuit was settled before a deposition was required. Also:
https://www.reddit.com/r/programming/comments/mzht1/comment/...
So they took components from a previous system, didn't test properly on the new system and today we inherit a whole lot of regulation for medical equipment because of this. Not that it's a bad thing that we ensure proper testing has occurred but there are so many cases where legally this applies even if it's very peripheral (like say a UI change in a menu item of something not related to medical equipment but is never the less under compliance)
> (like say a UI change in a menu item of something not related to medical equipment but is never the less under compliance)
I think regulating UI changes in medical equipment (or in other devices that could potentially kill people) is very much justified: Even if the software is nominally working as intended, if some update made unexpected ui changes that end up confusing the operator, this could lead to dangerous situations just as well as if the software had a bug.
I agree that this would be overblown for ui in non-critical places, but I didn't have the impression there is any heavy-handed regulation in those areas (otherwise, I'd expect we'd see fewer dark patterns and other "ux" redesigns that are clearly for "engagement optimizing" and not to make any task easier for the user)
Could you give some examples of non-medical situations where regulation gets in the way of ui redesigns?
I think regulating UI changes in medical equipment (or in other devices that could potentially kill people) is very much justified: Even if the software is nominally working as intended, if some update made unexpected ui changes that end up confusing the operator, this could lead to dangerous situations just as well as if the software had a bug.
I agree that this would be overblown for ui in non-critical places, but I didn't have the impression there is any heavy-handed regulation in those areas (otherwise, I'd expect we'd see fewer dark patterns and other "ux" redesigns that are clearly for "engagement optimizing" and not to make any task easier for the user)
Could you give some examples of non-medical situations where regulation gets in the way of ui redesigns?
>So they took components from a previous system, didn't test properly on the new system
That is also how the first Ariane 5 crashed.
That is also how the first Ariane 5 crashed.
Perhaps one of the best case studies in software ethics
And the only event I know of for which software was found to be the root cause for human deaths.
They you haven't spent any time at all searching. Software-induced plane crashes have been happening for decades, with the extremely high profile case of the 737 MAX being quite recent.
Honestly I don't think the 737 MAX was a software failure so much as a weird intersection of failures in:
- regulation (FAA incentivized not modifying the primary frame/wings),
- aerospace engineering (weight too high for existing control surfaces, necessitating a weird control surface on the tail),
- systems engineering (specifying activating MCAS during a 1-out-of-2 sensor failure instead of 2-out-of-3/etc),
- and training. The procedures for new MCAS system changes weren't drilled into pilots to perpetuate the perception that this was still the same 737 and didn't need in-depth FAA review. In fact, Boeing managers and the FAA removed references to MCAS from Boeing's flight crew operations manual (FCOM).
The software didn't seem like it had a bug or even particularly poorly designed for the intended system. I'm not sure I'd find the software engineers at fault...they followed the specification given by the systems engineers, and it would have looked reasonable to a software engineer. The intended overall system was fucking atrocious, however. I'd absolutely lay the blame for the 737 MAX incidents on the business managers, aerospace engineers, and especially the systems engineers.
MCAS involved a good amount of software, but it wasn't the software which caused the failure.
- regulation (FAA incentivized not modifying the primary frame/wings),
- aerospace engineering (weight too high for existing control surfaces, necessitating a weird control surface on the tail),
- systems engineering (specifying activating MCAS during a 1-out-of-2 sensor failure instead of 2-out-of-3/etc),
- and training. The procedures for new MCAS system changes weren't drilled into pilots to perpetuate the perception that this was still the same 737 and didn't need in-depth FAA review. In fact, Boeing managers and the FAA removed references to MCAS from Boeing's flight crew operations manual (FCOM).
The software didn't seem like it had a bug or even particularly poorly designed for the intended system. I'm not sure I'd find the software engineers at fault...they followed the specification given by the systems engineers, and it would have looked reasonable to a software engineer. The intended overall system was fucking atrocious, however. I'd absolutely lay the blame for the 737 MAX incidents on the business managers, aerospace engineers, and especially the systems engineers.
MCAS involved a good amount of software, but it wasn't the software which caused the failure.
What about the infamous [patriot missile failure](https://www-users.cse.umn.edu/~arnold/disasters/patriot.html) in which faulty floating point arithmetic caused a missile not to be intercepted, killing 28?
Ah, very cool example!
In school it was my literal case study in software ethics.