The Therac-25 Incident(thedailywtf.com)
thedailywtf.com
The Therac-25 Incident
https://thedailywtf.com/articles/the-therac-25-incident
11 comments
This is such a tragic story, I think anyone working in tech should be forced to read this once a year to remind them that the work we do can massively impact people (even if it's very abstracted from us!)
Tech people should really spend some time reading up on history, even if it's just very tech-relevant history like engineering disasters such as the Therac-25. Even origin stories are important: techno-libertarianism is absolutely baffling, as Silicon Valley and Internet were seeded by massive government/military spending.
Know where you came from. Apply lessons from the past.
Know where you came from. Apply lessons from the past.
One mistake is too many when human lives are on the line.
I first learned of this incident in the early 2000s from a computer book about how to code UML State charts in C/C++ [1]. It included a CD-ROM with a library that implemented a rather clever way of maintaining state using the call stack.
Unfortunately, the naive software fix for the Therac-25 is to add a flag (which is additional state) to the system which is to add complexity to the system.
[1] https://www.state-machine.com/doc/PSiCC.pdf
Unfortunately, the naive software fix for the Therac-25 is to add a flag (which is additional state) to the system which is to add complexity to the system.
[1] https://www.state-machine.com/doc/PSiCC.pdf
We had a term for implicit state machines at a place I used to work: infinite flag machines[0].
[0] I suspect the majority of hn folks are familiar with finite state machines, but just in case: https://en.m.wikipedia.org/wiki/Finite-state_machine
[0] I suspect the majority of hn folks are familiar with finite state machines, but just in case: https://en.m.wikipedia.org/wiki/Finite-state_machine
I remember starting to use software to stop a machine when the door was opened. A more experienced engineer stopped that and we added relay hard logic. That was 40 years ago, it's clear as day. Exactly the same issue with the Therac-25.
The real takeaway is that letting a single software engineer ship all the code for a safety-critical medical device is a really bad idea. Software engineering in the 80s was another dimension.
It is always sobering to read this story. I feel an obligation to read it through every time I stumble upon it.
A lot of what we do as software developers really doesn't matter, so it's easy to forget that sometimes it does.
Regulations are written in blood.
A lot of what we do as software developers really doesn't matter, so it's easy to forget that sometimes it does.
Regulations are written in blood.
>We know that the software for the Therac-25 was developed by a single person, using PDP 11 assembly language, over a period of several years.
>The programmer left AECL in 1986. In a lawsuit connected with one of the accidents, the lawyers were unable to obtain information about the programmer from AECL. In the depositions connected with that case, none of the AECL employees questioned could provide any information about his educational background or experience.
http://www.cse.msu.edu/~cse470/Public/Handouts/Therac/Side_b...
Surely in the 80s someone working professionally on such a system had some university education?
>The programmer left AECL in 1986. In a lawsuit connected with one of the accidents, the lawyers were unable to obtain information about the programmer from AECL. In the depositions connected with that case, none of the AECL employees questioned could provide any information about his educational background or experience.
http://www.cse.msu.edu/~cse470/Public/Handouts/Therac/Side_b...
Surely in the 80s someone working professionally on such a system had some university education?
Which would prevent how exactly such an incident? Of all the many reasons for this failure, that programmers educational background is probably the least one contributing factor.