On the Term “Blameless Postmortem”(tylercipriani.com)
tylercipriani.com
On the Term “Blameless Postmortem”
https://tylercipriani.com/blog/2022/01/27/on-the-term-blameless-postmortem/
21 comments
I'd like to hear more about how you can avoid names. How are actions items assigned? Genuinely curious, this sounds cool
We say 'developer' not 'Bob'. The actions are typically things that are not for one person, but rather our scrum teams as a whole. For example, had a junior tasked with a 'rush' action for modifying an index. They botched the script and dropped all documents when they attempted to modify an index. Took down everything.
The chain that got us there... Developers had drop access to the prod system - all of them. That got fixed. The account that they were using had more permissions then should be used. Reviews were not done, because rush rush and vacations, the developer did some stack overflow that had unintended consequences. Tests matter. A few other bits that were slipping became checklist items for the change control.
It was far more important to fix the processes and gaps than say 'Bob' screwed it up. The other bit was when folks were chewed on, they hit the mistakes. It consumed so much time, when someone would log in and 'fix' something with a system account, and we had to guess what broke the system.
It takes time. I was so, so very skeptical when we started. It sounded like someone just avoiding responsibility. First few, were. Looking at what broke, why it broke, and how do we avoid it in the future.
At the end of the year, I'll also use these writeups to see where our emphasis areas should be. 2021's top - humans suck at keeping x509 certificates up to date when manually added.
The chain that got us there... Developers had drop access to the prod system - all of them. That got fixed. The account that they were using had more permissions then should be used. Reviews were not done, because rush rush and vacations, the developer did some stack overflow that had unintended consequences. Tests matter. A few other bits that were slipping became checklist items for the change control.
It was far more important to fix the processes and gaps than say 'Bob' screwed it up. The other bit was when folks were chewed on, they hit the mistakes. It consumed so much time, when someone would log in and 'fix' something with a system account, and we had to guess what broke the system.
It takes time. I was so, so very skeptical when we started. It sounded like someone just avoiding responsibility. First few, were. Looking at what broke, why it broke, and how do we avoid it in the future.
At the end of the year, I'll also use these writeups to see where our emphasis areas should be. 2021's top - humans suck at keeping x509 certificates up to date when manually added.
I coined this term (https://codeascraft.com/2012/05/22/blameless-postmortems/) and believe (productive and earnest) critique can be valuable when it comes to terms like this one. (See also “devops”, “agile”, “serverless”, “toil”, “technical debt”, and others)
Paul Reed has made a great case for ‘blame-aware’ (https://medium.com/@jpaulreed/why-blameless-postmortems-migh...)
Rein Henrichs also explores the topic very well through a unique lens (https://youtu.be/KXrsvLMqF1Q)
As Cipriani points out, there is the term and there is the concept(s) the term is intended to convey. It does seem to me more folks know the term than have read the origin (my post above from 2012). This is fine, as long as productive dialogue continues in the industry about what the term was made to convey.
Paul Reed has made a great case for ‘blame-aware’ (https://medium.com/@jpaulreed/why-blameless-postmortems-migh...)
Rein Henrichs also explores the topic very well through a unique lens (https://youtu.be/KXrsvLMqF1Q)
As Cipriani points out, there is the term and there is the concept(s) the term is intended to convey. It does seem to me more folks know the term than have read the origin (my post above from 2012). This is fine, as long as productive dialogue continues in the industry about what the term was made to convey.
"blameless postmortem" was a thing at google well before 2012.
Possibly the concept existed but the “blameless” term is widely credited to the GP, John Allspaw, who became CTO of Etsy. At least I haven’t found anything definitive predating him.
Also impressive he’s had an HN acct for 12 years but only 5 karma!
Also impressive he’s had an HN acct for 12 years but only 5 karma!
The SRE book authors cited the post specifically (https://sre.google/sre-book/bibliography/#All12), and Googlers I’ve spoken to mentioned its influence there.
The Paul Reed post is excellent. The term "blame-aware" is a useful frame for discussion: it reminds people that the goal is not to blame AND it's human nature to place blame.
Thank you for posting this, and, of course, for the original concept :)
Thank you for posting this, and, of course, for the original concept :)
You can't debate the value of the term without considering the conditions that led to it. Why would someone call a postmortem "blameless"? Because (in some companies) there absolutely was a culture of blame, which made people less forthcoming and thoughtful about the causes of incidents, which limited the potential learnings. This term was not pulled out of thin air, or built upon some imaginary possible blame. It was designed to explicitly remove blame that was already present in the culture.
In particular, firing the dev that wrote the buggy code or the SRE that pushed the bad change is an obvious management reaction that "blameless postmortem" seeks to redress. I'm happy for OP that they've never worked somewhere that toxic but those places absolutely exist.
Oh, God. If you believe "disquisition" carries less negative connotations than "blameless postmortem", you completely failed at reading the audience.
Aviation uses the word "investigation", by the way. But they can only omit the "blameless" part because there are very strong guarantees that it will be blameless.
Aviation uses the word "investigation", by the way. But they can only omit the "blameless" part because there are very strong guarantees that it will be blameless.
I prefer “retrospective”, which doesn’t sound like a police investigation or bring to mind airplane crashes.
It’s also easier to do those on a weekly basis, so there’s less of a Pavlovian association of “bad thing happens” then “synonym for postmortem happens”.
It’s also easier to do those on a weekly basis, so there’s less of a Pavlovian association of “bad thing happens” then “synonym for postmortem happens”.
Root Cause Analysis is the term most used for this kind of exercise in many (non-software) engineering disciplines.
The article even mentions root cause so not sure how they didn't find that term in their searching.
When doing a root cause analysis where one or more of the identified causes is a person failed to perform an action adequately, the "blameless" bit is supposed to be about digging further to understand why they didn't.
Was it lack of training or experience? or maybe they were overloaded with work (fatigued, distracted)? or was it due to pressure to get the thing done by a deadline and therefore steps were skipped? etc
A retrospective, to me is more about continual improvement. Looking at what worked and what didn't in the last project/sprint. A root cause analysis is a different thing that you do when something has gone wrong that shouldn't have that had major consequences.
The article even mentions root cause so not sure how they didn't find that term in their searching.
When doing a root cause analysis where one or more of the identified causes is a person failed to perform an action adequately, the "blameless" bit is supposed to be about digging further to understand why they didn't.
Was it lack of training or experience? or maybe they were overloaded with work (fatigued, distracted)? or was it due to pressure to get the thing done by a deadline and therefore steps were skipped? etc
A retrospective, to me is more about continual improvement. Looking at what worked and what didn't in the last project/sprint. A root cause analysis is a different thing that you do when something has gone wrong that shouldn't have that had major consequences.
I like the idea here, but in my experience the biggest practical issue with postmortems is getting people to actually do them. A heavy term serves as a reminder that it's an important investigation, it has to get done, we can't just put it off until it fits conveniently into the schedule. I worry whether a lighter-sounding term would make it easier for people to work on their projects first and delay post-incident investigations indefinitely.
The name needs to change but also the attitude that as engineers, we build complex systems and assume everyone has the knowledge how to use it. A few world wide outages I've been a part of was caused by a task runner which didn't lint the command and allowed a broken bash one-liner to be executed across every system in parallel.
Yes, it's a simple mistake but how was a system allowed access to our global environment that this edge case was never calculated? In many of the meetings, the common issue is communication even between co-workers on the same team, and between internal platform providers. One case was an outage on the storage backend and realized after a long meeting that the internal SLA was much greater than we expected (and which the systems would timeout). It only worked for so long as storage utilization was extremely low.
Yes, it's a simple mistake but how was a system allowed access to our global environment that this edge case was never calculated? In many of the meetings, the common issue is communication even between co-workers on the same team, and between internal platform providers. One case was an outage on the storage backend and realized after a long meeting that the internal SLA was much greater than we expected (and which the systems would timeout). It only worked for so long as storage utilization was extremely low.
That means we need to take a very close look at whether "Real programmers" are actually anyone to emulate.
Programming culture has almost football field level of "No time for weakness" attitude.
If I see a possible failure mode of a system, and bring it up, someone's going to tell me to stop being a clicky click windows idiot and learn to be careful.
Trying to prevent human error in software isn't seen as a priority so nobody does it. They are concerned with the most reliable code rather than the most reli4 code-user-hardware-task-schedule-conditions system.
Programmers need to accept software fixes for human and hardware failures. It's a lot easier to add a confirmation dialog than it is to somehow become 100% reliable at not clicking the wrong thing.
Programming culture has almost football field level of "No time for weakness" attitude.
If I see a possible failure mode of a system, and bring it up, someone's going to tell me to stop being a clicky click windows idiot and learn to be careful.
Trying to prevent human error in software isn't seen as a priority so nobody does it. They are concerned with the most reliable code rather than the most reli4 code-user-hardware-task-schedule-conditions system.
Programmers need to accept software fixes for human and hardware failures. It's a lot easier to add a confirmation dialog than it is to somehow become 100% reliable at not clicking the wrong thing.
I like the term “Incident report”.
Recently, I was a co-organizer for a conference. The event was almost over, and we were in the process of scheduling our event postmortem when one of our fellow organizers suffered the loss of a close family member.
As you can imagine, we all became instant members of the “we need a better name for this process” club. Thank you, OP for giving us a completely viable alternative. I’m calling it a “disquisition” from here on in.
As you can imagine, we all became instant members of the “we need a better name for this process” club. Thank you, OP for giving us a completely viable alternative. I’m calling it a “disquisition” from here on in.
> of scheduling our event postmortem
Did your group somehow started using 'postmortem' instead of 'afterparty'? Because if every event you organize require scheduling a postmortem then it is no longer a postmortem and just a regular activity.
Did your group somehow started using 'postmortem' instead of 'afterparty'? Because if every event you organize require scheduling a postmortem then it is no longer a postmortem and just a regular activity.
In any live event, things go wrong. Tearing in to what went well, what didn’t go well, and suggesting process changes for the next event is an important process not only for making future events easier, but also for improving the mental
Health of the organizers who tend to focus on the “what went wrong” parts.
So technically, we only hold one if things went wrong.
If we ever held an event where everything went right, then yes, we’d totally call that one an afterparty.
So technically, we only hold one if things went wrong.
If we ever held an event where everything went right, then yes, we’d totally call that one an afterparty.
Yes,
> Tearing in to what went well, what didn’t go well, and suggesting process changes for the next event
is the regular [performance|postfactum|event] analysis, but it IS NOT the postmortem!
From your previous comment you understand what the term means "after death". Did you really think "after death" and "after event" are the same?
> Tearing in to what went well, what didn’t go well, and suggesting process changes for the next event
is the regular [performance|postfactum|event] analysis, but it IS NOT the postmortem!
From your previous comment you understand what the term means "after death". Did you really think "after death" and "after event" are the same?
However... this has become one of the most powerful changes in our extended group's engineering culture. The focus on what happened and how, rather than the who. Each time, it is a chance to examine how things went kittywampus and how we can break that chain in the future. For the most part, I don't even think we include the names of the people involved. The process grew on me, the more we used it. Folks would fess up, rather than hide it and creating a snipe hunt for root cause. We were able to categorize where our pain point were and what we could do to stop it - and it worked. It really made the folks who could analyze what happened and how we could do better with what we have, shine.