Night of the Runbooks: a DevOps horror story(bitfieldconsulting.com)
bitfieldconsulting.com
Night of the Runbooks: a DevOps horror story
https://bitfieldconsulting.com/blog/night-of-the-runbooks
58 comments
If the runbooks are this clear (and the monitoring this accurate), you don't really need an on call engineer at all tbh. Just have an automated system look at the error graph and roll back to the previous release if the errors go up too much. There's a bunch of blog posts out there of companies that already do this.
Im not sure why a new version was rolled out by a team who weren’t monitoring it for at least the next 5 minutes.
Indeed. If you have a detailed, unambiguous, exact process that can resolve issues without a human having to make a judgement call .. why are you giving it to a human to execute?
The human's role in an ops situation is like that of the "safety driver" in self driving cars: do nothing most of the time, deal with emergencies, and take the blame for the failure of the system.
The human's role in an ops situation is like that of the "safety driver" in self driving cars: do nothing most of the time, deal with emergencies, and take the blame for the failure of the system.
I highly recommend against runbooks. There should be no need for runbooks. Anything that can be identified down to a single solution should have an automated response.
What you should have is hintbooks. For each alert, have something that identifies why you made the alert. What dashboards to check, what logs to read, what else might be going wrong.
What you should have is hintbooks. For each alert, have something that identifies why you made the alert. What dashboards to check, what logs to read, what else might be going wrong.
Automated responses take time and tech to implement. Playbooks are a way to iterate into automation. Good automated solutions often don't emerge fully baked.
By the time you have enough clarity to create a "playbook", you probably have enough clarity to reproduce and to automate.
That said, there are still a ton of solutions where "playbook" is kind of the right call. "DB is out of Disk Space" is probably "Increase the size of the disk reservation" for most modern cloud-native companies. They rarely rise to the level of detail that proponents of "Playbooks" suggest, though. How do you do that? Is it clicking through the UI? Do you have the CLI command prepared? Have you thought about why disk space usage has grown suddenly? This is the sort of thing that may be a no brainer you were expecting (in which case - Why didn't you complete the operation in-hours?) or it could be the "That's funny[1]" moment that precedes a major discovery.
[1] https://quoteinvestigator.com/2015/03/02/eureka-funny/
That said, there are still a ton of solutions where "playbook" is kind of the right call. "DB is out of Disk Space" is probably "Increase the size of the disk reservation" for most modern cloud-native companies. They rarely rise to the level of detail that proponents of "Playbooks" suggest, though. How do you do that? Is it clicking through the UI? Do you have the CLI command prepared? Have you thought about why disk space usage has grown suddenly? This is the sort of thing that may be a no brainer you were expecting (in which case - Why didn't you complete the operation in-hours?) or it could be the "That's funny[1]" moment that precedes a major discovery.
[1] https://quoteinvestigator.com/2015/03/02/eureka-funny/
In my experience, this is signaled by companies hiring "devops engineers". It shows lack of understanding that "devops" means devs and ops people should work together. And usually means they want someone to hold the bag because they can't seem to make their system work well, for some reason.
> To make this dream come true is far beyond the scope of just dev ops.
Nah. It's not unreasonable. The oddest thing was the feature that would save state and logs in a document somewhere. Logs and dashboards already allow you to focus on certain times, so no need to store that state somewhere else. In fact, that would be worse as it would discourage people from digging further.
> the runbook will need to be updated constantly after each release.
Nah. Nothing in there was unique or special about that specific release outside of the failover. And even the failover is general enough to be a failover. Everything else is standardized to the point of being boring.
> This means there's a long enough release cycle for this to ever happen which is also unlikely.
Daily.
Listen, nothing here was crazy. I've lived this. It's not hard. It's just boring conventions and consistency.
Again, literally the strangest part was saving the state and logs in some extra document.
The other weird thing was not checking the release logs (The release time was weird, too, but I'll ignore that). Rolling that release back would have been my first thought.
And none of this is difficult.
I realize not everyone is going to have the same experience, but it's not a dream, and thinking it's only a dream ignores how within-reach it really is.
Nah. It's not unreasonable. The oddest thing was the feature that would save state and logs in a document somewhere. Logs and dashboards already allow you to focus on certain times, so no need to store that state somewhere else. In fact, that would be worse as it would discourage people from digging further.
> the runbook will need to be updated constantly after each release.
Nah. Nothing in there was unique or special about that specific release outside of the failover. And even the failover is general enough to be a failover. Everything else is standardized to the point of being boring.
> This means there's a long enough release cycle for this to ever happen which is also unlikely.
Daily.
Listen, nothing here was crazy. I've lived this. It's not hard. It's just boring conventions and consistency.
Again, literally the strangest part was saving the state and logs in some extra document.
The other weird thing was not checking the release logs (The release time was weird, too, but I'll ignore that). Rolling that release back would have been my first thought.
And none of this is difficult.
I realize not everyone is going to have the same experience, but it's not a dream, and thinking it's only a dream ignores how within-reach it really is.
To me the strangest thing was that she rolled back the change, but somehow not with cicd. Meaning it's going to blow up again on next deploy.
Not necessarily.
Wait, is the idea that your SRE/ Ops person doing rollbacks, failovers, and restarts without understanding why... is a good thing?
Runbooks are supposed to give you a clear path during an incident so you don't run down rabbit holes, but you still shouldn't be running them without understanding what the impact of their steps actually is.
I thought the horror part was that some random person, who doesn't know the system, was doing all this stuff without just contacting the people who do know it.
If the runbook was sufficient to handle it, great! The proper team will have just as easy a time doing the steps, and not be mucking around in something they don't own and won't be the ones to fix when the runbook process itself breaks something because Error A is not Error J, but un-familiarized person can't tell the difference.
Runbooks are supposed to give you a clear path during an incident so you don't run down rabbit holes, but you still shouldn't be running them without understanding what the impact of their steps actually is.
I thought the horror part was that some random person, who doesn't know the system, was doing all this stuff without just contacting the people who do know it.
If the runbook was sufficient to handle it, great! The proper team will have just as easy a time doing the steps, and not be mucking around in something they don't own and won't be the ones to fix when the runbook process itself breaks something because Error A is not Error J, but un-familiarized person can't tell the difference.
The way I read it the horror was reading how that innocent guy was performing actions "they" didn't really understand without the setting itself being conclusive enough that "they" could actually trust those instructions blindly.
Did I read it wrong?
Did I read it wrong?
I think the ending is supposed to flip it into being that the person not having that runbook is the horror part.
it depends if the company designed it that way, in a bigger company I would have the policy that all deployments can be rolled back (except some bigger migrations)
Sure, but what if an infra change that is not part of that automated deployment also needs to happen?
It's one thing to allow anyone to *call* for a rollback of a deployment. It's another thing entirely to allow anyone to, independently and without sign-offs, *perform* a rollback.
It's one thing to allow anyone to *call* for a rollback of a deployment. It's another thing entirely to allow anyone to, independently and without sign-offs, *perform* a rollback.
> I don't know anything about process-payments; it belongs to a completely different team. I wouldn't even know how to check it.
I get that the point of the article is to sell the value of run books, which are amazing, but are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Even with perfect run books that seems unwise.
I get that the point of the article is to sell the value of run books, which are amazing, but are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Even with perfect run books that seems unwise.
I don't think it's that uncommon. When I joined my current org we spent much of my second week going through the apps we are responsible for, etc. But a week wasn't nearly enough to learn everything I needed to know about them. I was soon after added to the on-call rotation and received my first incident shortly after. A few weeks later, repeat. Each time I did what I knew and turned to my team to help with the rest - it has worked out fine so far and I learned so much more by getting hands on with failures compared to passively reading code & looking at diagrams.
But I work in e-commerce, not life-critical systems of any sort..
But I work in e-commerce, not life-critical systems of any sort..
My org is large enough to have a dedicated 24/7 front line team that look after dozens (hundereda) of things.
The churn in that team is high that you can’t really give them any significant training on every system. Even a 20 minute video would take weeks to onboard for a system they’ll likely never touch.
As far as I’m concerned a run book is there to stop me from being needlessly escalated to at 3am. If there’s a genuine problem then sure, I or another SME need to be contacted. Most of the time there’s a problem it is a reduction in resilience (so no need for out of hours contact), or something out of our control (loss of power in a location, failure of a circuit, in which case the 24 hour team can raise with the appropriate people.
I can count on one hand the number of problems worthy of a call out, and that’s in the last 7 years.
As such the first entry in my run book is “judging importance”.
The churn in that team is high that you can’t really give them any significant training on every system. Even a 20 minute video would take weeks to onboard for a system they’ll likely never touch.
As far as I’m concerned a run book is there to stop me from being needlessly escalated to at 3am. If there’s a genuine problem then sure, I or another SME need to be contacted. Most of the time there’s a problem it is a reduction in resilience (so no need for out of hours contact), or something out of our control (loss of power in a location, failure of a circuit, in which case the 24 hour team can raise with the appropriate people.
I can count on one hand the number of problems worthy of a call out, and that’s in the last 7 years.
As such the first entry in my run book is “judging importance”.
I'd say this strongly depends on the kind of error and the quality of the application involved.
Like, if the error is something technical - something is upset and refuses to talk with the database now, that's something you can identify and handle with minimal instructions. I'm working on a side-project at work to automatically generate dashboards for these technical foundations of services in a central discoverable place. Or, is the error something functional? Like, payments from Suaheli aren't being taxed right. There, a generic on-call has no chance. But why the hell would this not be caught in tests, or emerge at night out of office ours, and outside of a planned maintenance if you need that?
And then you're getting to the quality and if your application is a humble workhorse or a feisty goat. With a lot of our systems, we've pushed it to the point so in doubt you can just restart an instance and it most likely will end up better. This makes on-call somewhat chill - in most cases, you just keep kicking the system one or two times overnight until more knowledgeable people come online in the morning and dump the problem in their lap. This includes out databases, just for the record.
On the other hand, I've worked also with enough systems which need like 15 steps to put them back together after a restart."Oh, you need to stop this first, then restart that, then run this jenkins job, then start this first thing, then run this other jenkins job - it'll fail, but that's normal, ..." In such a place, some kind of generic on-call would be impossible. until you get these systems under control.
Like, if the error is something technical - something is upset and refuses to talk with the database now, that's something you can identify and handle with minimal instructions. I'm working on a side-project at work to automatically generate dashboards for these technical foundations of services in a central discoverable place. Or, is the error something functional? Like, payments from Suaheli aren't being taxed right. There, a generic on-call has no chance. But why the hell would this not be caught in tests, or emerge at night out of office ours, and outside of a planned maintenance if you need that?
And then you're getting to the quality and if your application is a humble workhorse or a feisty goat. With a lot of our systems, we've pushed it to the point so in doubt you can just restart an instance and it most likely will end up better. This makes on-call somewhat chill - in most cases, you just keep kicking the system one or two times overnight until more knowledgeable people come online in the morning and dump the problem in their lap. This includes out databases, just for the record.
On the other hand, I've worked also with enough systems which need like 15 steps to put them back together after a restart."Oh, you need to stop this first, then restart that, then run this jenkins job, then start this first thing, then run this other jenkins job - it'll fail, but that's normal, ..." In such a place, some kind of generic on-call would be impossible. until you get these systems under control.
I am 1.5years in to a new role, on call for 1.4 years so far...
I still get calls about some service or application I have never heard of.
Technical debt is real and I lose a little bit of my life every on-call week.
I still get calls about some service or application I have never heard of.
Technical debt is real and I lose a little bit of my life every on-call week.
Exactly my experience. I remember when I joined my company, after a couple of months I thought my job was pretty demanding but oncall added another level of stress. That being said, it's certainly less stressful now. At least, I can solve a bunch of issues myself, and I'm not lost in the internal tools anymore. And it's pretty rare to be bother in the night. And it's not like we're doctors or something. Worst case, somebody won't access a service for a few hours...
There was a company I interviewed at which had a particular way of hiring. Although I did not end up working for them, it was something I tried doing when I was involved in hiring for people with ops experience.
And that is, being able to troubleshoot complex systems when you have no idea what is going on, under time pressure.
Every site is a unique combination of side effects that no one on the internet — or even within the company — have any idea on. Yet there is generally a way to start approaching how to figure out what is going on. Though experience with the components of the system helps a lot.
And that is, being able to troubleshoot complex systems when you have no idea what is going on, under time pressure.
Every site is a unique combination of side effects that no one on the internet — or even within the company — have any idea on. Yet there is generally a way to start approaching how to figure out what is going on. Though experience with the components of the system helps a lot.
I call it Zero-knowledge Troubleshooting on my resume. It’s amusing how often, when I still worked in an office, that I would overhear part of a discussion, and some detail would tickle my spider sense, without even being sure what the problem being discussed was, and I’d ask a question or two, and it would help unblock the process.
That sort of thing is literally the only thing I miss about the office.
That sort of thing is literally the only thing I miss about the office.
Neat term. I might steal that for my resume.
Intuition does play a good part of it for me, though I have worked with good troubleshooters who are more methodical and rely less on intuition for these situations. But then again, having these different takes can help the team as a whole to figure things out and fix things.
Intuition does play a good part of it for me, though I have worked with good troubleshooters who are more methodical and rely less on intuition for these situations. But then again, having these different takes can help the team as a whole to figure things out and fix things.
I’m brushing up my resume today, and I would say there is a good chance this phrase will show up on mine too.
Recent blog post along a similar vein.
https://rachelbythebay.com/w/2023/05/30/eng/
https://rachelbythebay.com/w/2023/05/30/eng/
Reminds me about one step within the interview process for Spotify (which I liked):
Live root-cause analysis of a failure which previously occured in Production - you explain your thought process (check connectivity between System A and B via ping, DNS name resolution, look at open ports, etc...) and they tell u the result like "ping returned 'Name or service not known'".
SWE role in infra.
Live root-cause analysis of a failure which previously occured in Production - you explain your thought process (check connectivity between System A and B via ping, DNS name resolution, look at open ports, etc...) and they tell u the result like "ping returned 'Name or service not known'".
SWE role in infra.
DNS, it’s always DNS.
Except in distributed data processing, where it's always skew.
Unwise but common. It’s a good sign of an org to stay away from IMO, especially if it means that the people creating services aren’t the people oncall for those services.
> are people really going on-call for the first time without any training or knowledge of the services they'll be supporting?
Yes. Very common
> Even with perfect run books that seems unwise.
Yes
Yes. Very common
> Even with perfect run books that seems unwise.
Yes
Certainly very common in my company. After 1 year in my team, I'm still discovering services we're supporting that I've never heard of!
But being oncall doesn't mean that you're supposed to fix every single issue by yourself. You're allowed to ask for help! Monitoring the alerts, taking care of some simple issues (broken CI), trying to fix what you can, and dispatching when you can't. This is usually doable by a new team member.
But being oncall doesn't mean that you're supposed to fix every single issue by yourself. You're allowed to ask for help! Monitoring the alerts, taking care of some simple issues (broken CI), trying to fix what you can, and dispatching when you can't. This is usually doable by a new team member.
I have taken on-call rotations with no training on the systems I'd be supporting.
Even worse, I built and ran the human process that systematically put other devs into that exact same situation. We got good results from the process (as compared to not having the process), but it's obviously not as good as it could possibly be.
Even worse, I built and ran the human process that systematically put other devs into that exact same situation. We got good results from the process (as compared to not having the process), but it's obviously not as good as it could possibly be.
Not uncommon, We had a bank where they tried on call rota per group instead of team. It was horrible, A ping in at 2 AM meant I had to go through code to triage the issue.
No amount of training is enough as features and code gets deployed every day.
It was horrible and I left.
No amount of training is enough as features and code gets deployed every day.
It was horrible and I left.
What did Morgan (in the dream) actually accomplish that a small shell script couldn’t?
Literally each step was “if error then do automated action”
Alas I have emails which go to a circuit provider when their circuit goes down as they can’t seem to monitor it. It does automated triage, for example checking my router is up and powered, checking the link light on their adva etc.
It tells them there’s no power outage. They inevitably ask the first question. “Is there a power problem”
A run book, or any instructions, is only as good as the person following them.
Literally each step was “if error then do automated action”
Alas I have emails which go to a circuit provider when their circuit goes down as they can’t seem to monitor it. It does automated triage, for example checking my router is up and powered, checking the link light on their adva etc.
It tells them there’s no power outage. They inevitably ask the first question. “Is there a power problem”
A run book, or any instructions, is only as good as the person following them.
I thought it was going to turn out to be a sophisticated phishing attack.
I was baffled by the lack of actual troubleshooting. Ok, so graphs show increased failures, so what? What is happening? Spike of inbound traffic? Hardware failure? Increased connection pauses because of DNS? Database migration mishap? Without at least understanding the underlying reason on the high level restarts and version rollbacks may not help or may even aggravate the situation. Am I missing anything here?
This is the most normal part, at 3AM you do whatever is necessary to get stuff working only until 9AM. Then at 9AM you have a real team to work on it.
They is responsible for this! And We is gonna make sure They pay for the time We lost working on It, to fix They's problem.
We is out! No diggity.
We is out! No diggity.
As expected, this is an Ops story, not DevOps
haha, it makes sense this is on a site with “consulting” in the name—its unrealistic and pointless. the M. Night Shyamalan twist is that the whole thing is the “horror” portion.
in the story our protagonist has no knowledge of the service she is on call for and displays no ability to troubleshoot, reason about, or understand the issue she’s responding to.
instead, she clicks a series of buttons in a runbook to resolve the most basic and most happy-path production issue you’ll ever see.
aaaaand this is about devops how?
in the story our protagonist has no knowledge of the service she is on call for and displays no ability to troubleshoot, reason about, or understand the issue she’s responding to.
instead, she clicks a series of buttons in a runbook to resolve the most basic and most happy-path production issue you’ll ever see.
aaaaand this is about devops how?
Anyone here have buttons in runbooks? We normally have links to Jenkins jobs in git runbooks, what are you using?
The one place I've set up button triggered runbooks was gitlab ci under scheduled jobs. Made things a lot easier for less technical people.
And here I was waiting for it to turn out to be a phishing attempt due to clicking the link in the alert.
As a non-native English speaker, I still find it difficult to read texts where a singular person is referred to by a plural pronoun. Every "they" becomes ambiguous and requires more mental cycles to unpack. Is it about Morgan, or some unrelated group of people, or Morgan plus the group?
I understand the cultural reasons, but I regret the loss of linguistic clarity.
I understand the cultural reasons, but I regret the loss of linguistic clarity.
The cultural reasons are not relevant, this is simply normal English being used naturally as it has been for centuries.
For example, consider the following. "I saw someone get hit by a bus this morning." "Oh, were they okay afterwards?".
"They" is clearly a singular pronoun here.
For example, consider the following. "I saw someone get hit by a bus this morning." "Oh, were they okay afterwards?".
"They" is clearly a singular pronoun here.
loss? It's been English usage since Chaucer. The "rule" that it shouldn't be singular came from the same people that changed English spellings because they wanted to show off their knowledge of Latin and Greek. All those difficult words that are irregular and don't make sense? Those people.
It's not the pronoun confusion about whether it's a devops team or an individual doing the various actions, it's the writing itself that's jarring, rambling and slightly disjointed.
I'd blame modern teaching material, because "they" is both singular and plural pronoun - just somewhat recently avoided in style guides.
The Wikipedia article on "Singular they" provides some background (the matter is more complex than the short remarks of others here suggest -- and quite interesting): https://en.wikipedia.org/wiki/Singular_they
non native speaker too, I was taught that "they" is the more formal/neutral pronoun to use, or preferred in passive case, isn't that the case here?
No, ‘they’ is never correctly used for a known single human being. In some styles of English it is acceptable for an unknown human being, for example ‘if someone enters here, ask them what they are doing.’
> As a non-native English speaker,
> I understand the cultural reasons, but I regret the loss of linguistic clarity.
Have you thought about the fact that you're using ignorance of English to complain about a very specific anglophone piece of politics?
> I understand the cultural reasons, but I regret the loss of linguistic clarity.
Have you thought about the fact that you're using ignorance of English to complain about a very specific anglophone piece of politics?
And rightly so, because noone writes prose like that. It sounds unnatural.
Looks more like a meme-hater's horror story.
It's also silly because if the problems are known this well the runbook will only be this good for a moment while the devs are working on fixing the bugs and the runbook will need to be updated constantly after each release.
This requires more testing than most are willing to do and for dev ops to still always be on call to find the new bugs first and document them this way while waiting for devs to fix them and for management to always reserve time to keep this updated.
This means there's a long enough release cycle for this to ever happen which is also unlikely.