Took the time to read the 84 page report that was published last month, and write up a summary. Pretty interesting bug, with a pretty significant impact.
It sounds like marketing, I know, but this was written by engineers and practitioners, and is a pretty comprehensive overview how modern tech orgs are dealing with incidents these days.
> Humans aren't great at incident response, and we all hate waking up at 2am to resolve an issue.
Agree that most people hate being woken at 2am, but disagree that humans aren't great at incident response. Speaking generally, I think we're about as good as it gets when it comes to adaptability and the kind reasoning that's necessary to investigate complex issues.
That said, I also think AI can play a massive role aiding humans, especially in undifferentiated tasks like checking deployments, code changes, past incidents, and when it comes to spotting patterns.
IMO the sweet spot is going to come from highly ergonomic AI products that enable collaborative incident response, rather AI incident management or any other marketing BS.
As the person who originally wrote the Monzo Response project (https://github.com/monzo/response), I expect you'll find some traction in smaller orgs, but when folks start doing things at scale they'll hit an inflection point where running their own incident software/not having folks to log feature requests with will force them to pick something more off-the-shelf.
Basically, nobody runs their on-call system on open-source because it's mission critical. At a certain point, IM platforms hit the same level of criticality.
'Fun' is a little strong, but I really wish more companies adopted this kind of workflow for TF. Being forced into a world of full-TF or fully-manual sucks, and the lack of signposting when you're in a blended world of manual and managed resources really doesn't help.
Tweet author here, and founder of a startup. I know this is isn't revolutionary, but I've iterated towards this state and now find it a critical part of my daily process.
> On-call payment is not expected to cover any time you spend working outside of hours. If you're paged and end up working in your evening, you should take time off in lieu. We trust you to manage this time yourself.
I'll make sure this is added this to the guide. Thanks for the nudge!
When you say there's 2-3 devs who can actually fix issues, is that because there are only 2-3 devs, or do you have more people available but they don't know how to fix?
Assuming the latter, my suggestion would be:
1) Setup a weekly on-call rotation, where one of the 3 people who can fix issues is dedicated on-call (this means the other 2 can properly switch off)
2) Find some other folks who might not be able to support immediately, but are happy to learn.
3) Configure a shadow rotation so there's always a primary responder who know what they're doing and a shadow responder who can learn on the job.
4) Make it clear the shadow person has the same expectations in terms of turning up, but their job is to ask questions, write things down, etc.
I did this in a former job (and wrote about it here https://monzo.com/blog/2018/09/20/on-call) and it worked pretty well. After a few months we had a healthy rotation of 8 on-callers, and a waitlist of shadow folks who wanted to join too.
This advice obviously came from a point of near-zero context, so if you'd like to chat more I'd be happy to!
It's hard to assess on-call comp without the context of pager load, stage of company and a myriad of other factors. We've done some research from the market though, and have some interesting results we'll be sharing soon! (see https://twitter.com/incident_io/status/1526191169054597120)
I take your point though, and if being on-call is having any meaningful impact on the time you spend working outside of hours, I agree $350 is low. For us, we're an early-stage startup with low numbers of alerts (long may it stay that way!) and the impact is low.
Not to try and defend it too much, but it's worth pointing out that the $350 is there to cover the inconvenience of being 30 mins from a laptop, rather than compensating for the time. We also give folks time off in lieu for any time they spend working outside of hours.
I don't disagree with your points, and having worked at both massive IT organisations, tiny startups and scale-ups in-between, it's clear that different organisations need different levels of rigour and process!
I see the guide less as a set of dogmatic rules to follow, and more of:
a) a set of sensible defaults for small to medium size orgs who are starting with little-to-no process
and
b) a source of inspiration for larger folks who maybe want to bring their processes out of the ITIL ages and into a world that's a little more applicable to the way folks work today.
As you say, the extremes of the ITIL <---> Agile spectrum is likely to be a mess, and where you should target on that spectrum is highly dependent on your starting point, your culture, and your appetite for change :)
Hey folks! One of the founders of incident.io here, and the one who's face is on the front cover :) I've written this content so many times at companies I've worked at in the past (though not to this quality!) so it's really nice to be able to share this and avoid others having to start from scratch.
Firstly, thanks for the kind words Quentin – appreciate it, and congrats on your progress so far.
incident.io is pitched slightly differently to Rootly, insomuch as we're not building an engineering product, but instead something that's designed to work for entire organisations. I saw first hand what this looks like at Monzo – a bank here in the UK – where incidents weren't just declared when Cassandra fell over (ahem, https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on...), but were also declared for things like excessive customer support queries and not enough people to serve them, regulatory issues, or a customer threatening staff in reception. All of these things require teams to form quickly, communicate well, and follow a process. We're building for this.
In terms of market and customer segments, we're working with a wide range of companies with up to 6k employees. That said, we're a perfect fit right now for folks in the 200-1500 people range.
Hey folks, author here. I work in the field of incident management and get to work with a lot of startups who are looking to put good practices in place early on.
There's been a common theme across them of over complicating things, relative to their size and general needs. This post is basically a distillation of things I often find myself recommending.