Full transparency: I head the customer success team for Transposit, a tool for automating incident response. Before this, I ran Production Operations for Hulu and had the opportunity to build an incident response from scratch.
Communication Tools
Internally: We primarily used Slack for real-time communication, enabling quick collaboration and integrations.
Externally: Our main external communication method was email for engaging with stakeholders.
Communication Framework
Here is an excellent framework on how to communicate effectively during incidents:
Conditions: The current scenario—what's happening right now?
Actions: The dance of deeds—what's being done?
Needs: Resources, time, or assistance required. What do you need?
Example: "Need the network team to join the bridge and assist with load balancer logs."
Example: "Require 10 minutes to dive into logs for insights."
This framework helps keep communication simple and removes emotional responses that sometimes enter these conversations.
Guidance for Incident Severity Response
Here's some guidance for additional resources depending on the severity of an ongoing incident.
Dispatch
The group must mobilize and respond to the ongoing disruption. Generally, this is the on-call person responding to help mitigate. We dispatch them to take action; their help is mandatory in solving the issue.
Building Dispatch Groups: On-call schedules generally indicate who the people are. If you don't have that available, start building a manual roster in a spreadsheet.
Inform
Stakeholders outside of engineering may need to be informed of ongoing disruptions. This group generally includes customer support, press relations, senior leadership, critical business partners, legal, etc. These teams likely have their own process for dealing with an incident and require less technical information.
Notify
The notification groups are unnecessary to resolve an incident or interact with those fixing it. Generally, this involves broad communication to an email distribution or public chat channel, informing everyone of the ongoing activity. This communication group may offer additional resources due to being notified.
You can automate a significant portion of this workflow by leveraging Slack, external status pages, emails, or even SMS directly. Let me know if I can help
Exciting topic! Many mixed review on AI in general. But it can do some really awesome things in the Incident Management space.
A great case of AI + IMS is Transposit (I work for them)! Of course AI can get overwhelming and maybe a little noisy at times. We differ by adding a "human-in-the-loop element" which allows you/ your team to have more control over what's automated vs what needs more of a "human touch".
The gist: Transposit is designed to help with the entire on-call lifecycle. We have AI and automation that helps your on-call engineers triage alerts by bringing context to them. We’ve built an AI-powered, Virtual Incident Commander who identifies key moments for your team and reminds them to communicate status updates (and even writes drafts of them for you!).
We are also working on new features that use AI to bring important information like 3rd party service outages to your on-call engineers at the right time as well as dynamically identifying the appropriate engineers to bring into an incident to augment your typical escalation process.
Couldn't agree more than costs can add up quickly!
Have you looked into an IMS (incident management solution) like Transposit? (I work for them) It's an "end to end" platform meaning it includes everything from when an incident is created, through investigation and remediation, to closing the ticket and creating a post incident review doc.
We are listening to teams who are saying it's getting expensive and have recently released a dynamic on-call structure so it's in the same platform and only paying for users that are actually being paged within that month while having your whole team available if needed.
Feel free to check us out if you're looking to reduce overall cost- Transposit.com
Communication Tools Internally: We primarily used Slack for real-time communication, enabling quick collaboration and integrations.
Externally: Our main external communication method was email for engaging with stakeholders.
Communication Framework Here is an excellent framework on how to communicate effectively during incidents:
Conditions: The current scenario—what's happening right now?
Actions: The dance of deeds—what's being done?
Needs: Resources, time, or assistance required. What do you need?
Example: "Need the network team to join the bridge and assist with load balancer logs."
Example: "Require 10 minutes to dive into logs for insights."
This framework helps keep communication simple and removes emotional responses that sometimes enter these conversations.
Guidance for Incident Severity Response Here's some guidance for additional resources depending on the severity of an ongoing incident.
Dispatch The group must mobilize and respond to the ongoing disruption. Generally, this is the on-call person responding to help mitigate. We dispatch them to take action; their help is mandatory in solving the issue.
Building Dispatch Groups: On-call schedules generally indicate who the people are. If you don't have that available, start building a manual roster in a spreadsheet.
Inform Stakeholders outside of engineering may need to be informed of ongoing disruptions. This group generally includes customer support, press relations, senior leadership, critical business partners, legal, etc. These teams likely have their own process for dealing with an incident and require less technical information.
Notify The notification groups are unnecessary to resolve an incident or interact with those fixing it. Generally, this involves broad communication to an email distribution or public chat channel, informing everyone of the ongoing activity. This communication group may offer additional resources due to being notified.
You can automate a significant portion of this workflow by leveraging Slack, external status pages, emails, or even SMS directly. Let me know if I can help