About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Starting Incident Management from Scratch -Introduction-

Starting Incident Management from Scratch -Introduction-

This article is aboutCyberAgent Group SRE Advent Calendar 2024This is the article for day 12.
 
Tanaka from the Service Reliability Group (SRG) of the Media Management Division (@tako_sonomono)is.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
 
This article outlines the purpose of implementing incident management and the three elements that should be established initially.
We will provide separate information on tools for smooth incident management, specific workflows, and problem-solving methods.
 
 

Introduction

In our daily operation of the system, we encounter many alerts.
Problems that could be addressed when the system or organization was small become numerous challenges as the system and organization scale up, as listed below.
  1. The system is constantly issuing alerts because the person in charge is absent.
  1. A specific developer is handling the alert response, leading to a situation where the process is becoming overly reliant on that individual.
  1. The recurrence rate of past incidents is high.
  1. There is no established response flow for incidents that have a significant impact on users, resulting in a long recovery time.
  1. There are no established criteria for determining whether or not user announcements are necessary; the supervisor's decision must be made on a case-by-case basis.
  1. A monitoring setting was not configured correctly, and the problem was discovered through a user inquiry.
These issues ultimately delay disaster recovery, which in turn undermines user trust in the service.
 
To address these challenges, we implement incident management.
In the following sections, I will discuss what needs to be done to solve these problems, including examples from Ameba.
 

The purpose of incident management

SRG is responsible for incident management across all of our media services, and we are also promoting activities to address the above issues at Ameba. Here, the main objectives of our incident management are the following three points:
 
  1. Shortening of MTTR (Mean Time to Recovery)
  1. Reducing the number of incidents
  1. Suppression of the decline in user satisfaction

1. Shortening of MTTR (Mean Time To Repair)

The time it took from the time the system failed or a problem occurred until it recovered.averageThis refers to...
Operational cycle in incident responsecontinuousThe goal is to shorten the time from the occurrence of a failure to its recovery by making these improvements.
 

2. Reducing the number of incidents

The goal is to reduce the occurrence of incidents by eliminating potential causes of incidents in advance, such as through postmortem procedures.
 

3. Suppression of the decline in user satisfaction

If a service is unavailable for several hours without any announcement from the service provider, users will lose trust in the service. This initiative aims to mitigate this decline in service confidence by optimizing communication with users during incidents.
 
The following sections will explain specifically what needs to be done to achieve these objectives.

Introduction

  1. Establishing an Incident Owner
  1. Establishment of an Incident Commander
  1. Triage decision
 

1. Establishment of Incident Ownership

The incident owner stated, "Introducing an incident management culture into the business andcontinuousThis is the role of "taking responsibility for improvements."
Because measuring the quantitative effectiveness of incident management after implementation is extremely difficult, it's not something that can be easily implemented in all organizations. In particular, for small organizations where the problem hasn't escalated, it's less likely to be a target for resource investment, so incident owners need to consider the state of their business before implementing it.
 
Furthermore, even after implementation is decided, continuous improvement is crucial in incident management, as evidenced by the concept of MTTR (Mean Time to Recovery). A person is needed to lead the team by setting medium- to long-term goals, such as collecting data by measuring MTTA if necessary, and formulating strategies for fostering a positive work culture together with team members.

2. Establishment of an Incident Commander

An Incident Commander (hereinafter referred to as IC) is a role that "is responsible for resolving incidents."
With a few exceptions, ICs generally do not perform recovery work themselves, but instead focus on communicating with relevant parties and giving work instructions to team members.
The role of the Incident Consultant (IC) becomes especially important in larger incidents, so it's a good idea to rotate through minor incidents to gain experience as an IC on a regular basis.
 
  • We will be responsible until the incident is resolved.
  • Decision-making regarding system recovery
  • Communication with relevant parties
  • Establishment of recovery response teams and communication networks (such as Slack channels).
  • Instructions to team members
  • Creating a Postmortem
 
The difference between this role and that of an incident owner is that it is a position established only when an incident occurs, and is therefore a one-off role.
We are not responsible for continuous improvement, such as ensuring that the incident management cycle is functioning correctly.
 
 

3. Triage decisions

Triage (severity) is determined to "decide the priority of response" to the incidents that occur.
Setting up triage makes it easier to determine the priority of responding to multiple incidents, to understand the severity of incidents from a third-party perspective (such as a business manager), and to develop and establish incident response flows and systems.
 
At Ameba, triage is determined using the following SEV1 - SEV5 system, and the response and system differ depending on the level.
For example, in incidents of SEV3 or higher, reporting the incident to users via the staff blog is mandatory, and the IC (Information Center) handles the response, including user communication. (Note: Ameba does not have a communication commander in the SRE (Service Reliability Engineering) sense. It is not considered necessary for services that do not require complex user communication.)
 
This allows engineers to focus solely on recovery work in the event of a high-impact incident.
Conversely, for incidents below SEV4, the workflow is not complex, so engineers take responsibility for recovery as ICs (Incident Computing Systems).
What to be careful about isYou can rely on IC to determine SEV.That's the point.
Delays in SEV (Situation Emission Value) determination will also delay fault recovery itself. It is important to create an environment where SEV determination can be easily requested from the IC (Integrated Circuit) when necessary.
 
 

In conclusion

This time, we presented three essential items for implementing incident management.
Next time, we will introduce specific workflows and problem-solving methods.
 
For reference, I've attached the slides I used in a previous SRE-related event!
 
SRG is looking for new team members. If you are interested, please contact us here.