Regarding SRE confidence measurement
This article is aboutCyberAgent Group SRE Advent Calendar 2024This is the article for the first day. Since it's the first day, I've written it in a relaxed style.
IntroductionA look back at 2024SRE Confidence MeasurementCapability measurementTrend analysis and improvementIn conclusion
Introduction
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
A look back at 2024
This year, not only my main job, butSRE NEXT 2024I think this past year has been a year in which I was able to contribute to the SRE community, including serving as Co-Chair. Honestly, it was incredibly busy, but ultimately, we were able to create an event with over 1,500 participants and a 96.1% satisfaction rate, so I'm glad I did it.
Archived videos from SRE NEXT 2024All of the events, including the on-site panel sessions, are available on YouTube, so we hope you'll take a look.
In my main job, we commercialized a cross-functional SRE organization in 2020, which allowed us to expand SRE support not only to our media business but also to our SGE (Game Division) and externally. I hope to share more about what we did to commercialize it and the SRE lifecycle that I touched on at the SRE NEXT panel session in a future blog post or presentation.
In this article, I'd like to write a little about measuring SRE reliability.
SRE Confidence Measurement
When running a cross-functional SRE organization, we provide SRE support to multiple products, but realistically, it's difficult to embed improvements across all products due to organizational resource limitations. Therefore, we have developed an approach to measure SRE maturity and SRE reliability to gain an overview of the entire business unit and collect data, making it easier to determine business priorities, and we are working on SRE improvement using this approach.
Regarding SRE maturity, last yearPresentation materials from the CyberAgent Developer ConferenceI hope you will take a look.
SRE maturity and SRE reliability measurements differ in their objectives, targets, and question sets, as outlined below. SRE reliability measurement aims to understand system reliability and risk within each executive's jurisdiction and targets executives. Furthermore, it is generally implemented across all key products.
Capability measurement
In measuring SRE confidence,
- security
- Capacity planning
- availability
- Operational optimization
We measure capabilities in four categories. The questions focus on high-risk areas, and while there are some exceptions depending on the conditions, only mandatory items are included, and we are promoting their implementation in our focus products.
Partial list of the following items requires a Yes/No response, and we also collect deliverables such as documents that serve as the basis for the evaluation at the time of response.
Trend analysis and improvement
From a governance perspective, we have a separate Google Sheets for each product for measurement purposes. We use Google Apps Script to aggregate the data into an analysis Google Sheet, and then we use Looker Studio to analyze trends by business unit and item.
While much of the information cannot be made public, and therefore may be difficult to understand, we analyze the collected data in the following manner to help us make efficient improvements.
In conclusion
Through the use of SRE reliability measurement, we gained the insight that even with the same technology stack and system design, evaluations may not be the same due to differences in organizational culture and development structure. Furthermore, we achieved results such as promoting the spread of SRE culture at the company level and using it as reference material when making decisions about technology investments.
Although not covered in this article, SRG develops and deploys SRE practices that improve the reliability and resilience of business units and products, based on data obtained through SRE reliability measurements and other means. We also plan to make these SRE practices publicly available in the future.
Once again, I hope that this Advent Calendar will help many people learn about the initiatives our group is undertaking. I'm personally looking forward to it as well.
SRG is looking for new team members.
If you are interested, please contact us here.