Regarding the introduction of Chaos Engineering
This is Kataoka from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This time, I would like to write about what needs to be considered and what we can expect when implementing chaos engineering.
I plan to post follow-up articles detailing any insights or problems I encountered after actually implementing the system.
What is Chaos Engineering?
The development of Netflix's Chaos Monkey led to the spread of the concept of "Design for Failure," which aims to create system configurations that anticipate failures.
The rapid proliferation of public clouds and the adoption of microservices architectures have led to increased system complexity, faster changes, and a greater likelihood of unpredictable failures.
Chaos engineering allows us to intentionally inject failures into a system (service) to identify its weaknesses against unknown failures and ultimately ensure its resilience.
While some companies have implemented it in their productions, many others have only implemented it for staging, or they want to implement it but face a lot of opposition from others. It seems that there are still many companies where implementation is not progressing smoothly due to various reasons.
The idea of "injecting a bug" often evokes strong anxiety and opposition, and some people imagine it as simply causing bugs indiscriminately. However, since "the bug itself is controllable," I believe that these anxieties and opposition can be overcome with thorough testing. If the sole objective is simply to implement the bug, it could lead to unnecessary bugs in the production environment, potentially causing the original meaning and purpose to be lost.
I think it will be a fairly long road before it's fully functional, but I believe that once it's working, it will clarify the issues we need to solve and lead to improved service reliability.
Regarding implementation
This article doesn't actually cover the process of implementing it in a product, but it's about our company's open-source software product that I'm considering implementing it in.BucketeerI had a meeting with the product owner of the (feature flag management and A/B testing platform) to discuss what we hope to achieve through chaos engineering in order to improve the reliability of Bucketeer, so I've summarized the points below.。
- We want to provide concrete justifications for SLI/SLO and ensure that they can always be guaranteed.
- We want to enhance resilience and demonstrate stability to our users.
Furthermore, chaos engineering offers numerous advantages. For example, it can be used for incident response training, MTTR reduction, vulnerability detection, SPOF verification, and as a good approach to problems that are often perceived as intuitively solved.
First, as an important element to consider when moving forward,Principles of Chaos EngineeringI think this will be helpful.
The following five principles are defined there:
- Minimize the blast radius.
- Formulate a hypothesis about steady-state behavior.
- Real-world events are diverse.
- Run Experiments in Production
- Automate experiments to run continuously.
A book published by C&R InstituteIntroduction to Chaos EngineeringThe book breaks these down into smaller steps. I've created a diagram illustrating the practical steps, which I'll attach here.
First and foremost, I felt it was crucial to coordinate with the product team regarding the definition of the steady state, the design of hypotheses, and the definition of variables.
Furthermore, I believe the following preparations are particularly important when proceeding with chaos engineering.
- There must be no difference between the production environment and the staging (testing) environment.
- Logging, metrics, tracing
- Load testing environment (an environment where requests similar to those in production can be sent)
- SLI/SLO Implementation
Even if you believe you are following the principles perfectly, neglecting these aspects will render all the effort invested in the implementation pointless.
Conversely, by promoting the introduction of chaos engineering, it becomes possible to identify and improve what is lacking in SRE activities.
And because we wanted to implement it for a service running on GKE, we chose Chaos Mesh.
I've also noted other chaos engineering tools that caught my attention.
Managed Service
- Gremlin
- AWS Fault Injection Simulator
- Azure Chaos Studio
Hosted Service
- Chaos Mesh
- Chaos Toolkit
- Litmus Chaos
- PowerfulSeal（Kraken）
This isn't limited to chaos engineering, but since tools don't solve everything, I think it's important to broaden your perspective by reviewing your usual rules of thumb and habits through the activities and operation of chaos engineering, and for the entire team to work together to address the system.
Trying out Chaos Mesh
Chaos MeshLet's try out a simple bug locally using this.
Chaos Mesh allows you to inject various types of failures into Kubernetes and Hosts.
In particular, the things I would like to do are
- Simulate GCP Faults (Simulate failure scenarios for GCP instances)
- Simulate Pod Faults (Simulate failure scenarios for Pods or containers)
- Simulate Stress Scenarios (apply a simple load to the CPU and memory)
- Simulate HTTP Faults (Simulates failure scenarios in HTTP requests and responses)
That's correct.
Let's try killing a Pod in a Pod Fault environment where three Nginx Pods are running.
We'll set the target namespace to "default," specify the target label "app:nginx," and inject a failure that randomly kills only one Pod.
You'll see that the pod was killed immediately.
You can determine if the Pod crash was truly caused by a Chaos Mesh by looking at the events.
Depending on the type of failure, you can specify the duration, perform continuous execution, or schedule execution. Therefore, instead of just performing immediate, one-off executions, you can continuously inject failures periodically (over a long period) and combine this with load testing to better understand the system's behavior.
This time I've given a very simple example, but by paying attention to how the system behaves in this situation, what happens to requests when there is constant access, and what happens to latency, you can deepen your understanding of the system.
Therefore, the preliminary preparations and hypothesis design mentioned earlier are extremely important.
In conclusion
The following articles and books were extremely helpful when I started learning about chaos engineering. I highly recommend them to anyone interested. I will continue to learn and think so that I don't leave things unfinished. And from now on, I would like to write articles about what I have tested and the problems I have encountered during operation as part of the practical application section.
* If you are interested in implementing chaos engineering, or if you are already using it, we would love to exchange information.
reference
SRG is looking for new team members.
If you are interested, please contact us here.