About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Multi-region load testing using k6

Multi-region load testing using k6

This article is aboutCyberAgent Group SRE Advent Calendar 2024This is the article for day 15.
 
This is Ohara (@No_oLimits) from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article describes a multi-region load test conducted using k6 (k6-operator) for a service targeting overseas markets.
 

Introduction

k6 is a load testing tool developed by Grafana.
The scenario will be written in JavaScript/TypeScript.
In the service I was in charge of, I only used it for API testing, but browser testing is also possible.
In addition to HTTP, it also supports WebSocket and gRPC protocols.
While it offers sufficient functionality in its default state, numerous extensions are also available to handle more complex load scenarios.
 

Load Testing Environment Architecture

overview

We will deploy k6 in three regions and load the target systems in the corresponding regions.
To run the k6 Operator, we will execute it on GKE Autopilot.
Details of the target system to be subjected to the load will be omitted.
 

Details - Scenario Execution

Installing k6-operator allows you to use a resource called TestRun.
runner
 
The load scenario will be uploaded to GCS in advance.
When a k6 Pod starts, the initContainer retrieves the scenario from GCS, places it in emptyDir, and references it from the k6 container.
Pods are granted read access to GCS by integrating with GSA/KSA Workload Identity.
 

Details - Monitoring

k6 displays stats upon test completion, but if you are running distributed tests, you will need to aggregate the metrics.
This time, we will build a monitoring environment using Prometheus + Grafana.
The reason is that k6 supports sending metrics to Prometheus remote write by default.
Sending metrics to other backends requires an extension.
  • Deploy Prometheus to each region.
  • k6 sends metrics to Prometheus within the region.
  • The data source for Grafana is thanos-query.
  • Requests to the Prometheus API are made via the Thanos sidecar using Thanos Query.
  • Enables inter-cluster access to internal load balancers (*1
 
The key is using Thanos to aggregate cross-region Prometheus metrics.
Thanos is a component for scaling Prometheus, and it also has various other functions.
This allows you to aggregate and monitor metrics across multiple regions.
 
*1) Make the internal load balancer accessible across clusters.
Thanos Query is configured to make requests to the LB IP address.
 

Troubleshooting Tips and Tricks

k6 won't start.

This is especially common the first time you run it.
Basically, nothing is launched, so we don't have access to any logs.
But actually, the k6-operator manager is the one that's outputting the logs.
If nothing starts up, check the manager logs.
And the most common reason it doesn't start is that the manifest is specified incorrectly.
 

K6 OOM died peacefully.

This time, we were conducting tests with 500,000 virtual users.
We were trying to define multiple attributes for each user and control their behavior during testing based on those attributes, but shortly after the test started, the Pods began to stop one after another due to Out of Memory (OOM).
The attribute definitions were stored in a file, and the process involved reading the file at startup before running the tests. However, it appears that an OutOfMemory (OOM) occurred because all VU threads within the k6 process were reading files for 500,000 users.
SharedArray
ShareArray is a memory space that can be shared between VUs. By setting attributes on this array when reading files, memory usage can be significantly reduced, thus avoiding OutOfMemory errors.
 

Prometheus load surge

There are probably two times when the load will increase significantly.
This is during k6 execution and when queries are issued from Grafana.
If you're experiencing excessive metric transmission to Prometheus while running k6, reviewing your CPU and memory resources is a common solution.
Also, in this state, a large number of transmission failure messages should be output to the k6 log, so please check it.
It can be difficult to pinpoint the cause of Prometheus becoming unresponsive when displaying graphs in Grafana, but in my experience, the most common reason was insufficient grouping of metrics sent by k6.
As explained in the article below, in k6, a tag is assigned by default to each request (this request is a request to the target system being subjected to load).
Then, that tag information is sent to Prometheus along with the metrics data.
 
Prometheus likely groups or sorts queries from Grafana based on tag information, so querying a large number of ungrouped metrics is expected to lead to a significant increase in load.
In this particular case, the issue was resolved by appropriately grouping the tag's name value for each request.
The `name` field is set to the request URL by default, but it is suspected that Prometheus interprets the URL, which contains the user ID, as a unique value for each user, preventing it from responding to queries from Grafana that group by `name`.
 

Things I liked about using the k6

k6-operator has a wide range of features.
At the time, I was struggling with how to deploy scenarios to k6 pods when a new version of k6-operator was released, and the initContainer spec was implemented in TestRun (which was called K6 resource at the time).
Additionally, previously it was necessary to add extensions and build the project to support Prometheus Remote-Write, but this is now also provided as a core feature.
Since it's frequently updated, more features will likely be added in the future.
 

Areas that need improvement

When conducting this multi-region load test, I wanted to synchronize the test timing across each region, but it seems there's no function to do so. So, I deployed at the right time and then manually synchronized them.
I think it would be perfect if we could set the test date and time as an option at runtime.
 

In conclusion

This article introduced multi-region load testing using k6.
I hope this article will be helpful to you.
SRG is looking for new team members. If you are interested, please contact us here.