About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Providing canary release using Argo Rollouts

Providing canary release using Argo Rollouts

This is Taniguchi from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article describes the process of using Arog Rollouts to implement canary release functionality, eliminating the risk of impacting all users with traditional bulk deployments and providing safe and rapid feature delivery through phased traffic control and automated monitoring.
 

Why is canary release necessary?

Improving system productivity and reliability is a critical challenge in development. In particular, releasing new features or fixes always carries the risk of unexpected bugs affecting the entire service.
One method to minimize this risk and ensure a safe and rapid release is called "canary release."
A canary release is a deployment method used when releasing a new version of an application. It involves initially releasing it to a limited number of users, verifying that there are no issues, and then gradually rolling it out to all users.
By implementing a canary release system, you can expect the following benefits:

Reducing release risks

Even if a problem occurs, the scope of impact can be limited, and a quick rollback is possible.

Quality verification in a production environment

You can test the performance and stability of new features in a production environment with actual user access.

Improving the user experience

By preventing major outages and providing stable service, we enhance user trust and satisfaction.

Ensuring the psychological safety of developers

Even if there are bugs or other issues in the deliverables, the damage is expected to be kept to a minimum, which reduces the psychological burden on developers when releasing the product.

Research on methods for implementing canary release

Even though we refer to it simply as "canary release," there are many tools and methods for achieving it.
We investigated the optimal approach, taking into account compatibility with existing platform environments and implementation costs.

Comparison of various tools

First, we considered several tools that could potentially enable canary releases.
  • Linkerd
      • It offers service mesh functionality similar to Istio, but we decided against it because we already have Istio in place.
  • Consul Connect
      • This is also a type of service mesh. It's a powerful option in multi-cloud environments, but it didn't meet our current requirements.
  • Spinnaker
      • While it's a highly functional CI/CD tool, we determined that its implementation would be too extensive for the sole purpose of canary releases.
  • Keptn
      • While advanced operational automation, such as measuring SLOs (Service Level Objectives), is possible, we determined that it would be too large-scale to implement this canary release.
While these tools are very powerful, each requires a considerable learning curve and operational burden to implement.
Therefore, we have adopted a strategy of making the most of the technology stack already implemented in the Platform.

Argo Rollouts Selection

In our environment, we already use Istio as a service mesh and Argo CD as our CD tool.
DestinationRule
However, Istio alone makes it difficult to implement "progressive delivery," which involves monitoring metrics such as error rates and automatically proceeding with rollouts or rollbacks based on the results.
That's when I came across "Argo Rollouts".
Argo Rollouts is a tool for enabling Kubernetes-native progressive delivery and is installed as a separate component from Argo CD.
We determined that implementing Argo Rollouts would allow us to integrate with our existing Istio system and achieve secure, automated canary releases without compromising the developer experience.

Argo Rollouts' Practice of Canary Release

From here, I will explain the steps I took to actually implement Argo Rollouts and try out canary release.

Installing Argo Rollouts

First, install the Argo Rollouts controller into the cluster.
Basically, you can install it by following the official documentation and using the following command.

Basic Canary Release

Rollout
strategy
AnalysisTemplate
Example of a Rollout resource
Example of AnalysisTemplate (when integrating with Datadog)
This setting ensures that when a new version is deployed, 10% of the traffic is initially directed to the new version.
AnalysisTemplate
However, this method alone has one drawback.
Traffic control for Argo Rollouts alone is based on the number of Pod replicas.
For example, if you have three Pods and try to direct 10% of your traffic to one, a new Pod will actually launch, and approximately 33% of your traffic will be directed to that new Pod.
To implement more rigorous traffic control, integration with Istio becomes necessary.

Advanced traffic control through Istio integration

DestinationRule
Rollout resource configured for Istio integration
Integrating VirtualService and DestinationRule
weight
This enables precise, percentage-based traffic control that is independent of the number of Pods.

Argo Rollouts Operation Guide

This section introduces the basic operation methods for actually using Argo Rollouts.

Operating Argo Rollouts

kubectl
Additionally, you can launch the dashboard on your local PC and check the status using a GUI.
Promote-Full

Points to note

ReplicaSet
While this may not pose a major problem, leaving it as is will result in retaining unnecessary resources, so care should be taken regarding the timing of resource deletion during operation.

summary

This article investigates and implements methods for achieving canary releases on a platform with the aim of improving productivity and reliability.
After comparing various tools, we chose Argo Rollouts because of its high compatibility with our existing technology stack, Istio, and its ability to enable progressive delivery.
By combining Argo Rollouts with Istio, we were able to incorporate metrics-based analysis, enable precise percentage-based traffic control, and build a secure and flexible release flow.
We will continue to make improvements and aim to provide more stable services in the future.
 
If you are interested in SRG, please contact us here.