I've placed my hopes in Cloud Service Mesh for Cloud Run.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article is aboutQualiArts Advent Calendar 2024This is day 13. I'm contributing this article because of my involvement as an Embedded SRE.
I explored and experimented with whether the issues I wanted to solve with Cloud Run could be resolved using Cloud Service Mesh for Cloud Run.
The information on this blog is current as of December 1, 2024.
What is Cloud Service Mesh for Cloud Run?Issues to be solvedChallenge A: The ever-increasing number of backend services in multiple development environments for applications using gRPC.Challenge B: Blue/Green deployment in an application using gRPC where all traffic switches instantly.The ideal solution I dreamed of. And yet, I can't come to a conclusion.Ideal A and results of the examination for problem AIdeal B for Problem BExamination of Ideal B with BookinfoConfiguration DescriptionTerraform sampleBased on our considerations, if we could specify tags for Cloud Run in NEG...Other things I dream about with Cloud Service Mesh for Cloud RunIn conclusion
What is Cloud Service Mesh for Cloud Run?
This feature is in preview at the time of writing.
Advanced traffic management features powered by Cloud Service Mesh are now available in Cloud Run.
For more details, please refer to the official documentation below.
Issues to be solved
Challenge A: The ever-increasing number of backend services in multiple development environments for applications using gRPC.
In our development environment, we deploy different versions of the application to Cloud Run for each environment, such as dev01 and dev02. We need to create a separate Backend Service for each Cloud Run instance, as shown in Figure 1.
One challenge with this configuration is that enabling Cloud Armor Enterprise Paygo/Annual*1 will result in charges based on the number of Backend Services, even if no policy is configured. This becomes a bottleneck when you want to try the same Cloud Armor Enterprise configuration in a Dev (or Stg) environment as you would in a Prod environment.
*1 Regarding the billing structure for Cloud Armor Enterprise Annual
For up to 100 protected resources, the fee is a fixed $3,000/month. For resources exceeding 100, the fee is $30/month per protected resource. (See Figure 2)
The protected resources are Backend Services and Backend Buckets, and they are counted even if Cloud Armor policies are not applied. For details, see below.documentPlease refer to the following.
Challenge B: Blue/Green deployment in an application using gRPC where all traffic switches instantly.
Traffic migrationWhen using revision switching, even if you configure it so that 100% of traffic switches to the Green revision, requests may still go to the Blue revision before the traffic switches after the change is applied. *2
This behavior is unacceptable in releases that are not backward compatible.
*2 In the case of REST APISession AffinityThis allows you to control whether users accessing a new revision are directed to an older version.
The ideal solution I dreamed of. And yet, I can't reach a conclusion.
Ideal A and results of the examination for problem A
To reduce the number of backend services, I considered whether it would be possible to consolidate NEGs using Cloud Service Mesh. (Figure 4)
Figure 4 is the ideal diagram I envisioned before researching Cloud Service Mesh for Cloud Run.
Bookinfo, a sample application provided by IstioWe considered this as an example.
(In the development environment where we originally intended to use this, we communicate using gRPC, so we will use gRPCRoute.)
Figure 5 shows the results of our investigation into Cloud Service Mesh for Cloud Run. Cloud Service Mesh controls the path to the Backend Service, but it does not have the functionality to consolidate NEGs or Backend Services.
Ideal B for Problem B
Figure 6 shows the configuration I considered, similar to the one for Task B. With this configuration, if we change the HTTPRoute, the ideal situation is that when reviews-v3 starts returning a response, reviews-v2 stops returning any responses at all.
(The API I actually want to use communicates via gRPC, so I would use gRPCRoute, but I'm using HTTPRoute for testing with Bookinfo.)
Figure 7 shows the actual configuration we were able to build. With Cloud Service Mesh, it's not possible to register NEGs with revision tags in the Backend Service, so we had to create a separate Cloud Run service for each release version. We concluded that this configuration was unacceptable from an operational standpoint.
We have not conducted performance tests to verify whether all traffic switches over as intended.
Examination of Ideal B with Bookinfo
Here's the Terraform I used when I tested whether it was actually possible to build it using the Istio sample application, Bookinfo.
Configuration Description
In this example, we are using HTTPRoute because we are utilizing Bookinfo, but the same settings can be configured using gRPCRoute.
google_network_services_http_route.reviewsThe behavior verification details are from translucens.I tried out Cloud Run's service mesh.This will be omitted as it is the same as on this blog.
Terraform sample
Here is a sample of Terraform code that builds Bookinfo.
Please modify the example section as needed.
Based on our considerations, if we could specify tags for Cloud Run in NEG...
INTERNAL_SELF_MANAGED
This constraint meant that we had to create a Cloud Run service for each release version.
Other things I dream about with Cloud Service Mesh for Cloud Run
It would be great if we could access Cloud Run directly from the load balancer via Cloud Service Mesh. However, this feature seems to deviate from its intended purpose, as it was introduced to improve traffic between Cloud Run services.
In conclusion
In this article, we attempted to solve the following problems with Cloud Service Mesh for Cloud Run and concluded that it cannot.
- Unifying the BackendService across multiple development environments in applications using gRPC.
- Blue/Green deployments for applications using gRPC, where traffic switches instantly.
SRG is looking for new team members.
If you are interested, please contact us here.