About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/A story about how I smoothly upgraded my cluster using Amazon Aurora MySQL's Blue/Green Deployments feature.

A story about how I smoothly upgraded my cluster using Amazon Aurora MySQL's Blue/Green Deployments feature.

CyberAgent Group SRE Advent Calendar 2023This is the article for day 20.
 
This is Onikai (@fat47) from the Service Reliability Group (SRG) of the Technology Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article is located within SRG.DBWGThe DB Working Group is releasing database-related materials that it provides to the entire company.
I hope this is of some help.
This article describes how I upgraded from Aurora MySQL Version 2 (MySQL 5.7 compatible) to Version 3 (MySQL 8.0 compatible) using Aurora's Blue/Green Deployments feature.
 

What are Aurora Blue/Green Deployments?

This is a feature available in RDS and Aurora, which was announced at Re:Invent 2022.
 
You can create and switch to a Green cluster from the management console, using the currently running RDS cluster as the Blue cluster.
To summarize the flow of functions,
  • You can easily create a mirrored environment (Green) from the production environment (Blue) directly from the console.
    • At this time, an endpoint for Green is also created, so you can verify its operation by pointing the application's database destination to Green.
  • Logical replication is established between the Blue and Green channels, so updates are synchronized.
  • You can switch between Blue and Green from the console.
    • The endpoint referenced by the application does not need to be changed; only the destination within the endpoint changes.
  • During the switchover, both Blue and Green write blocks are active.
  • If the B/G switching is not completed within the specified time, it will automatically switch back.
  • The Blue environment remains as a standalone environment after the switchover.
    • In the Blue environment, -old1 is added to the cluster name and instance name.
It will look something like this.
AWS公式ドキュメントから引用
Quoted from the official AWS documentation.

Upgrade procedure

This document describes the procedures for development and staging environments.
The application environment that we actually upgraded this time was built using Go.
  • Create a Green cluster with Aurora MySQL Version 3 using the B/G Deployments feature.
  • Change the application's target to the Green endpoint and perform operational verification.
    • If you change the write direction at this point, it will result in inconsistency with the Blue environment. Therefore, after confirming that it is working correctly, it is best to delete the Green cluster and create a new Green cluster.
  • If there are no problems during the operational check, the application's destination will be returned to the original endpoint.
  • Perform a B/G switch.
    • There will be approximately 1 minute of downtime until the switchover is complete.
  • Upgrade complete

Points to note when upgrading using the B/G function

The read/write downtime is approximately 1 minute.

When you perform a B/G switch, the endpoint's destination is changed on the backend.
During this time, applications connected to the database will experience read and write failures for approximately one minute.
You need to determine in advance whether this approximately one-minute downtime is acceptable for the services and applications you use.
If this timeframe is unacceptable, you will need to consider measures such as putting the entire service into maintenance mode beforehand.

Pay attention to the status of the parameter group's application.

When creating a Green cluster using the B/G function, you can specify a parameter group.
If you specify a parameter group that has been changed from the default settings, depending on the parameter content
An instance restart may be required.
In this case, this parameter is not applied when the instance starts and remains at its default value, so you need to manually restart the instance before switching to apply it.

Custom endpoints are also available.

If you use a custom endpoint on the Blue side, the same custom endpoint will be automatically configured on the Green side as well.
 

Once the switch is successful, it cannot be reversed.

For whatever reason, if you encounter problems after a version upgrade and want to revert to the old environment, you won't be able to.
After the switchover is successful, the old Blue instance simply remains as a separate environment.
 
Therefore, we can roll back by redirecting the application's database to Blue,
Since no replication has been established from Green, the system will revert to the data from the time the switchover was performed.
To completely revert to the original state, you need to put the service into maintenance mode to stop updates to the database, take a logical backup such as mysqldump from the Green environment, and restore it to the original version of the cluster.
It is not possible to use a snapshot taken from a Version 3 cluster to start a Version 2 cluster.
 
If these are unacceptable, you will need to avoid using the B/G function and instead create a separate cluster, build inter-cluster replication yourself, and then build reverse replication after the switchover.
 

Many parameters and behaviors have changed in Aurora Version 3, so please be aware of this.

While the B/G function is not relevant, many parameters and behaviors have changed in Aurora Version 3, so be sure to investigate and verify them thoroughly.
Some of these are introduced in another article.
 

In conclusion

Up until now, upgrading required creating a separate cluster and manually configuring inter-cluster replication,
Thanks to this Blue/Green Deployment feature, things have become considerably easier.
While it's convenient, there are some points to be aware of, so it's best to do thorough research before using it.
 
SRG is looking for new team members. If you are interested, please contact us here.