About SRGTechnical ArticlesRecruitment Information
About SRGTechnical ArticlesRecruitment Information
HOME/Technical Articles/A quick cluster upgrade using Amazon Aurora MySQL's Blue/Green Deployments feature

A quick cluster upgrade using Amazon Aurora MySQL's Blue/Green Deployments feature

CyberAgent Group SRE Advent Calendar 2023This is the 20th article.
 
This is Oniumi (@fat47) from the Service Reliability Group (SRG) of the Technology Headquarters.
#SRG(Service Reliability Group) is a group that mainly provides cross-sectional support for the infrastructure of our media services, improving existing services, launching new ones, and contributing to OSS.
This article is located in SRGDBWGWe will publish materials regarding the database provided by the DB Working Group for the entire company.
I hope this helps in some way.
This article describes how we used Aurora's Blue/Green Deployments feature to upgrade from Aurora MySQL Version 2 (compatible with MySQL 5.7) to Version 3 (compatible with MySQL 8.0).
 

What are Aurora Blue/Green Deployments?

This is a feature available with RDS and Aurora, which was announced at Re:Invent 2022 last year.
 
You can create and switch to a Green cluster from the management console, with the currently running RDS cluster as Blue.
To summarize the function flow:
  • You can easily create a mirror environment (Green) from the production environment (Blue) from the console.
    • At this time, an endpoint for Green is also created, so you can check the operation by pointing the application's DB destination to Green.
  • Logical replication is established between Blue and Green, so updates are synchronized.
  • Switch from Blue to Green from the console
    • The endpoint referenced by the application does not need to be changed, and the destination within the endpoint is changed.
  • During the switchover, both Blue and Green are write-blocked.
  • If the B/G switch is not completed within the specified time, it will automatically switch back.
  • After the switchover, the Blue environment will remain as a standalone environment.
    • -old1 will be added to the cluster name and instance name in the Blue environment.
It will look something like this.
AWS公式ドキュメントから引用
Quoted from the official AWS documentation

Upgrade Procedure

This section describes the steps for development and staging environments.
The application environment that was actually upgraded this time was built using Go.
  • Create a Green cluster with Aurora MySQL Version 3 using the B/G Deployments feature
  • Change the application's destination to the Green endpoint and check the operation.
    • If you change the write direction at this time, it will become inconsistent with the Blue environment, so after you have finished checking the operation, it is better to delete the Green cluster and create a new Green cluster again.
  • If the operation check shows no problems, return the application to its original endpoint.
  • Execute B/G switching
    • There will be approximately one minute of downtime until the switchover is complete.
  • Upgrade complete

Important points to note when upgrading using the B/G function

Read/write downtime is approximately 1 minute

When you switch B/G, the destination of the endpoint changes behind the scenes.
In this case, applications connected to the database will fail to read or write for approximately one minute.
You need to determine in advance whether the services and applications you use can tolerate this one minute of downtime.
If this time is unacceptable, you should consider putting the entire service into maintenance mode in advance.

Pay attention to the parameter group reflection status

When creating a Green cluster using the B/G feature, you can specify a parameter group.
If you specify a parameter group that is different from the default, depending on the parameter
You may be prompted to reboot your instance.
In this case, this parameter is not applied at the time of instance startup and remains at the default value, so you must manually restart the instance before switching to apply it.

Custom endpoints are also available

If you use a custom endpoint on the Blue side, the same custom endpoint will be automatically configured on the Green side.
 

Once the switchover is successful, it cannot be reversed.

If for some reason you encounter a problem after upgrading and want to revert to the old environment, you cannot.
After the switchover is successful, the old Blue instance will simply remain as an independent environment.
 
Therefore, you can switch back by pointing the application's DB destination to Blue,
Since there is no replication from Green, the data will be rewound to the time the switchover was performed.
To completely revert to the original state, you need to put the service into maintenance mode to stop updates to the database, take a logical backup such as mysqldump from the Green environment, and restore it to the original version of the cluster.
Snapshots taken on a Version 3 cluster cannot be used to start a Version 2 cluster.
 
If you cannot tolerate these, you will need to create a separate cluster and set up inter-cluster replication yourself as usual without using the B/G function, and then set up reverse replication after switching.
 

Please note that many parameters and behaviors have changed in Aurora Version 3.

Although the B/G function is not affected, many parameters and behaviors have changed in Aurora Version 3, so be sure to investigate and check them carefully.
Some of them are introduced in another article.
 

Conclusion

Up until now, to upgrade, you had to create a separate cluster and manually configure inter-cluster replication.
Thanks to the introduction of this Blue/Green Deployment feature, things have become much easier.
Although it is convenient, there are some things you need to be careful about, so it is best to do your research before using it.
 
SRG is looking for people to work with us. If you're interested, please contact us here.