About SRGArticlesRecruitment Information
About SRGArticlesRecruitment Information
HOME/Articles/A quick cluster upgrade using Amazon Aurora MySQL's Blue/Green Deployments feature

A quick cluster upgrade using Amazon Aurora MySQL's Blue/Green Deployments feature

CyberAgent Group SRE Advent Calendar 2023This is the 20th article.
 
This is Oniumi (@fat47) from the Service Reliability Group (SRG) of the Technology Headquarters.
#SRG(Service Reliability Group) mainly provides cross-sectional support for the infrastructure of our media services, improving existing services, launching new ones, contributing to OSS, etc.
This article is located in SRGDBWGWe will publish materials regarding the database that the (DB Working Group) provides to the entire company.
I hope this helps in some way.
In this article, I will describe how I used Aurora’s Blue/Green Deployments feature to upgrade from Aurora MySQL Version 2 (compatible with MySQL 5.7) to Version 3 (compatible with MySQL 8.0).
 

What are Aurora Blue/Green Deployments?

This is a feature available with RDS and Aurora that was announced at Re:Invent last year in 2022.
 
The currently running RDS cluster will be designated as Blue, and you can create and switch to the Green cluster from the management console.
To summarize the function flow:
  • You can easily create a mirror environment (Green) from the production environment (Blue) from the console.
    • At this time, an endpoint for Green is also created, so you can check the operation by pointing the application's database destination to Green.
  • Logical replication is established between Blue and Green, so updates are synchronized.
  • Switch from Blue to Green from the console
    • The endpoint referenced by the application does not need to be changed, and the destination within the endpoint can be changed.
  • During switching, both Blue and Green are write-blocked.
  • If the B/G switch is not completed within the specified time, it will automatically switch back.
  • After the switch, the Blue environment will remain a standalone environment.
    • -old1 will be added to the cluster and instance names in the Blue environment.
It will look something like this.
AWS公式ドキュメントから引用
Quoted from the official AWS documentation

Upgrade Procedure

This section describes the procedures for development and staging environments.
The application environment that was actually upgraded this time was built using Go.
  • Create a Green cluster with Aurora MySQL Version 3 using the B/G Deployments feature
  • Change the application's destination to the Green endpoint and check the operation.
    • If you change the write direction at this time, it will become inconsistent with the Blue environment, so it is better to delete the Green cluster after completing the operation check and then create a new Green cluster.
  • If the operation check shows no problems, return the application to its original endpoint.
  • Execute B/G switching
    • There will be about 1 minute of downtime until the switchover is complete.
  • Upgrade complete

Important points to note when upgrading using the B/G function

Read/write downtime is approximately 1 minute

When you switch B/G, the destination of the endpoint changes behind the scenes.
In that case, applications connected to the database will fail to read or write for approximately one minute.
You need to determine in advance whether a minute or so of downtime can be tolerated for the services and applications you use.
If this time is unacceptable, you should consider putting the entire service into maintenance mode in advance.

Pay attention to the reflection status of parameter groups

When creating a Green cluster with the B/G function, you can specify a parameter group.
If you specify a parameter group that is different from the default, depending on the parameter
You may be prompted to reboot the instance.
In this case, this parameter is not applied at instance startup and remains at the default value, so you must manually restart the instance before switching to apply it.

Custom endpoints available

If you use a custom endpoint on the blue side, the same custom endpoint will be automatically set on the green side.
 

Once the switch is successful, you cannot switch back.

Even if for some reason you run into a problem after upgrading and want to revert to the old environment, you can't.
After the switchover is successful, the old Blue instance will simply remain as an independent environment.
 
So, you can switch back by pointing the application's DB to Blue, but
Since there is no replication from Green, the data will rewind to the time the switch was performed.
To completely revert to the original state, you need to put the service into maintenance mode to stop updates to the database, take a logical backup such as mysqldump from the Green environment, and then restore it to the original version of the cluster.
Snapshots taken on a Version 3 cluster cannot be used to start a Version 2 cluster.
 
If these are not acceptable, you will need to create a separate cluster and set up inter-cluster replication yourself as before without using the B/G function, and then set up replication in the reverse direction after the switchover.
 

Please note that many parameters and behaviors have changed in Aurora Version 3.

This has nothing to do with the B/G function, but many parameters and behaviors have changed in Aurora Version 3, so be sure to investigate and check carefully.
Some of them are introduced in another article.
 

Conclusion

Until now, in order to upgrade, it was necessary to create a separate cluster and set up inter-cluster replication by yourself.
The introduction of this Blue/Green Deployment feature has made things much easier.
Although it is convenient, there are some things you need to be careful of, so be sure to do your research before using it.
 
SRG is looking for people to work with us. If you are interested, please contact us here.