About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/A dramatic improvement in Terraform execution time on AmebaPlatform

A dramatic improvement in Terraform execution time on AmebaPlatform

This article is aboutCyberAgent Group SRE Advent Calendar 2024This is the article for day 22.
 
This is Taniguchi from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article describes an activity that improved the execution time of Terraform running on GitHub Actions.
 

First

terraform apply
There were numerous requirements to modify resources such as security groups and ALB ListerRules, or to upgrade resource versions, and for a long time, it took a lot of time for all users to perform these operations.
This article describes the efforts made to improve Terraform running on GitHub Actions in order to address this situation.

The technology (service) used this time

  • terraform
  • GitHub Actions

Timing of execution of Terraform Plan and Apply for each environment

Basically, when you submit a Pull Request (PR) to a branch for a specific environment, a Plan is run, other people review it, and then an Apply is run when it's merged.
For example, if you want to perform a plan on the devlop environment, you submit a pull request to the develop branch, review the plan's contents, and if there are no problems, merge it and execute Apply.

Understanding the current situation

My gut feeling was that the Plan/Apply process was taking a considerable amount of time, so I measured how long it was currently taking so that I could measure how much improvement I actually achieved after making improvements. I started by using a Python script to call the GitHub API and retrieve the Plan and Apply times for one month. The following are the results obtained from the script.
From the results above, we can see that even looking at the median, both Plan and Apply take more than 5 minutes to execute.

Cause analysis and corrective plan

I have an idea of ​​the cause; initially, there were only a few products, so I was running both Plan and Apply on all of them. Of course, as the number of products increased, the execution time of Plan and Apply also increased. Furthermore, it was a real hassle when differences appeared in parts I hadn't changed, requiring me to investigate and resolve those changes.
Therefore,Revised planWith that in mind, we decided to change the system so that only products that have undergone changes on GitHub will run Plan/Apply.
However, with this method, changes that were previously noticed because Plan was run all the time may become undetectable, and for products that receive few updates, Plan may never be run, potentially leading to situations where the system fails to function when you try to execute it.
To avoid situations like this occurring when a failure happens, we've implemented a workflow that runs a Plan for all products once a day.

Implementation of Plan/Apply on GitHub Actions

In short, Terraform has the following directory structure.
git diff
Create a file with a unique filename that includes the directory path and PR number, and then upload it.
Initially, I had planned to upload to S3,GitHubI was told that Actions has a place to upload files called artifacts, so I decided to use that. There are limitations on size, retention period, number of files, and file name length, but I determined that none of them would be a problem.
By the way, uploading files is now easy with GitHub Actions.upload-artifactI used it.
After reviewing the Plan results and confirming there are no issues, we run Apply. As mentioned earlier, Apply runs when a user makes a merge. At the time of Apply, we use a unique path created with the PR number and other information to download the artifacts. Based on the downloaded files, we run Apply to the products that have been verified in Plan.

Implementation of health check

workflow_dispatch
Furthermore, while previously Plan/Apply failures were only notified via email, we have improved this by now sending notifications to a specific Slack channel, making it easier for the person in charge or the person executing the task to notice. This allows the person in charge and users to quickly become aware of failures.

Improvement results

These are the honest figures for the past month.
335.0 -> 100.0（改善率約70.15%）
The reason the Apply process is taking so long is that at the time of improvement, we were using EKS self-managed groups, so there were no node group updates on the Terraform side. However, now that we have implemented Karpenter, updating managed node groups takes a considerable amount of time, which is why the Max and Average values ​​are so high.

In conclusion

This article describes an example of improving the Plan/Apply process of Terraform using GitHub Actions on AmebaPlatform. This initiative significantly reduced the time required to execute Plan/Apply, improving productivity.
There seem to be other areas where productivity can be improved, so I'd like to make those improvements and share them on the blog again.
 
SRG is looking for new team members. If you are interested, please contact us here.