Verification of EKS Auto Mode
Suzuki from the Service Reliability Group (SRG) of the Media Management Division@sZma5a)is.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article shares the results of our technical verification of EKS's operational automation function, "EKS Auto Mode," and specifically explains its benefits and the technical limitations that led us to decide against implementing it.
Introduction
Amazon EKS, a managed Kubernetes service, simplifies the deployment and management of containerized applications, but still incurs significant operational costs, such as add-on updates and node management.
To further reduce these operational burdens and create an environment where developers can focus on more essential tasks, we considered introducing "EKS Auto Mode," a new feature that significantly automates EKS operations.
This article shares the results of our technical evaluation, the benefits we gained, and the reasons why we decided against its implementation.
What is EKS Auto Mode?
EKS Auto Mode is a feature designed to reduce the operational burden of Amazon EKS.
It allows you to delegate node management tasks such as node provisioning and scaling, OS patching, and middleware management tasks such as updating specific official add-ons to AWS. This abstracts the complexity of cluster operations and is expected to significantly reduce the effort required for infrastructure management.
In EKS Auto Mode, the following components are primarily managed by AWS:
- Amazon VPC CNI (Container Network Interface)
- AWS Load Balancer Controller
- CoreDNS (Cluster DNS)
- kube-proxy (network proxy)
- Karpenter (Node Provisioner)
- AWS EBS CSI driver (Container Storage Interface Driver for EBS)
Reduced operational workload through implementation
The biggest advantage of implementing EKS Auto Mode is the significant reduction in operational workload.
It eliminates the need for manual updates of official EKS add-ons.
In our environment, updating each add-on, including research and verification, took approximately two weeks.
When you add up all these tasks and consider the update frequency of each add-on, it works out to approximately 26 to 30 days of work per month. By implementing EKS Auto Mode, these add-ons are managed and verified by AWS before being updated, significantly reducing verification time.
Furthermore, security fixes are applied automatically, which significantly reduces the burden of vulnerability management—a major advantage.
Cost trade-offs
While it reduces operational workload, using EKS Auto Mode incurs additional charges.
Based on our calculations in our environment, implementation is expected to increase overall AWS usage costs by approximately 3.5%.
We need to weigh this cost increase against the aforementioned reduction in man-hours to decide whether or not to implement it. In our environment, we determined that this cost increase was acceptable compared to the man-hours saved.
How to apply EKS Auto Mode
When you enable EKS Auto Mode from the dashboard, managed add-on controllers such as Karpenter and AWS Load Balancer Controller are first added to the EKS control plane managed by AWS.
In addition, a NodeClass is provided for creating nodes that support Auto Mode. By using this to place a Node Pool under the management of a managed Karpenter, the nodes belonging to that pool will be automatically provisioned in a way that allows them to utilize the EKS Auto Mode functionality.
The important point is that while custom resources and controllers are provided upon activation, the actual functional differences in operation will not become apparent unless you switch or add Node Pools. Regarding billing, it is per node, and the target nodes are those under managed Karpenter.
Furthermore, because the managed add-ons have different API versions than the self-managed ones, they can coexist within the same cluster, making it easier to adopt flexible deployment patterns such as phased migration or adoption of only certain workloads.
Technical limitations and challenges
During our testing, we discovered that EKS Auto Mode has several significant technical limitations.
Add-ons cannot be downgraded.
Managed add-ons are updated in conjunction with the cluster version, so you cannot arbitrarily downgrade the version.
Fixed network interface
The container network interface (CNI) is fixed to the AWS VPC CNI.
In our environment, we were already using the AWS VPC CNI, so there were no issues. However, if you are using a different CNI, migration will be necessary.
Notes on migrating TargetGroupBinding
When migrating from an existing self-managed AWS Load Balancer Controller to EKS Auto Mode, reusing the same target group can cause conflicts and lead to downtime.
To ensure a safe migration, it is necessary to create a new target group and transfer the traffic to it.
Node lifecycle
terminationGracePeriodSecondsFixed node OS
The node's OS is fixed to "Bottlerocket," a Linux distribution optimized for container execution. Therefore, if you have installed OS-level customizations or security tools that monitor at the kernel level, these may not be compatible.
.local
.localSecurity Groups for Pods (SGP) is unavailable.
The biggest challenge we encountered during testing was the lack of the "Security Groups for Pods" feature, which allows assigning individual security groups to each Pod.
In our environment, we utilize this feature extensively due to our multi-tenant setup, and we concluded that securing an alternative to meet our security requirements would be difficult.
Summary of Verification Results and Future Prospects
EKS Auto Mode is a very attractive feature, with the potential to reduce add-on management workload by up to 30 days per month.
However, it became clear that there are technical limitations that we cannot compromise on in our environment, such as the inability to use .local domains and the lack of support for Security Groups for Pods. In particular, the inability to use Security Groups for Pods, which is related to security, was the decisive factor in postponing its implementation.
In conclusion, we have decided to postpone the implementation of EKS Auto Mode at this time, but since EKS Auto Mode itself is still new, we expect these issues to be resolved in the future. Therefore, our team has decided that while migration is difficult at present, we will actively pursue its implementation once these constraints are resolved.
For this reason, we will be closely monitoring the support for Security Groups for Pods, and will proceed with implementation again once the issue is resolved.
SRG is looking for new team members.
If you are interested, please contact us here.