Deep Dive into VPC CNI: A Thorough Analysis of IPAMD and Security Groups for Pods

AWS VPC Add-on

• CNI Plugin
In accordance with the CNI specifications, communicate with IPAMD using gRPC to obtain an IP address and configure network settings.

• IPAMD Plugin
Each node manages an AWS ENI and pre-assigns IP addresses for rapid Pod startup.

structure

• Pod IP assignment
IP address assignment to Pods is done using EC2 ENIs and IPAMD. The general flow is as follows:

• Network routing
Network routing between Pods and from Pods to the outside world is configured using multiple routing tables and IPTABLES. The differences in their usage are as follows:
• Pod ↔ External Network: IPTABLE
Like a normal SNAT, this is performed as packets pass through the virtual network interface (veth) and EC2 ENI (such as eth).
• Pod ↔ Pod: Routing Table
Communication between Pods and communication from Pods to the outside world primarily takes place via veth and eth, and various routing tables are implicitly applied.

Lifecycle

1. /host/opt/cni/bin

2. Launch IPAMD

3. grpc-health-probe

4. Make it runIt is currently being used as an InitContainer, so it is commented out.
InitContainer

5. Copy the CNIConfig file to the designated directory in kubelet.

6. Waiting for IPAMD to start up

Details of CNI Plugin

• PluginMainWithError

• Specifications of the CNI Plugin itself:https://www.cni.dev/docs/spec/#cni-operations

1. Load settings
POD_SECURITY_GROUP_ENFORCING_MODE
• add cmd: error loading config from args

2. Loading Kubernetes arguments
cniTypes.LoadArgs

3. Setting up gRPC connection to IPAMD

4. Additional API Request for Network
AddNetwork
AddNetwork

5. Implementation of network settings
driverClient.SetupBranchENIPodNetwork
PodVlanId

1. Load settings
It's the same as the Add command.

2. Loading Kubernetes arguments
It's the same as the Add command.

3. Deletion attempt based on previous results
tryDelWithPrevResult

4. Setting up gRPC connection to IPAMD

5. Network Deletion API Request
DelNetwork

6. Cleaning up network resources
This process involves releasing Pod IPs, etc. If SGP is used, it will also delete the Branch ENI.

Details of IPAMD Plugin

1. The initialization process includes the following:
• Verifying connection with the Kubernetes API Server
• k8s API Client initialization
• Initializing a Recorder to issue k8s Events
• Initialization of IPAMD Client

2. NodeIPPoolManager
ipamd.log

3. Starting the Prometheus Metrics API Server in a separate goroutine
/metrics:61678

4. Starting the Introspective API Server in a separate goroutine
Similarly, it will start unless disabled by environment variables. This Introspection API is used to monitor and diagnose the operational status of IPAMD. Currently, the Introspection API provides the following four main functions:
• Retrieve ENI information from IPAMD Context
• Retrieve a specific ENI configuration name from Node information.
• Get debug information for network settings
• Get debug information for IPAMD environment settings

5. Starting the gRPC Server

big picture

failed to assign an IP address to container

1. Processing when PodENI is enabled (using SGP)
1. vpc.amazonaws.com/pod-eni
The Pods targeted by SGP will be modified by VPCCNI in the following two places:
2. "vpc.amazonaws.com/pod-eni"

2. If an IP address does not exist in the annotation, retrieve the IP address from the EC2 ENI Datastore.

3. If an IP address exists in the annotation, add that IP address to the existing VPC IP Pool.

4. Sends a response to add an IP address to the CNI Plugin.

• Some pods are getting stuck without external network

• AWS EKS - add cmd: failed to assign an IP address to container

• Using an EC2 Instance Type that does not support SGP (e.g., using a t-series instance type)

• The number of Pods is sufficient to create a Trunk ENI.Branch ENI countIt exceeded expectations.

Security Groups For Pods

structure

Setup instructions and explanation of ENV

• ENABLE_POD_ENI=true

• POD_SECURITY_GROUP_ENFORCING_MODE=standard

• DISABLE_TCP_EARLY_DEMUX=true

• Strict mode
• All inbound/outbound traffic to Pods with Security Groups (SGs) is controlled solely by the SGs of the Branch ENI. On the other hand, all inbound/outbound traffic between Pods enters the VPC.

• Standard mode
• (Within the VPC) All communications are subject to both the Primary ENI and Branch ENI Security Groups (SGs).
• All inbound/outbound traffic to Pods with SG is controlled solely by the Branch ENI's SG. However, inbound/outbound traffic from kubelet is controlled by the Node SG, and the Branch ENI's SG rules do not apply to it.When using SGP, you need to configure both NodeSG and Branch ENI SGs on the Pod simultaneously.
• The following conditions apply to outbound traffic to locations outside the VPC (External VPN/Direct Connection/External VPC).
• AWS_VPC_K8S_CNI_EXTERNALSNAT
• AWS_VPC_K8S_CNI_EXTERNALSNAT

Any of the WARM targets do not impact the scale of the branch ENI pods so you will have to set the WARM_{ENI/IP/PREFIX}_TARGET based on the number of non-branch ENI pods. If you are having the cluster mostly using pods with a security group consider setting WARM_IP_TARGET to a very low value instead of default WARM_ENI_TARGET or WARM_PREFIX_TARGET to reduce wastage of IPs/ENIs.

constraints

• As mentioned earlier, when creating a SecurityGroupPolicy, the SecurityGroup to be configured for a Pod must include both Branch ENI SG and Node SG.

• Since Trunk ENIs are included in the number of ENIs a Node can have, if the number of ENIs on a Node reaches the number of ENIs supported by the Instance Type being used, Trunk ENIs will not be created, and therefore, Pods to which Security Groups (SGs) are applied will not be created on that Node.

• There is a limit to the number of Branch ENIs (i.e., Pods) that a runk ENI can create. This is not mentioned in the documentation, so please refer to this [link/reference].Source codePlease calculate the actual number from the Map. The calculation method is as follows:

• IsTrunkingCompatible: true

• Pod restarts after applying SGP take longer than usual, so please adopt the optimal Deployment Update Strategy.

References

In conclusion