Deep Dive into VPC CNI: An In-Depth Analysis of IPAMD and Security Groups For Pods

AWS VPC Add-on

• CNI Plugin
In accordance with the CNI specification, communicate with IPAMD via gRPC to obtain an IP and perform network configuration.

• IPAMD Plugin
Manage AWS ENI on each node and pre-assign IP addresses for quick pod startup.

structure

• Pod IP allocation
IP address assignment to Pods is done using EC2 ENI and IPAMD, the general flow is as follows:

• Network Routing
Network routing between Pods and from Pods to the outside world is configured using multiple routing tables and IPTABLEs. They are used for the following reasons:
• Pod ↔ External network: IPTABLE
Like regular SNAT, this is performed when packets pass through a virtual network interface (veth) and an EC2 ENI (e.g. eth).
• Pod ↔ Pod: routing table
Communication between Pods and from Pods to the outside world is mainly carried out via veth and eth, and various routing tables are implicitly applied.

Life Cycle

1. /host/opt/cni/bin

2. Launch IPAMD

3. grpc-health-probe

4. RunIt is currently commented out because it is used as an InitContainer.
InitContainer

5. Copy the CNIConfig file to the specified directory on the kubelet.

6. Wait for IPAMD to start

CNI Plugin Details

• PluginMainWithError

• Specifications of the CNI Plugin itself:https://www.cni.dev/docs/spec/#cni-operations

1. Loading the settings
POD_SECURITY_GROUP_ENFORCING_MODE
• add cmd: error loading config from args

2. Reading Kubernetes arguments
cniTypes.LoadArgs

3. Configuring a gRPC connection to IPAMD

4. Add Network API Request
AddNetwork
AddNetwork

5. Implementing network settings
driverClient.SetupBranchENIPodNetwork
PodVlanId

1. Loading the settings
Same as the Add command

2. Reading Kubernetes arguments
Same as the Add command

3. Delete attempt with previous results
tryDelWithPrevResult

4. Configuring a gRPC connection to IPAMD

5. Network deletion API request
DelNetwork

6. Cleaning up network resources
This will release the Pod IP, etc. If an SGP is used, it will also delete the Branch ENI.

IPAMD Plugin Details

1. The initialization process includes:
• Checking connection with k8s API Server
• k8s API Client initialization
• Initializing the Recorder to emit k8s Events
• Initializing the IPAMD Client

2. NodeIPPoolManager
ipamd.log

3. Starting Prometheus Metrics API Server in a separate goroutine
/metrics:61678

4. Starting the introspection API Server in a separate goroutine
It will also be started if not disabled by an environment variable. This introspection API is used to monitor and diagnose the operation status of IPAMD. The introspection API currently provides four main functions:
• Get ENI information from IPAMD Context
• Get a specific ENI setting name from the node information
• Get debug information for network settings
• Get debug information for IPAMD preferences

5. Starting the gRPC Server

big picture

failed to assign an IP address to container

1. Processing when PodENI is enabled (using SGP)
1. vpc.amazonaws.com/pod-eni
The SGP target Pod will be modified by VPCCNI in the following two places:
2. "vpc.amazonaws.com/pod-eni"

2. If the IP does not exist in the annotation, get the IP from the EC2 ENI Datastore

3. If an IP exists in the annotation, add that IP to the existing VPC IP Pool.

4. Respond to IP addition to CNI Plugin

• Some pods are getting stuck without external network

• AWS EKS - add cmd: failed to assign an IP address to container

• Using an EC2 instance type that does not support SGP (e.g., using a t series instance)

• The number of Pods that can be created by Trunk ENIBranch ENI countexceeded.

Security Groups For Pods

structure

Setting method and ENV explanation

• ENABLE_POD_ENI=true

• POD_SECURITY_GROUP_ENFORCING_MODE=standard

• DISABLE_TCP_EARLY_DEMUX=true

• Strict Mode
• All inbound/outbound traffic to a Pod with a SG is controlled solely by the SG of the Branch ENI, while all inbound/outbound traffic between Pods is within the VPC.

• Standard mode
• (Within VPC) All communications are subject to the SG of both the Primary ENI and Branch ENI.
• All inbound/outbound traffic to pods with SG is controlled only by the SG of the branch ENI. However, inbound/outbound traffic from the kubelet is controlled by the Node SG, and the rules of the SG of the branch ENI do not apply.When using SGP, you must configure the NodeSG and Branch ENI SG for the Pod at the same time.
• The following conditions apply to outbound traffic to outside the VPC (External VPN/Direct Connection/External VPC).
• AWS_VPC_K8S_CNI_EXTERNALSNAT
• AWS_VPC_K8S_CNI_EXTERNALSNAT

Any of the WARM targets do not impact the scale of the branch ENI pods so you will have to set the WARM_{ENI/IP/PREFIX}_TARGET based on the number of non-branch ENI pods. If you are having the cluster mostly using pods with a security group consider setting WARM_IP_TARGET to a very low value instead of default WARM_ENI_TARGET or WARM_PREFIX_TARGET to reduce wastage of IPs/ENIs.

constraints

• As mentioned above, when creating a SecurityGroupPolicy, the SecurityGroup set for the Pod must include both the Branch ENI SG and the Node SG.

• Since trunk ENIs are included in the number of ENIs that a node can have, if the number of ENIs on a node reaches the number of ENIs supported by the instance type in use, trunk ENIs will not be created, and pods with SG applied on the node will not be created.

• The number of Branch ENIs (i.e., the number of Pods) that can be created with a runk ENI is limited. This is not listed in the documentation, so please refer toSource codePlease calculate the actual number from the map below. The calculation method is as follows.

• IsTrunkingCompatible: true

• Since restarting a Pod after applying SGP takes longer than usual, please use the most appropriate Deployment Update Strategy.

References

Conclusion