Deep Dive into VPC CNI: An in-depth look at IPAMD and Security Groups For Pods

AWS VPC Add-on

• CNI Plugin
According to the CNI specification, communicate with IPAMD via gRPC to obtain an IP and perform network settings.

• IPAMD Plugin
Manage AWS ENI on each node and pre-assign IP addresses for quick Pod startup.

structure

• Pod IP allocation
IP address assignment to a Pod is done using EC2 ENI and IPAMD. The general flow is as follows:

• Network Routing
Network routing between Pods and from Pods to the outside world is configured using multiple routing tables and IPTABLE. The differences in their usage are as follows:
• Pod ↔ External network: IPTABLE
Like regular SNAT, this is performed as packets traverse virtual network interfaces (veth) and EC2 ENIs (eth, etc.).
• Pod ↔ Pod: Routing table
Communication between Pods and from Pods to the outside world is mainly carried out via veth and eth, and various routing tables are implicitly applied.

Life Cycle

1. /host/opt/cni/bin

2. Start IPAMD

3. grpc-health-probe

4. RunCurrently it is commented out because it is used as an InitContainer.
InitContainer

5. Copy the CNIConfig file to the specified directory on the kubelet.

6. Wait for IPAMD to start

CNI Plugin Details

• PluginMainWithError

• Specifications of the CNI Plugin itself:https://www.cni.dev/docs/spec/#cni-operations

1. Loading the configuration
POD_SECURITY_GROUP_ENFORCING_MODE
• add cmd: error loading config from args

2. Reading Kubernetes arguments
cniTypes.LoadArgs

3. Setting up a gRPC connection to IPAMD

4. Add Network API Request
AddNetwork
AddNetwork

5. Implementing network settings
driverClient.SetupBranchENIPodNetwork
PodVlanId

1. Loading the configuration
Same as Add command

2. Reading Kubernetes arguments
Same as Add command

3. Delete attempt due to previous result
tryDelWithPrevResult

4. Setting up a gRPC connection to IPAMD

5. Network Delete API Request
DelNetwork

6. Cleaning up network resources
It will release the Pod IP, etc. If SGP is used, it will also delete the Branch ENI.

More about IPAMD Plugin

1. The initialization process involves:
• Checking the connection to the k8s API Server
• k8s API Client initialization
• Initializing the Recorder to emit k8s Events
• Initializing the IPAMD Client

2. NodeIPPoolManager
ipamd.log

3. Run Prometheus Metrics API Server in a separate goroutine
/metrics:61678

4. Run the introspection API Server in a separate goroutine
It will also be started if not disabled by an environment variable. This Introspection API is used to monitor and diagnose the operation of IPAMD. Currently, the Introspection API provides four main functions:
• Get ENI information from IPAMD Context
• Get a specific ENI setting name from node information
• Get network configuration debug information
• Get IPAMD configuration debug information

5. Starting the gRPC Server

big picture

failed to assign an IP address to container

1. Processing when PodENI is enabled (using SGP)
1. vpc.amazonaws.com/pod-eni
Pods that are SGP-targeted will be modified by VPCCNI in the following two places:
2. "vpc.amazonaws.com/pod-eni"

2. If the IP does not exist in the annotation, get the IP from the EC2 ENI Datastore

3. If an IP exists in the annotation, add that IP to the existing VPC IP Pool.

4. Respond to IP addition to CNI Plugin

• Some pods are getting stuck without external network

• AWS EKS - add cmd: failed to assign an IP address to container

• Using an EC2 Instance Type that does not support SGP (e.g. using a t-series instance)

• The number of Pods that can be created by Trunk ENIBranch ENI countexceeded.

Security Groups For Pods

structure

Setting method and ENV explanation

• ENABLE_POD_ENI=true

• POD_SECURITY_GROUP_ENFORCING_MODE=standard

• DISABLE_TCP_EARLY_DEMUX=true

• Strict Mode
• All inbound/outbound traffic to a pod with a SG is controlled only by the SG of the branch ENI, while all inbound/outbound traffic between pods is inside the VPC.

• Standard Mode
• (Within the VPC) All communications are subject to the SG of both the Primary ENI and Branch ENI.
• All inbound/outbound traffic to Pods with SG is controlled only by the Branch ENI SG. However, inbound/outbound traffic from the kubelet is controlled by the Node SG and the Branch ENI SG rules do not apply.When using SGP, you must configure the NodeSG and Branch ENI SG on the Pod at the same time.
• For outbound traffic to outside the VPC (External VPN/Direct Connection/External VPC), the following conditions apply:
• AWS_VPC_K8S_CNI_EXTERNALSNAT
• AWS_VPC_K8S_CNI_EXTERNALSNAT

Any of the WARM targets do not impact the scale of the branch ENI pods so you will have to set the WARM_{ENI/IP/PREFIX}_TARGET based on the number of non-branch ENI pods. If you are having the cluster mostly using pods with a security group consider setting WARM_IP_TARGET to a very low value instead of default WARM_ENI_TARGET or WARM_PREFIX_TARGET to reduce wastage of IPs/ENIs.

constraints

• As mentioned above, when creating a SecurityGroupPolicy, the SecurityGroup set for the Pod must include both the Branch ENI SG and the Node SG.

• Since trunk ENIs are included in the number of ENIs that a node can have, if the number of ENIs on a node reaches the number of ENIs supported by the instance type being used, trunk ENIs will not be created and Pods with SG applied on the node will not be created.

• There is a limit to the number of branch ENIs (i.e. the number of Pods) that can be created with a runk ENI. This is not listed in the documentation, so please refer toSource codePlease calculate the actual number from the map. The calculation method is as follows.

• IsTrunkingCompatible: true

• Since restarting a Pod after applying SGP takes longer than usual, please use the most appropriate Deployment Update Strategy.

References

Conclusion