Understanding Cilium’s Network Routing Modes — Native Routing and Encapsulation
When deploying Cilium in Kubernetes environments, one of the most critical architectural decisions involves selecting the appropriate routing mode for pod-to-pod communication across nodes. This becomes particularly complex when nodes are distributed across different network segments, separated by routers or other network infrastructure. The fundamental challenge lies in ensuring that pod traffic can traverse these network boundaries while maintaining performance, security, and operational simplicity.
Conceptual Foundation of Multi-Network Kubernetes Deployments
In production Kubernetes environments, it’s increasingly common to encounter scenarios where worker nodes are distributed across multiple network segments. This distribution might occur due to various reasons including datacenter topology constraints, high availability requirements spanning multiple availability zones, or simply the organic growth of infrastructure over time. When nodes exist in different Layer 2 broadcast domains, the complexity of pod networking increases significantly because the default assumption of direct Layer 2 connectivity between all nodes no longer holds true.
The test environment described in the original article presents a realistic scenario with three nodes: a control plane node (k8s-ctr) and one worker node (k8s-w1) residing in the 192.168.10.0/24 network, while another worker node (k8s-w0) sits in the 192.168.20.0/24 network. These networks are connected through a router VM that acts as the gateway between the segments. This setup mirrors many real-world deployments where nodes might be distributed across different racks, data halls, or even geographical locations connected through MPLS or other WAN technologies.
The autoDirectNodeRoutes Option
The autoDirectNodeRoutes configuration option in Cilium represents a sophisticated approach to handling route installation based on network topology detection. When set to true, this option instructs Cilium to automatically detect which nodes share the same Layer 2 network segment and only install direct routes for pod CIDRs of nodes within the same segment. This intelligence prevents the creation of invalid routes that would attempt to reach pods on remote nodes through non-existent direct paths.
Let’s examine how this manifests in practice. When Cilium starts with autoDirectNodeRoutes=true, it performs network discovery to understand the topology:
# On k8s-ctr node, examining the routing table
ip route show | grep 172.20
172.20.0.0/24 via 172.20.0.52 dev cilium_host proto kernel src 172.20.0.52
172.20.1.0/24 via 192.168.10.101 dev eth1 proto kernel
# Note: No direct route to 172.20.2.0/24 (k8s-w0's pod CIDR) because it's in a different networkThe routing table on k8s-ctr shows direct kernel routes only to pod CIDRs of nodes in the same network segment. The route to 172.20.1.0/24 (k8s-w1’s pods) goes directly via 192.168.10.101 because both nodes share the 192.168.10.0/24 network. However, there’s no direct route to 172.20.2.0/24 (k8s-w0’s pods) because that node resides in a different network segment.
This automatic detection mechanism relies on ARP resolution and neighbor discovery. Cilium attempts to determine if a node is directly reachable by checking if it exists in the same broadcast domain. If ARP resolution succeeds without requiring a gateway, the node is considered local, and a direct route is installed. Otherwise, the node is deemed remote, and alternative routing mechanisms must be employed.
Native Routing Mode
Native routing mode, also known as direct routing mode, represents the most straightforward approach to pod networking where packets traverse the network infrastructure using their original pod IP addresses without any encapsulation. This mode offers several advantages including minimal overhead, simplified packet inspection for network security tools, and easier troubleshooting since packets remain in their original form throughout their journey.
However, when nodes span multiple networks, native routing mode faces significant challenges. Let’s deploy a test application to observe these challenges in action:
# Deploy the webpod application across all nodes
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: webpod
spec:
replicas: 3
selector:
matchLabels:
app: webpod
template:
metadata:
labels:
app: webpod
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- webpod
topologyKey: "kubernetes.io/hostname"
containers:
- name: webpod
image: traefik/whoami
ports:
- containerPort: 80
EOFAfter deployment, when we attempt to communicate with pods across network boundaries, we observe failures:
# From curl-pod on k8s-ctr, trying to reach all webpod instances
kubectl exec -it curl-pod -- sh -c 'for i in $(seq 1 10); do curl -s --connect-timeout 1 webpod | grep Hostname || echo "timeout"; sleep 1; done'
Hostname: webpod-697b545f57-9pg9x
Hostname: webpod-697b545f57-wpsqb
timeout
timeout
Hostname: webpod-697b545f57-9pg9xThe timeouts occur when the service attempts to route traffic to the pod on k8s-w0 (172.20.2.128) because the intermediate router lacks the necessary routing information. When we examine the router’s perspective:
# On the router VM
tcpdump -i any host 172.20.2.128 -nn
15:35:04.760194 eth1 In IP 172.20.0.170.41574 > 172.20.2.128.80: Flags [S], seq 127827144
15:35:04.760222 eth0 Out IP 172.20.0.170.41574 > 172.20.2.128.80: Flags [S], seq 127827144
# Packet goes out eth0 (default route) instead of eth2 (toward k8s-w0)The router receives packets destined for 172.20.2.128 but lacks specific routing information for the 172.20.2.0/24 network. Consequently, it forwards these packets via its default route (eth0), leading them to be dropped or lost. This demonstrates the fundamental challenge of native routing in multi-network environments: every intermediate network device must possess complete routing information for all pod CIDRs.
To temporarily resolve this issue, we can manually add static routes on the router:
# On the router VM - adding static routes for pod CIDRs
ip route add 172.20.0.0/24 via 192.168.10.100 # k8s-ctr pods via its node IP
ip route add 172.20.1.0/24 via 192.168.10.101 # k8s-w1 pods via its node IP
ip route add 172.20.2.0/24 via 192.168.20.100 # k8s-w0 pods via its node IP# Verify the routes
ip route | grep 172.20
172.20.0.0/24 via 192.168.10.100 dev eth1
172.20.1.0/24 via 192.168.10.101 dev eth1
172.20.2.0/24 via 192.168.20.100 dev eth2With these routes in place, cross-network pod communication immediately begins working:
# Testing again from curl-pod
kubectl exec -it curl-pod -- sh -c 'for i in $(seq 1 6); do curl -s --connect-timeout 1 webpod | grep Hostname; sleep 1; done'
Hostname: webpod-697b545f57-r5lfw
Hostname: webpod-697b545f57-wpsqb
Hostname: webpod-697b545f57-9pg9x
Hostname: webpod-697b545f57-r5lfw
Hostname: webpod-697b545f57-wpsqb
Hostname: webpod-697b545f57-9pg9xWhile manual route configuration solves the immediate problem, it introduces significant operational challenges. In dynamic Kubernetes environments where nodes might be added, removed, or have their pod CIDRs changed, maintaining these routes manually becomes impractical. This is particularly true in large-scale deployments with hundreds or thousands of nodes. Each intermediate router would need routes for every pod CIDR, and any change would require updates across all routing devices.
Furthermore, the Path MTU (Maximum Transmission Unit) must be carefully considered in native routing mode. As packets traverse different network segments, they might encounter links with varying MTU sizes. Without proper PMTU discovery or conservative MTU settings, packets might be dropped due to size constraints, leading to mysterious connectivity issues that are difficult to diagnose.
Encapsulation Mode
Encapsulation mode provides an elegant solution to the cross-network routing challenge by wrapping pod packets within another packet that uses node IP addresses for routing. This approach, commonly implemented using VXLAN (Virtual Extensible LAN) or Geneve protocols, creates an overlay network that abstracts away the underlying network topology complexity.
Before transitioning to encapsulation mode, we need to ensure the kernel has the necessary modules loaded:
# Check for VXLAN support in the kernel
grep -E 'CONFIG_VXLAN=y|CONFIG_VXLAN=m' /boot/config-$(uname -r)
CONFIG_VXLAN=m
# Load the VXLAN kernel module
modprobe vxlan
# Verify the module is loaded
lsmod | grep vxlan
vxlan 155648 0
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 32768 1 vxlan
# Load the module on all nodes
for node in k8s-w0 k8s-w1; do
ssh $node 'sudo modprobe vxlan'
doneNow, let’s transition Cilium from native routing to encapsulation mode. Note that this transition might cause temporary connectivity disruption:
# Monitor connectivity during the transition
kubectl exec -it curl-pod -- ping -i 0.5 $(kubectl get pod -l app=webpod --field-selector spec.nodeName=k8s-w0 -o jsonpath='{.items[0].status.podIP}')
# In another terminal, perform the Cilium upgrade
helm upgrade cilium cilium/cilium --namespace kube-system --version 1.18.0 --reuse-values \
--set routingMode=tunnel \
--set tunnelProtocol=vxlan \
--set autoDirectNodeRoutes=false \
--set installNoConntrackIptablesRules=false
# Restart Cilium pods to apply the new configuration
kubectl rollout restart -n kube-system ds/ciliumDuring the transition, you might observe:
64 bytes from 172.20.2.128: icmp_seq=45 ttl=62 time=1.30 ms
64 bytes from 172.20.2.128: icmp_seq=46 ttl=62 time=2.14 ms
# Brief interruption during restart
Request timeout for icmp_seq 47
Request timeout for icmp_seq 48
64 bytes from 172.20.2.128: icmp_seq=49 ttl=64 time=1.82 ms
64 bytes from 172.20.2.128: icmp_seq=50 ttl=64 time=1.45 msAfter the transition, Cilium creates a VXLAN tunnel interface on each node:
# Examine the VXLAN interface
ip -d link show cilium_vxlan
4: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 12:34:10:b3:d5:85 brd ff:ff:ff:ff:ff:ff
vxlan id 2 srcport 0 0 dstport 8472 nolearning ttl inherit ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx
# Check the modified routing table
ip route | grep cilium_host
172.20.0.0/24 via 172.20.0.250 dev cilium_host proto kernel src 172.20.0.250
172.20.1.0/24 via 172.20.0.250 dev cilium_host proto kernel src 172.20.0.250 mtu 1450
172.20.2.0/24 via 172.20.0.250 dev cilium_host proto kernel src 172.20.0.250 mtu 1450Notice that all pod CIDR routes now point to a virtual router IP (172.20.0.250) on the cilium_host interface, and remote pod CIDRs have a reduced MTU of 1450 bytes to accommodate the VXLAN header overhead.
With encapsulation mode active, let’s examine the traffic flow. First, remove any manual routes we added earlier on the router:
# On the router VM - remove manual routes
ip route del 172.20.0.0/24 via 192.168.10.100
ip route del 172.20.1.0/24 via 192.168.10.101
ip route del 172.20.2.0/24 via 192.168.20.100Even without these routes, pod communication across networks continues to work:
# Test connectivity
kubectl exec -it curl-pod -- curl -s webpod | grep Hostname
Hostname: webpod-697b545f57-9pg9x
# Generate traffic and capture on the router
kubectl exec -it curl-pod -- sh -c 'for i in $(seq 1 100); do curl -s webpod >/dev/null 2>&1; done' &
# On the router, capture VXLAN traffic
tcpdump -i any udp port 8472 -nn -c 10
17:08:19.935977 eth1 In IP 192.168.10.100.34509 > 192.168.20.100.8472: OTV, flags [I] (0x08), overlay 0, instance 46181
IP 172.20.0.170 > 172.20.2.128: ICMP echo request, id 4887, seq 1, length 64
17:08:19.936002 eth2 Out IP 192.168.10.100.34509 > 192.168.20.100.8472: OTV, flags [I] (0x08), overlay 0, instance 46181The captured traffic reveals the encapsulation structure. The outer packet uses node IP addresses (192.168.10.100 to 192.168.20.100) which the router knows how to route. The inner packet containing the actual pod-to-pod communication (172.20.0.170 to 172.20.2.128) is transparently carried across the network infrastructure.
For deeper packet analysis, we can use advanced tools:
# Capture detailed VXLAN traffic
tcpdump -i any udp port 8472 -w /tmp/vxlan_detailed.pcap -c 1000# Analyze with tshark
tshark -r /tmp/vxlan_detailed.pcap -Y "vxlan" -T fields \
-e frame.time_relative -e ip.src -e ip.dst -e vxlan.vni -e vxlan.flags.i | head -20
0.000000000 192.168.10.100 192.168.20.100 46181 1
0.002451000 192.168.20.100 192.168.10.100 38544 1
0.003124000 192.168.10.100 192.168.20.100 46181 1The VXLAN Network Identifier (VNI) field carries the security identity information, allowing Cilium to maintain network policies even across the overlay network. The instance value (46181, 38544) corresponds to the Cilium security identities of the communicating pods.
Let’s examine the complete packet flow in detail:
# Enable detailed BPF tracing on a Cilium node
kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- \
cilium monitor --type trace --from-endpoint $(kubectl get cep curl-pod -o jsonpath='{.status.id}')
# In another terminal, generate a single request
kubectl exec -it curl-pod -- curl -s webpod --max-time 1
# Sample trace output showing encapsulation
xx drop (Policy denied) flow 0x0 to endpoint 0, , identity 2->0: 172.20.0.170:54892 -> 10.96.71.46:80 tcp SYN
xx trace (Xlate) flow 0x0 to endpoint 0, , identity 2->38544: 172.20.0.170:54892 -> 172.20.2.128:80 tcp SYN
xx trace (Encap) flow 0x0 to endpoint 0, , identity 2->38544: 192.168.10.100 -> 192.168.20.100 vxlan SYNThe trace shows three stages: initial policy evaluation, translation from service IP to pod IP, and finally encapsulation with node IP addresses for VXLAN transport.
Performance Implications and MTU Considerations
Encapsulation mode introduces overhead that affects both packet size and processing. The VXLAN header adds 50 bytes to each packet (8 bytes VXLAN header + 20 bytes outer IP header + 8 bytes UDP header + 14 bytes outer Ethernet header). This overhead reduces the effective MTU for pod traffic:
# Test MTU with different packet sizes
kubectl exec -it curl-pod -- sh -c '
for size in 1422 1450 1451 1472 1473 1500; do
echo -n "Size $size: "
if ping -c 1 -W 1 -s $size -M do 172.20.2.128 >/dev/null 2>&1; then
echo "OK"
else
echo "Failed"
fi
done
'
Size 1422: OK
Size 1450: OK
Size 1451: Failed
Size 1472: Failed
Size 1473: Failed
Size 1500: FailedThe maximum working payload size of 1450 bytes aligns with the MTU setting we observed in the routing table. This reduction in effective MTU can impact application performance, particularly for bulk data transfers where larger packets would be more efficient.
CPU overhead is another consideration. Each packet requires encapsulation at the source node and decapsulation at the destination node. Modern CPUs with hardware offloading capabilities can mitigate this overhead:
# Check for VXLAN hardware offload support
ethtool -k eth1 | grep vxlan
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
rx-udp_tnl-segmentation: off [fixed]In environments without hardware offload support, the CPU impact can be measured:
# Monitor CPU usage during traffic generation
kubectl exec -it curl-pod -- sh -c 'ab -n 10000 -c 10 http://webpod/' &
top -b -n 5 -d 2 | grep -E "Cpu|cilium"Hubble Observability in Different Routing Modes
Hubble, Cilium’s observability platform, provides insights into network flows regardless of the routing mode. However, the information captured differs slightly between modes:
# Start Hubble forwarding
cilium hubble port-forward &
# Observe flows in encapsulation mode
hubble observe --protocol tcp --from-pod default/curl-pod --last 10
Aug 9 06:37:06.637: default/curl-pod:46181 -> 10.96.71.46:80 to-proxy FORWARDED (TCP Flags: SYN)
Aug 9 06:37:06.637: default/curl-pod:46181 <> default/webpod-697b545f57-r5lfw:38544 post-xlate-fwd TRANSLATED
Aug 9 06:37:06.639: default/curl-pod:51902 -> default/webpod-697b545f57-r5lfw:80 to-overlay FORWARDED (TCP Flags: SYN)
Aug 9 06:37:06.641: default/curl-pod:51902 <- default/webpod-697b545f57-r5lfw:80 to-endpoint FORWARDED (TCP Flags: SYN-ACK)The “to-overlay” verdict indicates that the packet is being sent through the VXLAN overlay network, while in native routing mode, you would see “to-network” instead.
For more detailed analysis, we can use Hubble’s metrics:
# Get flow metrics by verdict
hubble metrics flow --verdict FORWARDED --protocol tcp
VERDICT PROTOCOL SOURCE DESTINATION COUNT
FORWARDED tcp default/curl-pod default/webpod-697b545f57-9pg9x 142
FORWARDED tcp default/curl-pod default/webpod-697b545f57-r5lfw 89
FORWARDED tcp default/curl-pod default/webpod-697b545f57-wpsqb 234When debugging connectivity issues across routing modes, several advanced techniques prove invaluable. First, examine the BPF program statistics:
# Check BPF program attachment points
kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- cilium bpf metrics list
Program Direction Interface Packets Bytes Errors
bpf_lxc.o Ingress lxc_health 523421 42365891 0
bpf_lxc.o Egress lxc_health 498322 40128374 0
bpf_overlay.o Ingress cilium_vxlan 892341 823421893 0
bpf_overlay.o Egress cilium_vxlan 901237 834219384 0For packet-level debugging, use the datapath trace functionality:
# Trace a specific connection
kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- \
cilium bpf trace --src 172.20.0.170 --dst 172.20.2.128 --dport 80
Tracing policy verdict for 172.20.0.170 -> 172.20.2.128:80
-> Egress: ALLOWED by policy 38544
-> Encapsulation: VXLAN to node 192.168.20.100
-> Transmitted via cilium_vxlanConnection tracking states provide insights into established flows:
# View connection tracking entries
kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- \
cilium bpf ct list global | grep 172.20
TCP IN 172.20.0.170:54892 -> 172.20.2.128:80 lifetime=119829 RxPackets=24 RxBytes=1823 TxPackets=31 TxBytes=28934
TCP OUT 172.20.2.128:80 -> 172.20.0.170:54892 lifetime=119829 RxPackets=31 RxBytes=28934 TxPackets=24 TxBytes=1823When deploying Cilium in production environments with multiple network segments, several factors warrant careful consideration. The choice between native routing and encapsulation modes significantly impacts operational complexity, performance characteristics, and troubleshooting approaches.
For environments where all intermediate network infrastructure can be controlled and configured with appropriate routes, native routing mode offers superior performance and simplified packet inspection. This approach works well in private cloud environments or on-premises deployments where the network team can implement dynamic routing protocols like BGP to automatically distribute pod CIDR routes.
Conversely, encapsulation mode provides better portability and reduced operational overhead in environments where modifying intermediate network infrastructure is impractical or impossible. Public cloud environments, multi-tenant scenarios, or deployments spanning multiple data centers often benefit from encapsulation mode’s ability to abstract away underlying network complexity.
The transition between routing modes requires careful planning. While Cilium supports dynamic reconfiguration, the change causes temporary disruption to established connections. In production environments, this transition should be performed during maintenance windows with appropriate rollback procedures prepared.
Network policy enforcement continues to function regardless of the routing mode selected. Cilium’s identity-based security model ensures that policies are enforced at the source and destination nodes, whether packets traverse the network natively or through encapsulation. However, the visibility of traffic to external security appliances differs significantly between modes, with native routing providing complete visibility while encapsulation hides pod-level details from intermediate devices.
Performance monitoring becomes crucial in production deployments. Key metrics to track include packet loss rates across network boundaries, latency variations between same-network and cross-network pod communication, CPU utilization on nodes performing encapsulation/decapsulation, and throughput degradation due to MTU reduction in encapsulation mode.
Regular testing of failover scenarios ensures resilience. This includes validating behavior when a node in a different network segment becomes unreachable, testing recovery time when routing information changes, and verifying that health checks correctly detect and route around failures.
The integration with existing monitoring and logging infrastructure requires attention to the routing mode. In native routing mode, traditional network monitoring tools can observe pod traffic directly, while encapsulation mode requires either overlay-aware monitoring tools or instrumentation at the Cilium layer using Hubble.
Cilium Service LB-IPAM and L2 Announcements
One of the most significant gaps between public cloud and on-premises Kubernetes deployments is the availability of native load balancer services. In public cloud environments, creating a LoadBalancer type service triggers cloud provider integrations that automatically provision external load balancers, assign public IP addresses, and configure the necessary routing. However, in private cloud or bare-metal environments, this luxury doesn’t exist by default. Services of type LoadBalancer remain in a perpetual “pending” state for external IP assignment, forcing operators to rely on NodePort services or deploy additional components like MetalLB. Cilium’s LB-IPAM (LoadBalancer IP Address Management) combined with L2 Announcements provides a comprehensive solution to this challenge, bringing cloud-like load balancing capabilities to any environment.
Understanding the LoadBalancer Service Challenge in Private Clouds
To fully appreciate the significance of LB-IPAM, we must first understand the fundamental networking challenge it addresses. In Kubernetes, services abstract away the complexity of pod networking by providing stable endpoints for applications. The LoadBalancer service type represents the highest level of this abstraction, intended to expose services externally with dedicated IP addresses. In public clouds, this process is seamless because cloud providers offer APIs for dynamic load balancer provisioning. Amazon’s Elastic Load Balancer, Google’s Cloud Load Balancing, or Azure’s Load Balancer automatically handle IP allocation, health checking, and traffic distribution.
In private cloud environments, however, there’s no such automated infrastructure. When you create a LoadBalancer service, Kubernetes has no mechanism to fulfill the external IP request. The service controller expects an external system to provide this IP and configure the necessary network paths. Without this integration, operators are forced to use less elegant solutions like NodePort services, which require clients to connect to high-numbered ports on node IPs, or to manually configure external load balancers like HAProxy or NGINX, adding operational complexity and potential points of failure.
The networking challenge extends beyond simple IP allocation. Once an IP is assigned to a service, the network infrastructure must know how to route traffic to that IP. In Layer 2 networks, this requires ARP (Address Resolution Protocol) responses to associate the IP with a MAC address. In Layer 3 networks, routing protocols like BGP must advertise the IP’s location. Without these mechanisms, the assigned IP remains unreachable from outside the cluster.
LB-IPAM Architecture and IP Pool Management
Cilium’s LB-IPAM introduces a sophisticated IP address management system specifically designed for LoadBalancer services in private environments. The architecture consists of several key components working in concert. At its core, LB-IPAM manages pools of IP addresses that can be dynamically assigned to services. These pools are defined through CiliumLoadBalancerIPPool custom resources, which specify ranges of available addresses and optional selection criteria.
Let’s explore the IP pool creation and management process in detail:
# Create a comprehensive IP pool with multiple ranges
cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
name: "production-lb-pool"
spec:
blocks:
- start: "192.168.10.211"
stop: "192.168.10.215"
- start: "192.168.10.220"
stop: "192.168.10.225"
serviceSelector:
matchExpressions:
- key: "environment"
operator: In
values: ["production", "staging"]
EOFThe IP pool specification supports sophisticated selection mechanisms. You can create multiple pools for different purposes, such as separating production and development services, or allocating specific ranges for different application tiers. The controller watches for LoadBalancer service creation and automatically assigns IPs from matching pools:
# View all IP pools and their utilization
kubectl get ippools -o wide
NAME DISABLED CONFLICTING IPS AVAILABLE IPS USED AGE
cilium-lb-ippool false False 5 0 2m
production-lb-pool false False 11 0 30s
# Create a service and observe IP allocation
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: webpod
labels:
app: webpod
environment: production
spec:
type: LoadBalancer
selector:
app: webpod
ports:
- port: 80
targetPort: 80
EOF
# Check the allocated IP
kubectl get svc webpod -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
192.168.10.211
# Verify pool usage updated
kubectl get ippools production-lb-pool -o jsonpath='{.status}' | jq
{
"allocations": {
"192.168.10.211": {
"namespace": "default",
"service": "webpod"
}
},
"conditions": [
{
"type": "Ready",
"status": "True",
"lastTransitionTime": "2024-08-09T10:00:00Z"
}
]
}The allocation algorithm considers several factors when assigning IPs. First, it checks service selectors against pool selectors to find matching pools. Then, it selects the first available IP from the matched pool, maintaining a consistent allocation pattern. The system also supports IP reservation and specific IP requests through annotations, providing flexibility for services requiring predetermined addresses.
L2 Announcements
With IPs allocated through LB-IPAM, the next challenge is making these IPs reachable from the local network. Cilium’s L2 Announcements feature solves this by implementing a virtual IP (VIP) mechanism using ARP responses. This approach is similar to traditional high-availability solutions like Keepalived or VRRP, but integrated directly into the Kubernetes networking layer.
The L2 announcement mechanism works by having Cilium agents respond to ARP requests for LoadBalancer IPs. When a client in the same Layer 2 network tries to reach a LoadBalancer IP, it broadcasts an ARP request asking “who has this IP?” The Cilium agent on the designated leader node responds with its MAC address, causing traffic to flow to that node, which then performs load balancing to the actual service endpoints.
Let’s enable and configure L2 announcements with comprehensive monitoring:
# Enable L2 announcements in Cilium
helm upgrade cilium cilium/cilium --namespace kube-system --version 1.18.0 --reuse-values \
--set l2announcements.enabled=true \
--set l2announcements.leaseDuration=3s \
--set l2announcements.leaseRenewDeadline=1s \
--set l2announcements.leaseRetryPeriod=500ms
# Restart Cilium pods to apply configuration
kubectl rollout restart -n kube-system ds/cilium
# Verify the configuration
cilium config view | grep -E "enable-l2|lease"
enable-l2-announcements true
enable-l2-neigh-discovery true
l2-announcements-lease-duration 3s
l2-announcements-lease-renew-deadline 1s
l2-announcements-lease-retry-period 500msNow, create an L2 announcement policy that defines which services should be announced and from which nodes:
# Create a comprehensive L2 announcement policy
cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
name: production-l2-policy
spec:
serviceSelector:
matchExpressions:
- key: tier
operator: In
values: ["frontend", "api"]
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/worker
operator: Exists
- key: kubernetes.io/hostname
operator: NotIn
values: ["k8s-w0"] # Exclude nodes in different L2 segments
interfaces:
- ^eth[0-9]+$
- ^ens[0-9]+$
- ^eno[0-9]+$
externalIPs: true
loadBalancerIPs: true
EOFThe policy specification provides fine-grained control over announcement behavior. The interface patterns ensure announcements occur on the correct network interfaces, crucial in multi-homed systems. Let’s observe the L2 announcement mechanism in action:
# Identify the leader node for our service
kubectl -n kube-system get lease cilium-l2announce-default-webpod -o jsonpath='{.spec.holderIdentity}'
k8s-w1
# On the leader node, verify the announcement database
CILIUMPOD1=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w1 -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n kube-system $CILIUMPOD1 -- cilium-dbg status --all-controllers | grep -A5 l2-announce
l2-announce-responder
Status: Running
Entries: 1
Last announced: 2s ago
Announcements sent: 145
Conflicts detected: 0
# View the actual announcement entries
kubectl exec -n kube-system $CILIUMPOD1 -- cilium-dbg shell -- db/show l2-announce
IP NetworkInterface MAC State
192.168.10.211 eth1 08:00:27:eb:84:f3 ActiveFrom outside the cluster, we can now observe ARP behavior:
# From the router VM, clear ARP cache and test discovery
arp -d 192.168.10.211 2>/dev/null
arping -I eth1 192.168.10.211 -c 3
ARPING 192.168.10.211 from 192.168.10.200 eth1
Unicast reply from 192.168.10.211 [08:00:27:eb:84:f3] 0.648ms
Unicast reply from 192.168.10.211 [08:00:27:eb:84:f3] 0.523ms
Unicast reply from 192.168.10.211 [08:00:27:eb:84:f3] 0.498ms
Sent 3 probes (1 broadcast(s))
Received 3 response(s)# Verify ARP table entry
arp -n | grep 192.168.10.211
192.168.10.211 ether 08:00:27:eb:84:f3 C eth1# Test actual service connectivity
curl -v http://192.168.10.211
* Connected to 192.168.10.211 (192.168.10.211) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.10.211
< HTTP/1.1 200 OK
Hostname: webpod-697b545f57-wpsqb
Leader Election and High Availability
One of the most critical aspects of L2 announcements is the leader election mechanism that ensures high availability. Since only one node can respond to ARP requests for a given IP (to avoid network confusion), Cilium implements a distributed leader election using Kubernetes leases. This mechanism provides automatic failover when the leader node becomes unavailable.
Let’s examine the leader election process in detail.
# Monitor all L2 announcement leases
watch -n 1 'kubectl -n kube-system get lease | grep cilium-l2announce'
NAME HOLDER AGE
cilium-l2announce-default-webpod k8s-w1 5m23s
cilium-l2announce-default-netshoot k8s-ctr 2m15s
# Get detailed lease information
kubectl -n kube-system get lease cilium-l2announce-default-webpod -o yaml
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
name: cilium-l2announce-default-webpod
namespace: kube-system
spec:
holderIdentity: k8s-w1
leaseDurationSeconds: 3
acquireTime: "2024-08-09T10:15:23.123456Z"
renewTime: "2024-08-09T10:20:45.654321Z"
leaseTransitions: 2The lease mechanism uses configurable durations for determining leader liveliness. Let’s test failover behavior by simulating node failure:
# Start continuous monitoring from outside the cluster
# Terminal 1 - Monitor service availability
while true; do
response=$(curl -s -w "\n%{http_code} %{time_total}s" http://192.168.10.211 2>&1)
echo "$(date '+%H:%M:%S.%3N') - $response" | head -1
sleep 0.5
done
# Terminal 2 - Monitor ARP changes
while true; do
arp_entry=$(arp -n | grep 192.168.10.211 | awk '{print $3}')
echo "$(date '+%H:%M:%S.%3N') - MAC: $arp_entry"
sleep 1
done
# Terminal 3 - Monitor lease holder
while true; do
holder=$(kubectl -n kube-system get lease cilium-l2announce-default-webpod -o jsonpath='{.spec.holderIdentity}' 2>/dev/null)
echo "$(date '+%H:%M:%S.%3N') - Leader: $holder"
sleep 1
done
# Now simulate node failure by stopping kubelet on the leader node
ssh k8s-w1 'sudo systemctl stop kubelet'During failover, you’ll observe:
10:25:00.123 - Hostname: webpod-697b545f57-9pg9x
10:25:00.623 - Hostname: webpod-697b545f57-wpsqb
10:25:01.124 - Hostname: webpod-697b545f57-r5lfw
# Leader failure detected
10:25:03.625 - curl: (7) Failed to connect to 192.168.10.211 port 80: No route to host
10:25:04.126 - curl: (7) Failed to connect to 192.168.10.211 port 80: No route to host
# New leader elected, gratuitous ARP sent
10:25:05.627 - MAC: 08:00:27:b8:d6:9b # Changed from 08:00:27:eb:84:f3
10:25:06.128 - Hostname: webpod-697b545f57-9pg9x
10:25:06.629 - Leader: k8s-ctrThe failover process typically completes within 3–5 seconds, depending on lease configuration. The new leader immediately sends gratuitous ARP announcements to update network devices with the new MAC address association.
Advanced Traffic Flow Analysis
Understanding the actual packet flow in L2 announcement mode reveals important architectural considerations. When external traffic arrives at the leader node, it undergoes several stages of processing:
# Enable detailed packet tracing on the leader node
LEADER_NODE=$(kubectl -n kube-system get lease cilium-l2announce-default-webpod -o jsonpath='{.spec.holderIdentity}')
LEADER_POD=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=$LEADER_NODE -o jsonpath='{.items[0].metadata.name}')
# Start BPF tracing
kubectl exec -n kube-system $LEADER_POD -- cilium monitor --type trace --type drop --type capture
# Generate traffic from router
curl http://192.168.10.211
# Trace output analysis
<- host flow 0x0 identity world->reserved:world state new ifindex eth1 orig-ip 192.168.10.200:54234 -> 192.168.10.211:80 tcp SYN
-> endpoint 2314 flow 0x0 identity reserved:world->38544 state established ifindex lxc123 orig-ip 192.168.10.200:54234 -> 172.20.1.18:80 tcp SYN
<- endpoint 2314 flow 0x0 identity 38544->reserved:world state reply ifindex lxc123 orig-ip 172.20.1.18:80 -> 192.168.10.200:54234 tcp SYN-ACK
-> host flow 0x0 identity 38544->world state reply ifindex eth1 orig-ip 192.168.10.211:80 -> 192.168.10.200:54234 tcp SYN-ACKThe trace reveals that incoming packets undergo DNAT (Destination NAT) from the LoadBalancer IP to the actual pod IP. For local pods, this translation happens in the kernel’s netfilter stack. For remote pods, an additional step occurs:
# When traffic goes to a pod on a different node
# Capture on the leader node
tcpdump -i any -nn 'host 192.168.10.211 or host 172.20.2.128' -c 20
15:30:45.123456 IP 192.168.10.200.54234 > 192.168.10.211.80: Flags [S], seq 1234567890
15:30:45.123478 IP 192.168.10.101.54234 > 172.20.2.128.80: Flags [S], seq 1234567890 # SNAT to node IP
15:30:45.124567 IP 172.20.2.128.80 > 192.168.10.101.54234: Flags [S.], seq 987654321, ack 1234567891
15:30:45.124589 IP 192.168.10.211.80 > 192.168.10.200.54234: Flags [S.], seq 987654321, ack 1234567891This reveals an important limitation: when the leader node forwards traffic to pods on other nodes, it must perform SNAT (Source NAT) to ensure return traffic comes back through the leader. This means pods see the leader node’s IP as the source, not the actual client IP, which can impact logging and security policies.
Multiple Services and Resource Optimization
In production environments, you’ll typically run multiple LoadBalancer services. L2 announcements handle this elegantly by electing separate leaders for each service, distributing the load:
# Deploy multiple services
for i in {1..5}; do
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-$i
spec:
replicas: 2
selector:
matchLabels:
app: app-$i
template:
metadata:
labels:
app: app-$i
spec:
containers:
- name: app
image: nginxdemos/hello
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: app-$i-lb
labels:
app: app-$i
spec:
type: LoadBalancer
selector:
app: app-$i
ports:
- port: 80
targetPort: 80
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
name: policy-app-$i
spec:
serviceSelector:
matchLabels:
app: app-$i
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: "true"
interfaces:
- eth1
loadBalancerIPs: true
EOF
done
# Check leader distribution
kubectl -n kube-system get lease | grep cilium-l2announce | awk '{print $1, $2}' | column -t
cilium-l2announce-default-app-1-lb k8s-w1
cilium-l2announce-default-app-2-lb k8s-ctr
cilium-l2announce-default-app-3-lb k8s-w1
cilium-l2announce-default-app-4-lb k8s-ctr
cilium-l2announce-default-app-5-lb k8s-w1The leader distribution algorithm attempts to balance services across available nodes, though it doesn’t guarantee perfect distribution. For more control, you can influence leader selection through node selectors in the L2 announcement policy.
Advanced IP Management
Sometimes services require specific IP addresses for compatibility with existing systems or DNS configurations. LB-IPAM supports explicit IP requests through annotations:
# Request a specific IP for a service
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: legacy-app
annotations:
"lbipam.cilium.io/ips": "192.168.10.215"
spec:
type: LoadBalancer
selector:
app: legacy
ports:
- port: 80
targetPort: 8080
EOF
# Verify the assigned IP
kubectl get svc legacy-app -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
192.168.10.215
# Check pool allocation
kubectl get ippool cilium-lb-ippool -o jsonpath='{.status.allocations}' | jq
{
"192.168.10.211": {
"namespace": "default",
"service": "webpod"
},
"192.168.10.215": {
"namespace": "default",
"service": "legacy-app"
}
}If the requested IP is unavailable or outside any configured pool, the service will remain pending with an event explaining the issue:
# Request an invalid IP
kubectl patch svc legacy-app -p '{"metadata":{"annotations":{"lbipam.cilium.io/ips":"10.0.0.1"}}}'
# Check events
kubectl describe svc legacy-app | tail -5
Events:
Type Reason Age From Message
---- ------ ---- ---- --------
Warning AllocationFailed 5s cilium-operator IP 10.0.0.1 not available in any pool
Normal IPAMRequestUnsatisfied 5s cilium-lbipam Failed to satisfy IP requestIP Sharing with Port Multiplexing
In environments with limited IP addresses, LB-IPAM supports sharing IPs across multiple services using different ports. This is accomplished through the sharing-key annotation:
# Create multiple services sharing the same IP
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: web-frontend
annotations:
"lbipam.cilium.io/sharing-key": "shared-web"
spec:
type: LoadBalancer
selector:
app: frontend
ports:
- name: http
port: 80
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: web-api
annotations:
"lbipam.cilium.io/sharing-key": "shared-web"
"lbipam.cilium.io/ips": "192.168.10.220"
spec:
type: LoadBalancer
selector:
app: api
ports:
- name: api
port: 8080
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: web-admin
annotations:
"lbipam.cilium.io/sharing-key": "shared-web"
"lbipam.cilium.io/ips": "192.168.10.220"
spec:
type: LoadBalancer
selector:
app: admin
ports:
- name: admin
port: 9090
targetPort: 9090
EOF
# Verify all services share the same IP
kubectl get svc -l 'app in (frontend,api,admin)' -o custom-columns=NAME:.metadata.name,IP:.status.loadBalancer.ingress[0].ip,PORT:.spec.ports[0].port
NAME IP PORT
web-frontend 192.168.10.220 80
web-api 192.168.10.220 8080
web-admin 192.168.10.220 9090
# Test each service
for port in 80 8080 9090; do
echo "Testing port $port:"
curl -s http://192.168.10.220:$port | head -2
doneServices sharing an IP through sharing-key must use the same L2 announcement leader, ensuring consistent ARP responses:
# Verify shared services use the same leader
kubectl -n kube-system get lease | grep -E 'web-frontend|web-api|web-admin'
cilium-l2announce-default-web-frontend k8s-ctr 2m
cilium-l2announce-default-web-api k8s-ctr 2m
cilium-l2announce-default-web-admin k8s-ctr 2mPerformance Monitoring and Optimization
In production deployments, monitoring the performance of LB-IPAM and L2 announcements is crucial. Key metrics include ARP response latency, failover time, and traffic distribution:
# Create a monitoring script for ARP performance
cat << 'EOF' > /tmp/monitor-arp.sh
#!/bin/bash
IP=$1
INTERFACE=$2
echo "Monitoring ARP for $IP on $INTERFACE"
while true; do
start=$(date +%s%N)
timeout 1 arping -c 1 -I $INTERFACE $IP > /dev/null 2>&1
end=$(date +%s%N)
latency=$(((end-start)/1000000))
echo "$(date '+%Y-%m-%d %H:%M:%S') - ARP latency: ${latency}ms"
sleep 5
done
EOF
chmod +x /tmp/monitor-arp.sh
/tmp/monitor-arp.sh 192.168.10.211 eth1 &
# Monitor Cilium's internal metrics
kubectl exec -n kube-system $CILIUMPOD1 -- cilium metrics list | grep l2_announce
cilium_l2_announce_entries_total{node="k8s-w1"} 3
cilium_l2_announce_responses_sent_total{node="k8s-w1"} 1847
cilium_l2_announce_conflicts_detected_total{node="k8s-w1"} 0
cilium_l2_announce_leader_elections_total{node="k8s-w1"} 5
cilium_l2_announce_leader_election_attempts_total{node="k8s-w1"} 7For production environments, integrate these metrics with your monitoring stack:
# Prometheus query examples for monitoring L2 announcements
# Rate of ARP responses
rate(cilium_l2_announce_responses_sent_total[5m])# Leader election stability
increase(cilium_l2_announce_leader_elections_total[1h])# IP pool utilization
(cilium_ipam_ips_allocated / cilium_ipam_ips_total) * 100
When L2 announcements don’t work as expected, systematic troubleshooting helps identify the root cause.
# 1. Verify L2 announcements are enabled
cilium config view | grep enable-l2-announcements
# Should return: enable-l2-announcements: true
# 2. Check if policies are correctly applied
kubectl get ciliuml2announcementpolicies -o wide
NAME SERVICE-SELECTOR NODE-SELECTOR INTERFACES
policy1 app=webpod kubernetes.io/hostname!=k8s-w0 ^eth[1-9]+
# 3. Verify lease acquisition
kubectl -n kube-system describe lease cilium-l2announce-default-webpod
# Look for:
# - Holder Identity: should have a node name
# - Renew Time: should be recent (within lease duration)
# 4. Check Cilium agent logs for L2 announcement issues
kubectl logs -n kube-system $CILIUMPOD1 cilium-agent | grep -i "l2.*announce"
level=info msg="L2 announcement initialized" subsys=l2-announcer
level=info msg="Elected as leader for service" service=default/webpod subsys=l2-announcer
level=info msg="Starting ARP responder" ip=192.168.10.211 interface=eth1 subsys=l2-announcer
# 5. Verify network interface configuration
kubectl exec -n kube-system $CILIUMPOD1 -- ip link show eth1
# Should show UP state and correct MTU
# 6. Test ARP resolution from different network locations
# From same L2 segment
arping -I eth1 192.168.10.211
# Should get responses
# From different L2 segment (should fail)
ssh k8s-w0 arping -I eth1 192.168.10.211
# Should timeout - this is expected as L2 announcements only work within the same broadcast domainMigration Strategies from NodePort to LoadBalancer
Organizations often start with NodePort services and later migrate to LoadBalancer services with LB-IPAM. Here’s a zero-downtime migration strategy:
# Existing NodePort service
kubectl get svc existing-app -o yaml > existing-app-backup.yaml
kubectl get svc existing-app
NAME TYPE CLUSTER-IP PORT(S) AGE
existing-app NodePort 10.96.100.50 80:30080/TCP 30d
# Step 1: Create a temporary LoadBalancer service pointing to the same pods
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: existing-app-lb
annotations:
"lbipam.cilium.io/ips": "192.168.10.225"
spec:
type: LoadBalancer
selector:
app: existing-app # Same selector as original service
ports:
- port: 80
targetPort: 8080
EOF
# Step 2: Test the new LoadBalancer service
curl http://192.168.10.225
# Verify it works correctly
# Step 3: Update DNS/configuration to point to new IP
# This is application-specific
# Step 4: Monitor traffic shift
# Watch connection counts on both services
watch -n 1 'kubectl get svc existing-app existing-app-lb -o custom-columns=NAME:.metadata.name,ENDPOINTS:.spec.clusterIP | xargs -I {} sh -c "echo {}; curl -s {}/metrics | grep http_requests_total"'
# Step 5: Once traffic has shifted, delete the old NodePort service
kubectl delete svc existing-app
# Step 6: Rename the LoadBalancer service to the original name
kubectl get svc existing-app-lb -o yaml | sed 's/existing-app-lb/existing-app/g' | kubectl apply -f -
kubectl delete svc existing-app-lbLB-IPAM and L2 announcements often need to integrate with existing DNS infrastructure. Dynamic DNS updates can be automated using external-dns or custom controllers:
# Deploy external-dns for automatic DNS updates
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
namespace: kube-system
spec:
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: k8s.gcr.io/external-dns/external-dns:v0.13.5
args:
- --source=service
- --source=ingress
- --domain-filter=example.com
- --provider=rfc2136
- --rfc2136-host=192.168.10.53
- --rfc2136-port=53
- --rfc2136-zone=example.com
- --rfc2136-tsig-secret=abcdefgh
- --rfc2136-tsig-secret-alg=hmac-sha256
- --rfc2136-tsig-keyname=externaldns-key
- --txt-owner-id=k8s-cluster-1
env:
- name: EXTERNAL_DNS_RFC2136_TSIG_SECRET
valueFrom:
secretKeyRef:
name: external-dns
key: rfc2136-tsig-secret
EOF
# Annotate services for DNS registration
kubectl annotate svc webpod external-dns.alpha.kubernetes.io/hostname=webpod.example.com
kubectl annotate svc webpod external-dns.alpha.kubernetes.io/ttl=60
# Verify DNS updates
dig @192.168.10.53 webpod.example.com
; <<>> DiG 9.16.1 <<>> @192.168.10.53 webpod.example.com
;; ANSWER SECTION:
webpod.example.com. 60 IN A 192.168.10.211While L2 announcements provide basic availability through leader election, production deployments often require more sophisticated health checking:
# Implement custom health checking for L2 announced services
cat << 'EOF' > /tmp/health-monitor.sh
#!/bin/bash
SERVICE_IP=$1
HEALTH_ENDPOINT="${2:-/healthz}"
THRESHOLD="${3:-3}"
failures=0
while true; do
if curl -sf http://${SERVICE_IP}${HEALTH_ENDPOINT} > /dev/null; then
failures=0
echo "$(date '+%Y-%m-%d %H:%M:%S') - Service healthy"
else
((failures++))
echo "$(date '+%Y-%m-%d %H:%M:%S') - Health check failed ($failures/$THRESHOLD)"
if [ $failures -ge $THRESHOLD ]; then
echo "$(date '+%Y-%m-%d %H:%M:%S') - Triggering failover"
# Force leader election by deleting the lease
kubectl -n kube-system delete lease cilium-l2announce-default-webpod
failures=0
sleep 10
fi
fi
sleep 2
done
EOF
chmod +x /tmp/health-monitor.sh
/tmp/health-monitor.sh 192.168.10.211 /healthz 3 &This approach provides application-aware health checking beyond simple node availability, ensuring services remain accessible even when applications fail without node failure.
