Why Cilium Matters

Gasida Seo’s Week 1 Study

Concepts

Let me break down why Cilium is such a game-changer for Kubernetes networking, and trust me, once you understand what’s happening under the hood, you’ll see why so many of us are excited about it.

Picture this: You’re running a Kubernetes cluster with a few thousand services. Every time a packet needs to find its way from one pod to another, your traditional CNI is basically doing the networking equivalent of flipping through a phone book. Page by page. Entry by entry. Now imagine doing that millions of times per second. That’s essentially what iptables-based networking has been doing for years, and honestly, it’s kind of insane that we’ve put up with it for so long.

Here’s where Cilium comes in, and why eBPF is such a big deal. Instead of that phone book approach, imagine having instant search — like having Google for your network packets. Type in what you’re looking for, and boom, you’re there. No scrolling, no waiting. That’s the difference between O(n) and O(1) complexity, and in the real world, it means the difference between your network choking at scale versus humming along like it’s no big deal.

But let me back up a bit and explain what eBPF actually is, because it’s the secret sauce that makes all of this possible. You know how normally, if you want to change how Linux handles networking, you’d have to either modify the kernel (good luck with that) or load kernel modules (hello, stability nightmares)? eBPF basically says “forget all that” and gives you a way to inject your own code into the kernel safely. It’s like being able to reprogram your car’s engine while it’s running, without any risk of blowing it up.

The history here is actually pretty cool. Back in 1992, some folks at Berkeley created BPF (Berkeley Packet Filter) for packet filtering — think tcpdump and friends. It was fine for what it was, but pretty limited. Fast forward to 2014, and this engineer named Alexei Starovoitov basically rebuilt the whole thing from scratch, turning it into what we now call eBPF. The “extended” part is kind of an understatement — it’s like calling a smartphone an “extended telephone.”

What makes eBPF special is that it’s paranoid about safety. Before any eBPF program runs in your kernel, a verifier goes through it with a fine-tooth comb. Will this program ever get stuck in an infinite loop? Nope, rejected. Could it access memory it shouldn’t? Rejected. It’s like having a really strict code reviewer who never sleeps and never misses anything. Once your code passes all these checks, it gets compiled into native machine code using JIT compilation, so it runs at basically the same speed as if you’d written it directly into the kernel.

Now, Cilium takes this eBPF superpower and applies it to Kubernetes networking. When a packet shows up at a network interface, traditional CNIs would send it on this long journey through the Linux networking stack — up through various layers, through iptables rules, maybe through some bridge interfaces, and eventually to its destination. Cilium just shortcuts the whole thing. Using XDP (eXpress Data Path), it can grab packets right as they come off the network card and make routing decisions immediately.

The performance difference is honestly ridiculous. I have seen benchmarks on the web where iptables-based solutions max out around 500,000 to 1 million packets per second while burning through CPU cycles like crazy. Cilium with XDP? We’re talking 20 to 50 million packets per second, and your CPU is basically yawning. It’s not even a fair fight.

But here’s what really gets me excited about Cilium — it’s not just about raw speed. The architecture is genuinely elegant. Take IP address management, for instance. Traditional CNIs often use this rigid approach where each node gets a fixed subnet, and that’s it. Run out of IPs… Too bad. Cilium’s cluster-scope IPAM is dynamic. It manages IP addresses across the entire cluster intelligently, and if a node needs more addresses, it can get them. It’s like having a smart thermostat versus an old manual one… it just adapts to what you need.

For moving packets between nodes, Cilium gives you options. You can use encapsulation mode, where it wraps your pod traffic in VXLAN or Geneve tunnels. This is the “it just works” option, no special network configuration needed. Your packets get wrapped up, sent through your regular network, and unwrapped at the destination. However, when you’re adding extra headers to every packet, which means less room for actual data (lower MTU) and a bit of CPU overhead for all that wrapping and unwrapping.

If you want maximum performance and your network can handle it, there’s native routing mode. This is where Cilium gets clever with Linux routing tables. Instead of tunneling, it just tells the kernel “hey, packets for this pod subnet go to that node” and lets the regular routing infrastructure handle it. No wrapping, no overhead, just pure speed. The catch is your underlying network needs to know how to route these pod IPs, which might mean setting up BGP or ensuring all your nodes are on the same L2 segment.

One of the killer features is that Cilium can completely replace kube-proxy. If you’ve ever looked at a large Kubernetes cluster’s iptables rules, you know it’s a mess. Thousands upon thousands of rules that get rebuilt every time a service changes. Cilium throws all of that out and implements service load balancing entirely in eBPF. Service-to-endpoint mappings live in BPF maps where lookups are instant. Even better, it can do socket-level load balancing, which means it picks a backend when a connection is established rather than for every single packet. It’s smarter and way more efficient.

The security story is equally impressive. Because Cilium operates at the kernel level and can inspect traffic at Layer 7, it can implement security policies that actually understand your applications. You’re not just filtering on “port 80”. You can say “allow GET requests to /api/users but block DELETE requests” or “permit Kafka consume operations but not produce operations.” And because it’s all in eBPF, this deep inspection doesn’t tank your performance like traditional solutions would.

Hubble, Cilium’s observability platform, is the cherry on top. Instead of needing to sample packets or run expensive packet captures, Hubble gets flow information directly from the eBPF programs that are already processing your traffic. You get complete visibility into what’s happening in your cluster — who’s talking to whom, what protocols they’re using, where the latencies are — all with minimal overhead. The UI is actually pretty slick too, giving you this real-time map of your services and their interactions.

What really strikes me about Cilium is how it enables architectural patterns that just weren’t practical before. With traditional networking overhead, you’d often colocate services just to avoid network hops, or implement caching layers to reduce service calls. With Cilium’s efficiency, the network becomes basically transparent. You can structure your services based on what makes logical sense, not what minimizes network overhead.

Looking forward, the implications are huge. Hardware vendors are starting to support eBPF offload to smart NICs, which means the same eBPF programs running in your kernel today could run directly on network hardware tomorrow. Imagine getting that 50 million packets per second performance with zero CPU usage. It’s not science fiction — it’s actively being developed.

Here’s the thing: Cilium isn’t just solving today’s problems. It’s building on a foundation — eBPF — that’s fundamentally more capable than what came before. As Kubernetes clusters get larger and applications get more complex, the old ways of doing things simply won’t cut it. You can’t just keep adding more iptables rules and hope for the best. You need something that maintains consistent performance whether you have 10 services or 10,000.

Some Backgrounds

Let me walk you through the foundational concepts you need to understand before diving into Cilium. When Kubernetes was designed, the creators laid out four fundamental networking requirements that every cluster must solve. These aren’t suggestions. They’re hard requirements that make Kubernetes work.

• First, containers within the same pod need to communicate as if they’re on the same machine. This happens through a clever trick. All containers in a pod share the same network namespace. When kubelet starts a pod, it first creates a “pause container” that does nothing but hold the network namespace open. Every other container in that pod then joins this namespace, allowing them to talk to each other over localhost. Simple, elegant, and it just works.
• Second, every pod needs its own unique IP address that’s reachable from any other pod in the cluster. This is where things get interesting. Your nodes might be on a 192.168.x.x network, but your pods might use 10.244.x.x addresses. Somehow, a pod on node A needs to send packets to a pod on node B, even though they’re on completely different subnets. This is the core problem that CNI plugins solve.
• Third, services need to work. When you create a Kubernetes service, it gets a virtual IP that doesn’t actually exist on any machine. Traffic sent to this IP needs to magically find its way to the right pods. Traditionally, kube-proxy handles this with iptables magic, but as we’ll see, Cilium has other ideas.
• Finally, external traffic needs a way in. LoadBalancer services need to actually work, which usually involves cloud provider integration or specialized controllers.

CNI (Container Network Interface) is Kubernetes’ answer to the second requirement which is pod-to-pod networking. Rather than baking one networking solution into Kubernetes, the creators defined an interface and said “implement this however you want.” This brilliant decision led to an explosion of networking plugins…

• Flannel keeps it simple with basic overlay networking.
• Calico brings BGP and network policies.
• AWS VPC CNI integrates directly with Amazon’s network.
• and Cilium… well, Cilium rewrites the rules entirely.

When a pod starts, the container runtime calls the CNI plugin binary with some JSON configuration. The plugin’s job is to set up networking for that pod (e.g. assign an IP, create network interfaces, set up routes, whatever it takes.) When the pod stops, the runtime calls the plugin again to clean everything up.

Now we get to the secret sauce that makes Cilium special — eBPF. To understand why eBPF is such a big deal, you need to understand the problem it solves.

In traditional Linux, there’s a hard boundary between user space (where your applications run) and kernel space (where the operating system lives). This boundary exists for good reasons, e.g., you don’t want random applications crashing your kernel. But it also creates a performance problem. Every time your application needs to do something with the network, it has to ask the kernel through system calls.

Look at that simple HTTP request in the code example. Behind the scenes, it’s making multiple system calls: socket() to create a socket, connect() to establish the connection, write() to send data. Each system call is a context switch between user and kernel space. It adds up.

For years, if you wanted to change how Linux handled networking, you had two terrible options.

• Modify the kernel source (good luck getting that upstream)
• Write kernel modules (hope you don’t crash the system)

The networking stack itself became this massive pile of code that started in the modem era and kept growing.

eBPF changes everything. It lets you run custom programs inside the kernel, safely. The “safely” part is crucial — before your eBPF program runs, a verifier analyzes every possible execution path. It checks that your program will terminate (no infinite loops), that all memory accesses are safe, and that you’re not doing anything that could crash the kernel. If it passes, your program gets JIT-compiled to native code and runs at kernel speed.

The key insight is that eBPF provides hooks throughout the kernel where you can attach your programs. Want to inspect every packet as it arrives? There’s a hook for that (XDP). Want to make routing decisions? There’s a hook (TC). Want to trace function calls? Hooks everywhere. And in Cilium’s case, those XDP and TC hooks are where the magic happens.

Traditional CNI plugins set up networking using the kernel’s existing mechanisms, like creating virtual interfaces, setting up routes, maybe some iptables rules. It works, but you’re still limited by the kernel’s networking stack.

Cilium says “what if we just… didn’t do that?” Instead of working within the kernel’s networking limitations, it uses eBPF to implement its own data plane. When a packet arrives, Cilium’s eBPF programs can inspect it, make routing decisions, apply security policies, and send it on its way — all without going through the traditional networking stack.

The performance implications are staggering. Where traditional solutions might handle a million packets per second, Cilium with XDP can handle 20–50 million. Where iptables rules create linear slowdowns, eBPF maps provide constant-time lookups.

Getting to Know Cilium

Cilium is what happens when you take eBPF that kernel programming framework we just geeked out about and use it to completely reimagine container networking. It’s a CNI provider, sure, but calling Cilium “just another CNI” is like calling a Tesla “just another car.” Technically true, but you’re missing the whole point.

Here’s the backstory. Cilium is the brainchild of Isovalent, a company that’s basically the eBPF dream team. These folks don’t just use eBPF; they’re actively pushing the boundaries of what’s possible with it. And Cilium? It’s their flagship product, now a CNCF graduated project sitting alongside Kubernetes itself.

Let me break down what Cilium actually does, starting with the basics and working up to the mind-blowing stuff.

IP Address Management (IPAM)

Every CNI needs to hand out IP addresses to pods, that’s table stakes. But Cilium’s approach to IPAM shows the attention to detail that runs through the entire project.

In cluster scope mode (the default), Cilium uses CiliumNode custom resources to manage IP allocation. The operator creates these automatically, and they define which IP ranges each node can use for its pods. What’s clever here is that it’s dynamic. If a node needs more IPs, Cilium can allocate additional ranges on the fly. Try doing that with traditional CNIs where each node gets a fixed /24 and that’s it.

There’s also Kubernetes host scope mode, where Cilium defers to Kubernetes’ own IP allocation. This is great for compatibility. If you’re migrating from another CNI or have specific requirements around how IPs are assigned, Cilium can play along. Just make sure your kube-controller-manager has --allocate-node-cidrs enabled, or you can even use node annotations to manually specify ranges.

Pro tip: Whatever mode you choose, don’t try to change IP pools after the fact. Just add new ranges instead. Cilium reserves a couple of addresses per range for the node itself and broadcast, so plan accordingly. Anything smaller than a /29 is asking for trouble.

Traffic Routing: Two Modes, One Goal

Here’s where Cilium starts to flex. For moving packets between nodes, you get two options, and they represent fundamentally different philosophies.

Encapsulation mode is the default, and it’s the “just works” option. Cilium wraps your pod traffic in VXLAN or Geneve tunnels, creating an overlay network on top of whatever physical network you’ve got. It’s like having a private highway system for your pods, they don’t need to know or care about the underlying infrastructure.

So, you’re running a Kubernetes cluster. You’ve got servers with their own IP addresses, but your pods live in a completely different world, with their own separate IP range. How does a packet from a pod on one server find its way to a pod on another when the physical network hardware — the routers and switches — has no clue what a “pod” even is?

The answer is a clever bit of networking magic called an overlay network.

Think of it this way. Your physical server network is like the public road system. The routers are the intersections that know how to direct traffic between major street addresses (your server IPs). An overlay network is like building a private, high-speed tunnel system that runs right over those public roads. Inside the tunnels, your pods can have their own private addresses and zip around freely. The public road system doesn’t need to know anything about what’s happening inside the tunnels; it just sees traffic moving from one tunnel entrance to another.

This is the trick that decouples your application’s network from the physical hardware. Your network team doesn’t have to reconfigure a single switch.

With VXLAN, you’re looking at Layer 2 Ethernet frames wrapped in UDP packets (port 8472, in case your firewall folks ask). Each overlay network gets a 24-bit VNI (VXLAN Network Identifier), which means you can have about 16 million separate networks. That’s a lot of isolation. Geneve is similar but more flexible — it can carry additional metadata that Cilium uses for identity and policy enforcement.

If overlay networks are the tunnels, VXLAN (Virtual Extensible LAN) is the most common blueprint used to build them. It’s the reliable, battle-tested workhorse of the container world. Here’s how it works, but let’s ditch the jargon.

Imagine you’re sending a letter from one office department to another in a different building.

The Inner Letter: Your pod creates a normal packet, addressed to another pod. This is your original message, sealed in an inner envelope.

The Mailroom Clerk: This packet hits a special component on the server called a VTEP (VXLAN Tunnel Endpoint). The VTEP acts like your mailroom clerk.

The Outer Envelope: The VTEP takes your entire inner envelope and stuffs it into a bigger, more durable outer envelope (a UDP packet).

Special Instructions: On the outer envelope, the VTEP writes two things: the destination server’s IP address (the other building’s street address) and a VNI (VXLAN Network Identifier).

The VNI is like a special department code that says, “This belongs to the ‘Blue Project’ virtual network.”

Delivery: The physical network sees only the outer envelope and dutifully delivers it to the destination server. The receiving server’s mailroom clerk opens it, discards the outer envelope, and hands the original letter to the correct pod.

The real genius here is the VNI. It gives you about 16 million unique department codes, blowing past the old 4,096 limit of VLANs. For running multiple tenants or projects on the same hardware, that’s a game-changer.

If VXLAN is the reliable workhorse, Geneve (Generic Network Virtualization Encapsulation) is its younger, smarter sibling. It was designed to do everything VXLAN does, but with more flexibility for the future.

Geneve’s killer feature is its ability to carry extra metadata.

Think of it this way. If VXLAN uses a standard, fixed-size outer envelope, Geneve uses one of those expandable ones with a clear window where you can attach all sorts of extra sticky notes.

Why does that matter?

A CNI like Cilium can attach a sticky note that says, “This packet comes from a pod with security identity #12345.”

When the packet arrives, the destination server can read that note and instantly decide whether to allow or deny the traffic without having to do extra work. It’s this ability to carry extra context that makes modern CNIs so powerful and efficient.

This tunneling magic isn’t completely free. There are a couple of trade-offs you should know about.

First, overhead. Wrapping every packet in an outer envelope adds about 50 bytes. This means the “letter” you can fit inside is slightly smaller (a lower MTU), which can occasionally cause issues if an application tries to send a packet that’s too big.

Second, CPU work. There’s a tiny computational cost to wrap and unwrap every packet. Modern CPUs are champs at this, but at a massive scale, it’s a factor.

But here’s the big win: operational simplicity. For these minor costs, you get a network that just works, anywhere. Public cloud, your own data center, even across multiple sites, it doesn’t matter. Your pods get a consistent, predictable network without you having to become a networking guru.

The beauty of encapsulation is its simplicity. Open the right UDP ports, and you’re done. No network team meetings, no BGP configurations, no arguing about IP ranges. Your pods get their own private network, and life is good.

The downside is that every packet needs extra headers for the tunnel, which reduces your effective MTU. You’re also spending CPU cycles on encapsulation and decapsulation. For many workloads, this overhead is negligible. But if you’re pushing serious traffic or counting every microsecond of latency, you might want option two.

Native routing mode is for when performance really matters. Enable it with routing-mode: native, and Cilium stops tunneling entirely. Instead, it manipulates the Linux routing table directly, making pod traffic look like any other routed traffic on your network.

Each pod gets a lxc interface (don't let the name fool you. It's got nothing to do with LXC containers), and Cilium ensures the routing table knows how to reach pods on other nodes. The packets flow through your network exactly like packets from any other application. No tunnels, no overhead, just pure speed.

The catch is that your network needs to understand pod IPs. If all your nodes are on the same L2 segment, you can use auto-direct-node-routes: true and Cilium figures it out. Otherwise, you'll need BGP to advertise routes, or ensure your routers know about pod subnets. This usually means having that meeting with the network team.

Replacing kube-proxy

In a traditional Kubernetes cluster, kube-proxy sits on every node, managing iptables rules that implement services. When you create a service, kube-proxy updates potentially thousands of iptables rules. When you have thousands of services, you get tens of thousands of rules. Each packet has to traverse these rules sequentially. It’s O(n) complexity at its worst.

When looking at large clusters where creating a new service takes minutes to become functional. Not because Kubernetes is slow, but because regenerating and applying all those iptables rules takes forever. And that’s just the performance hit, iptables can only match on IP and port, so forget about sophisticated L7 policies.

Cilium throws all of this out. When you enable kube-proxy replacement, Cilium implements services entirely in eBPF. Service-to-endpoint mappings live in BPF maps with O(1) lookups. Creating or updating a service is nearly instantaneous, just update a map entry. No rule regeneration, no performance degradation at scale.

But it gets better. Cilium can do socket-level load balancing, making the backend selection when a connection is established rather than per-packet. This is more efficient and provides better session affinity. And because it’s all eBPF, Cilium can implement load balancing algorithms that would be impossible with iptables.

Architecture

Let’s pop the hood and see how Cilium actually works. The architecture is elegant in its simplicity, which is impressive given what it accomplishes.

The Cilium agent runs as a DaemonSet — one pod per node. This is the workhorse, watching the Kubernetes API for changes and programming eBPF accordingly. It’s also got cilium-dbg built in, a CLI tool that's invaluable for troubleshooting. You can inspect eBPF maps, trace packet paths, and generally see what's happening under the hood.

The Cilium operator handles cluster-wide operations. It manages CiliumNode resources, coordinates IP allocation, and handles garbage collection. While the agent is required for basic CNI functionality, the operator enables Cilium’s advanced features. Skip it at your own peril.

Then there’s Hubble, which deserves its own section. The Hubble server runs embedded in each Cilium agent, exposing flow information via gRPC and Prometheus metrics. The Hubble relay aggregates data from all nodes, giving you that cluster-wide view. And the Hubble UI? It’s legitimately beautiful. Watching service dependencies visualized in real-time never gets old.

For state storage, Cilium primarily uses Kubernetes custom resources, which makes sense — use the infrastructure you’ve got. But it can also talk directly to etcd if needed, which opens up some interesting deployment options for advanced users.

Before you dive into Cilium, you need to understand its vocabulary. These aren’t just random terms — they represent fundamental concepts in how Cilium thinks about networking.

Labels in Cilium are like labels in Kubernetes, but more. They’re not just metadata; they’re the foundation of identity and policy. Cilium labels can come from Kubernetes, containers, or be reserved for special purposes. They determine how endpoints are identified and how policies are applied.

Endpoints are Cilium’s unit of network identity. Usually, an endpoint is a pod, but nodes are endpoints too. So are health check endpoints and other special entities. Each endpoint gets an IP and a set of labels that define its identity.

Identity is where Cilium gets really clever. Instead of using IPs for policy (which change all the time in Kubernetes), Cilium assigns numerical identities based on label combinations. Pods with the same labels get the same identity, regardless of their IPs. This makes policies both more stable and more efficient.

There are some special reserved identities you should know:

• reserved:host (1) - Traffic from the local node
• reserved:world (2) - Traffic from outside the cluster
• reserved:health (4) - Health checking traffic
• reserved:init (5) - Endpoints that don't have their real identity yet

These identities are managed through a distributed KV store (usually CiliumIdentity custom resources). When Cilium sees a new label combination, it allocates a new identity. This happens automatically and transparently.

Practice

Installing with Flannel CNI

Before jumping into the deep end with eBPF-based networking, I wanted to build a solid foundation with a more traditional CNI plugin. So I spent my Sunday afternoon setting up a Kubernetes cluster with Flannel.

My goal was simple: create a reproducible Kubernetes environment using Vagrant and VirtualBox. Nothing fancy — just one control plane node and two worker nodes, enough to see how pod networking actually works across multiple machines.

I started with a Vagrantfile that would spin up three Ubuntu 24.04 VMs. The beauty of using Vagrant is that I can destroy everything and rebuild it in minutes. Perfect for learning through breaking things (which, spoiler alert, I did several times).

After running vagrant up and waiting for all the provisioning scripts to complete, I SSHed into the control plane and ran the classic kubectl get nodes. The nodes were there, but something looked off:

NAME      STATUS     ROLES           AGE     VERSION   INTERNAL-IP
k8s-ctr   NotReady   control-plane   8m33s   v1.33.2   192.168.10.100
k8s-w1    NotReady   <none>          6m55s   v1.33.2   10.0.2.15
k8s-w2    NotReady   <none>          5m26s   v1.33.2   10.0.2.15

Wait, why do both worker nodes have the same internal IP? That can’t be right.

Turns out, VirtualBox gives each VM two network interfaces by default: a NAT interface (eth0) and a host-only interface (eth1). Kubernetes was picking up the NAT interface, which has the same IP on every VM. Classic mistake.

The fix was straightforward but took some digging. I had to tell kubelet explicitly which IP to use by adding the --node-ip flag to its configuration.

NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
sed -i "s/^\(KUBELET_KUBEADM_ARGS=\"\)/\1--node-ip=${NODEIP} /" /var/lib/kubelet/kubeadm-flags.env
systemctl restart kubelet

After doing this on all nodes and restarting, the IPs looked much better. Small victory.

With the nodes showing correct IPs, they were still in NotReady state. The CoreDNS pods were stuck in Pending. This is expected – without a CNI plugin, Kubernetes doesn't know how to set up networking for pods.

Time to install Flannel. I chose Flannel because it’s straightforward and uses VXLAN by default, which I wanted to understand better. Using Helm made the installation clean.

helm repo add flannel https://flannel-io.github.io/flannel/
helm install flannel --namespace kube-flannel flannel/flannel

Watching the cluster come to life after installing the CNI is always satisfying. The nodes switched to Ready, CoreDNS pods started running, and suddenly I had a functioning cluster.

This is where things got interesting. I wanted to understand what Flannel actually did to make pod networking work. So I started poking around with various networking tools.

Before Flannel, each node just had its basic interfaces. After Flannel, I could see new interfaces appearing.

• flannel.1: The VXLAN interface that creates the overlay network
• cni0: A bridge on each node where pods connect

Running ip route showed how Flannel set up routing between nodes.

10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink 
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink

Each node got its own /24 subnet from the cluster's pod CIDR (10.244.0.0/16). Simple and elegant.

Theory is great, but I wanted to see this working with actual pods. I deployed a simple web application with two replicas, making sure they landed on different nodes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      containers:
      - name: webpod
        image: traefik/whoami

The whoami image is perfect for testing — it returns information about the pod handling the request. After creating a service for the deployment, I could curl it and watch the requests load balance between pods.

Hostname: webpod-697b545f57-qlpz9
Hostname: webpod-697b545f57-lwbqs
Hostname: webpod-697b545f57-lwbqs
Hostname: webpod-697b545f57-qlpz9

This is where I really fell down the rabbit hole. I wanted to understand how kube-proxy was handling the service load balancing. Time to dive into iptables.

Looking at the NAT table revealed the magic.

iptables -t nat -S | grep 10.96.102.141  # The service ClusterIP

I could see the probability-based load balancing right there in the rules:

-A KUBE-SVC-CNZCPOCNCNOROALA -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-NBAMRQIHYPBRH37V
-A KUBE-SVC-CNZCPOCNCNOROALA -j KUBE-SEP-WEW7NHLZ4Y5A5ZKF

For a service with two endpoints, the first rule matches 50% of the time, sending traffic to the first pod. The remaining traffic falls through to the second rule, going to the second pod. Clever and simple.

Cillium Playground

I started by methodically removing Flannel from the cluster. This wasn’t just about uninstalling a Helm chart; it required understanding what Flannel had created and ensuring a clean slate for Cilium.

First, I uninstalled the Flannel Helm release and deleted the namespace. But that’s just the beginning. Flannel creates several network interfaces and iptables rules that persist even after the pods are gone:

• flannel.1: The VXLAN tunnel interface
• cni0: The bridge interface connecting local pods
• Multiple veth pairs connecting to pod network namespaces
• Extensive iptables rules for SNAT/masquerading

I had to manually clean up these artifacts on each node. The veth interfaces were particularly tricky — they were still connected to running pod namespaces, showing that my application pods (a simple webpod deployment and a curl-pod for testing) were still using the Flannel-created network infrastructure.

After cleaning up Flannel and attempting to install Cilium, I hit my first major roadblock. The Cilium agent pods were crashing, but not on all nodes — only on k8s-w1. The error messages were cryptic at first, but diving into the pod descriptions revealed the issue:

Node: k8s-w1/10.0.2.15

Wait, that’s not right. My worker node should be using 192.168.10.101, not 10.0.2.15. It turned out that all my nodes were reporting the same INTERNAL-IP address (10.0.2.15), which is VirtualBox’s NAT interface IP. This was causing Cilium’s node discovery to fail spectacularly.

When kubelet starts, it tries to detect the node’s IP address, and in a multi-interface environment, it often picks the wrong one. In my case, it was consistently choosing eth0 (NAT) instead of eth1 (host-only network).

The solution required updating the kubelet configuration on each node to explicitly specify which IP to use:

NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
sed -i "s/^\(KUBELET_KUBEADM_ARGS=\"\)/\1--node-ip=${NODEIP} /" /var/lib/kubelet/kubeadm-flags.env
systemctl restart kubelet

This fixed the INTERNAL-IP reporting, but I still had inconsistencies. The static pods (kube-apiserver, etcd, etc.) were still bound to the wrong IP. A quick restart of these pods resolved that issue.

Just when I thought I was making progress, I encountered an unexpected adversary: the LLVM compiler used to compile Cilium’s BPF programs crashed with a segmentation fault. The stack trace pointed to an optimization pass failure:

level=fatal msg="alignchecker compile failed" error="Failed to compile bpf_alignchecker.o: exit status 1"

This was running on ARM64 architecture (Apple Silicon through Vagrant), and it seemed the specific combination of LLVM 18.1.8 and the BPF target was causing issues. The crash occurred during the EarlyCSE (Common Subexpression Elimination) optimization pass.

After multiple attempts with different Cilium configurations, I finally found a working combination. The key was to:

1. Completely clean up all network artifacts from the previous CNI
2. Ensure all nodes report correct IP addresses before installing Cilium
3. Use a more conservative Cilium configuration that avoided certain BPF features
4. Switch to a different pod CIDR (172.20.0.0/16) to avoid any potential conflicts

The final Helm installation command that worked:

helm install cilium cilium/cilium --version 1.17.5 --namespace kube-system \
--set k8sServiceHost=192.168.10.100 --set k8sServicePort=6443 \
--set kubeProxyReplacement=true \
--set routingMode=native \
--set autoDirectNodeRoutes=true \
--set ipam.mode="cluster-pool" \
--set ipam.operator.clusterPoolIPv4PodCIDRList={"172.20.0.0/16"} \
--set ipv4NativeRoutingCIDR=172.20.0.0/16 \
--set endpointRoutes.enabled=true \
--set installNoConntrackIptablesRules=true \
--set bpf.masquerade=true \
--set ipv6.enabled=false

After hours of debugging, seeing all nodes report “Ready” with Cilium running successfully was incredibly satisfying. The cluster now runs with Cilium’s advanced eBPF-based networking, providing better performance and observability through Hubble.

The curl test confirmed everything was working:

curl-pod:~# curl webpod
Hostname: webpod-75f74965fd-gwwz8

DNS resolution worked (though with some “recursion not available” warnings that are actually normal for Cilium’s DNS proxy), pod-to-pod communication functioned correctly, and the service load balancing operated as expected.

After successfully installing Cilium in my homelab, I should dive deeper into understanding how pod-to-pod communication actually works under the hood.

The goal was to understand exactly how Cilium handles different communication patterns: same-node pod communication, cross-node pod communication, and service-based load balancing.

My investigation started with Cilium’s IPCache, which is essentially the brain of pod-to-pod routing decisions. Using the Cilium CLI, I discovered how each pod IP is mapped to its identity and tunnel endpoint:

# Check the IPCache entries
c0 cilium-dbg map get cilium_ipcache | grep -E "$WEBPOD1IP|$WEBPOD2IP"

The output revealed something fascinating:

172.20.2.10/32    identity=4066 encryptkey=0 tunnelendpoint=192.168.10.101
172.20.0.22/32    identity=4066 encryptkey=0 tunnelendpoint=0.0.0.0

Notice how webpod-1 has a tunnel endpoint (192.168.10.101), while webpod-2 shows 0.0.0.0? This immediately told me that Cilium knows webpod-1 is on a different node, requiring tunneling, while webpod-2 is local.

Next, I wanted to see the actual eBPF programs handling the traffic. Using bpftool, I discovered multiple instances of cil_from_container and cil_to_container programs:

kubectl exec -it -n kube-system $CILIUM_POD -c cilium-agent -- \
  bpftool prog list | grep -E "cil_from_container|cil_to_container"

These programs are attached to each pod’s network interface and handle all ingress and egress traffic. The variety of program IDs (1376, 1396, 1407, etc.) showed that each endpoint gets its own set of programs, customized for that specific pod’s security policies and routing needs.

When testing communication from curl-pod to webpod-1 (different nodes), I set up ngrep on both nodes to capture the traffic. What I expected to see was straightforward — packets leaving k8s-w2 and arriving at k8s-w1.

However, the initial attempts to capture traffic revealed an interesting challenge. The packets were traveling through the VXLAN tunnel, and standard tcpdump on eth1 wasn’t showing the encapsulated traffic as expected. This led me to realize that Cilium was operating in tunnel mode rather than native routing mode.

The most interesting discovery came when testing Service-based communication. I wanted to see Cilium’s Socket-Based LoadBalancing in action, so I traced the system calls:

kubectl exec curl-pod -- strace -e trace=connect curl -s webpod

The output showed:

connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("10.96.153.236")}, 16) = 0

The application clearly connected to the Service IP (10.96.153.236). But when I ran tcpdump, I expected to see the magic of socket-level load balancing where the connection would be redirected at the socket level to bypass iptables DNAT.

Instead, tcpdump revealed something unexpected:

172.20.0.105.36162 > 10.96.153.236.80: tcp 0
10.96.153.236.80 > 172.20.0.105.36162: tcp 0

The packets were still using the Service IP! This led me to check the kube-proxy replacement status:

kubectl exec -it -n kube-system ds/cilium -c cilium-agent -- \
  cilium-dbg status --verbose | grep -A 20 "KubeProxyReplacement"

And there it was:

KubeProxyReplacement:   False
Socket LB:              Disabled

Despite installing Cilium with --set kubeProxyReplacement=true, the feature wasn't actually enabled. This explained why I was seeing traditional Service IP communication rather than the optimized socket-level redirection I expected.

Without kube-proxy replacement:

1. The application connects to the Service IP
2. iptables rules (maintained by kube-proxy) perform DNAT
3. The packet is routed to the backend pod
4. Return traffic goes through reverse NAT

With proper kube-proxy replacement enabled, I would have seen:

1. The application connects to the Service IP
2. eBPF programs intercept at the socket level
3. Direct connection to the backend pod IP
4. No iptables involvement

To better understand the networking stack, I performed comprehensive system call analysis:

kubectl exec curl-pod -- strace -c curl -s webpod

The statistics revealed the complexity of a simple HTTP request:

• 63 mmap calls for memory management
• 47 openat calls (with 30 errors — typical for library loading)
• 3 connect calls (DNS and HTTP)
• Multiple socket operations

The getsockname calls confirmed that the source IP remained the pod IP (172.20.0.105) throughout the communication, showing that at least source IP preservation was working correctly.