Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

Cilium Network Deep Dive: Routing, Masquerading, and DNS

Sigrid Jin
9 min readAug 2, 2025

--

Press enter or click to view image in full size
https://docs.cilium.io/en/stable/network/concepts/masquerading/

Today we’re going to dig into how Cilium handles network traffic in a Kubernetes cluster. I’ll show you exactly what happens when pods talk to each other, when they reach out to external networks, and how DNS resolution works. Everything here is based on actual lab testing, so you can follow along and see the same results.

Setting Up Our Test Environment

Let me show you the cluster I’m working with. First thing — always check your network configuration:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl cluster-info dump | grep -m 2 -E "cluster-cidr|service-cluster-ip-range"
                            "--service-cluster-ip-range=10.96.0.0/16",
                            "--cluster-cidr=10.244.0.0/16",

So we have:

  • Service IPs: 10.96.0.0/16
  • Pod IPs: 10.244.0.0/16

Here’s our node setup:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get node -owide
NAME      STATUS   ROLES           AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   8m30s   v1.33.2   192.168.10.100   <none>        Ubuntu 24.04.2 LTS   6.8.0-53-generic   containerd://1.7.27
k8s-w1    Ready    <none>          6m18s   v1.33.2   192.168.10.101   <none>        Ubuntu 24.04.2 LTS   6.8.0-53-generic   containerd://1.7.27

Two nodes — control plane at 192.168.10.100 and a worker at 192.168.10.101. Pretty standard setup.

Now, Kubernetes assigns each node a slice of the pod CIDR:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
k8s-ctr	10.244.0.0/24
k8s-w1	10.244.1.0/24

See that? Control plane gets 10.244.0.0/24, worker gets 10.244.1.0/24. This is important — it means pods on different nodes have different IP ranges.

Cilium is using Kubernetes IPAM mode:

(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium config view | grep ^ipam
ipam                                              kubernetes

This means Kubernetes decides which IPs to assign, and Cilium just uses them. Simple.

Deploying Test Applications

Let’s create some pods to test with. I’m using a deployment with anti-affinity to force pods onto different nodes:

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - webpod
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: nicolaka/netshoot
        command: ["nginx", "-g", "daemon off;"]
      terminationGracePeriodSeconds: 0
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
spec:
  selector:
    app: webpod
  ports:
  - port: 80
    targetPort: 80
EOF

The anti-affinity rule ensures our two pods land on different nodes. This is crucial for testing cross-node communication.

Now let’s create a pod we can use to test from:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  nodeName: k8s-ctr
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF

I’m explicitly placing this on the control plane node with nodeName: k8s-ctr.

Let’s check what Cilium sees:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ciliumendpoints
NAME                      SECURITY IDENTITY   ENDPOINT STATE   IPV4           IPV6
curl-pod                  30577               ready            10.244.0.171
webpod-697b545f57-4bxtv   4981                ready            10.244.1.105
webpod-697b545f57-ngq7g   4981                ready            10.244.0.225

Perfect! We have:

  • curl-pod at 10.244.0.171 (control plane node)
  • One webpod at 10.244.1.105 (worker node)
  • Another webpod at 10.244.0.225 (control plane node)

How Service Communication Works

Let’s see what happens when we access the service:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- curl webpod | grep Hostname
Hostname: webpod-697b545f57-ngq7g

Now watch this with Hubble:

(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f --protocol tcp --pod curl-pod
Jul 31 15:53:02.473: default/curl-pod (ID:30577) <> 10.96.107.144:80 (world) pre-xlate-fwd TRACED (TCP)
Jul 31 15:53:02.473: default/curl-pod (ID:30577) <> default/webpod-697b545f57-ngq7g:80 (ID:4981) post-xlate-fwd TRANSLATED (TCP)
Jul 31 15:53:02.473: default/curl-pod:57278 (ID:30577) -> default/webpod-697b545f57-ngq7g:80 (ID:4981) to-endpoint FORWARDED (TCP Flags: SYN)

Look at that flow:

  1. pre-xlate-fwd: Before translation, we see the service IP (10.96.107.144)
  2. post-xlate-fwd: After translation, it's going to the actual pod
  3. to-endpoint FORWARDED: Cilium allows and forwards the packet

This is Cilium’s eBPF magic — it intercepts the service IP and translates it to a real pod IP, all in the kernel.

Pod-to-Pod Routing Across Nodes

Now for the interesting part — what happens when pods on different nodes talk directly?

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- ping 10.244.1.105
PING 10.244.1.105 (10.244.1.105) 56(84) bytes of data.
64 bytes from 10.244.1.105: icmp_seq=1 ttl=62 time=1.08 ms
64 bytes from 10.244.1.105: icmp_seq=2 ttl=62 time=0.548 ms

Let’s capture this on the physical interface:

(⎈|HomeLab:N/A) root@k8s-ctr:~# tcpdump -i eth1 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:26:56.439502 IP 10.244.0.171 > 10.244.1.105: ICMP echo request, id 50, seq 1, length 64
21:26:56.439938 IP 10.244.1.105 > 10.244.0.171: ICMP echo reply, id 50, seq 1, length 64

This is important! The pod IPs (10.244.0.171 and 10.244.1.105) are directly visible on the physical network. No NAT happening here. This is what we call “native routing” — the pod IPs are treated as real IPs on the network.

Understanding Masquerading

Let’s check Cilium’s masquerading configuration:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it -n kube-system ds/cilium -c cilium-agent  -- cilium status | grep Masquerading
Masquerading:            BPF   [eth0, eth1]   10.244.0.0/16 [IPv4: Enabled, IPv6: Disabled]

(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium config view  | grep ipv4-native-routing-cidr
ipv4-native-routing-cidr                          10.244.0.0/16

This tells us:

  • Masquerading is enabled using BPF (not iptables!)
  • Native routing (no NAT) for 10.244.0.0/16

So what does this mean? Let’s test by pinging another Kubernetes node:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- ping 192.168.10.101
PING 192.168.10.101 (192.168.10.101) 56(84) bytes of data.
64 bytes from 192.168.10.101: icmp_seq=1 ttl=63 time=1.03 ms

Capture the traffic:

(⎈|HomeLab:N/A) root@k8s-ctr:~# tcpdump -i eth1 icmp -nn
21:56:44.937410 IP 10.244.0.171 > 192.168.10.101: ICMP echo request, id 56, seq 1, length 64
21:56:44.938100 IP 192.168.10.101 > 10.244.0.171: ICMP echo reply, id 56, seq 1, length 64

Still using the pod IP! The worker node knows about pod CIDRs, so no NAT needed.

External Communication

Now let’s see what happens when we talk to something outside the cluster. I have a router at 192.168.10.200:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- ping 192.168.10.200
PING 192.168.10.200 (192.168.10.200) 56(84) bytes of data.
64 bytes from 192.168.10.200: icmp_seq=1 ttl=63 time=1.03 ms

Capture on the control plane:

(⎈|HomeLab:N/A) root@k8s-ctr:~# tcpdump -i eth1 icmp -nn
22:10:47.709153 IP 192.168.10.100 > 192.168.10.200: ICMP echo request, id 43011, seq 1, length 64
22:10:47.709566 IP 192.168.10.200 > 192.168.10.100: ICMP echo reply, id 43011, seq 1, length 64

And on the router:

root@router:~# tcpdump -i eth1 icmp -nn
22:10:47.678986 IP 192.168.10.100 > 192.168.10.200: ICMP echo request, id 43011, seq 1, length 64
22:10:47.679310 IP 192.168.10.200 > 192.168.10.100: ICMP echo reply, id 43011, seq 1, length 64

Whoa! The source IP changed! It’s now 192.168.10.100 (the node IP) instead of 10.244.0.171 (the pod IP).

Why? Because the router has no idea what 10.244.0.0/16 is. If we sent the packet with the pod IP, the router wouldn’t know where to send the reply. So Cilium does SNAT (Source NAT) automatically.

Configuring SNAT Exceptions with ip-masq-agent

Sometimes you don’t want SNAT. Maybe you have internal networks that DO know about pod CIDRs. Let’s configure exceptions:

(⎈|HomeLab:N/A) root@k8s-ctr:~# helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
--set ipMasqAgent.enabled=true --set ipMasqAgent.config.nonMasqueradeCIDRs='{10.10.1.0/24,10.10.2.0/24}'

This tells Cilium: “Don’t do SNAT for traffic going to 10.10.1.0/24 or 10.10.2.0/24”.

Let’s verify the configuration:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get cm -n kube-system ip-masq-agent -o yaml | yq
{
  "apiVersion": "v1",
  "data": {
    "config": "{\"nonMasqueradeCIDRs\":[\"10.10.1.0/24\",\"10.10.2.0/24\"]}"
  },
  "kind": "ConfigMap",
  "metadata": {
    "name": "ip-masq-agent",
    "namespace": "kube-system"
  }
}

Check the BPF map:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf ipmasq list
IP PREFIX/ADDRESS
10.10.1.0/24
10.10.2.0/24
169.254.0.0/16

Now when I try to reach 10.10.1.200:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- curl -s 10.10.1.200

On the router:

root@router:~# tcpdump -i eth1 tcp port 80 -nnq
23:57:19.584903 IP 172.20.1.71.56878 > 10.10.1.200.80: tcp 0
23:57:20.644870 IP 172.20.1.71.56878 > 10.10.1.200.80: tcp 0

See? Now it’s using the pod IP (172.20.1.71) directly! But there’s a problem — no response. Why? Because the router doesn’t know how to route back to 172.20.1.0/24.

Let’s fix that:

root@router:~# ip route add 172.20.1.0/24 via 192.168.10.100
root@router:~# ip route add 172.20.0.0/24 via 192.168.10.101

Now try again:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- curl -s 10.10.1.200
<h1>Web Server : router</h1>

Success! The traffic now flows with the real pod IP:

root@router:~# tcpdump -i eth1 tcp port 80 -nnq
23:56:15.768505 IP 172.20.1.71.41642 > 10.10.1.200.80: tcp 0
23:56:15.769443 IP 10.10.1.200.80 > 172.20.1.71.41642: tcp 0
23:56:15.769717 IP 172.20.1.71.41642 > 10.10.1.200.80: tcp 75
23:56:15.785994 IP 10.10.1.200.80 > 172.20.1.71.41642: tcp 256

This is useful when you have internal networks that need to see real pod IPs for security or auditing.

DNS Resolution in Kubernetes

Now let’s explore how DNS works. First, check what DNS configuration pods get:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5

This comes from kubelet:

(⎈|HomeLab:N/A) root@k8s-ctr:~# cat /var/lib/kubelet/config.yaml | grep cluster -A1
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local

The ndots:5 is interesting. It means any name with less than 5 dots gets the search domains appended. So google.com becomes:

  • google.com.default.svc.cluster.local
  • google.com.svc.cluster.local
  • google.com.cluster.local
  • google.com.localdomain
  • google.com

Let’s see CoreDNS in action:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc,ep -n kube-system kube-dns
NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   81m

NAME                 ENDPOINTS                                                     AGE
endpoints/kube-dns   172.20.0.171:53,172.20.1.166:53,172.20.0.171:53 + 3 more...   81m

The kube-dns service at 10.96.0.10 load balances to CoreDNS pods.

Let’s trace a DNS query. First, scale down CoreDNS to make it easier to follow:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl scale deployment -n kube-system coredns --replicas 1

Now watch what happens when we query:

(⎈|HomeLab:N/A) root@k8s-ctr:~# tcpdump -i any udp port 53 -nn

In another terminal:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- nslookup kube-dns.kube-system.svc

The tcpdump shows:

00:30:05.466544 lxcbc5692d6cb9d In  IP 172.20.1.71.57990 > 172.20.1.166.53: 16875+ [1au] A? kube-dns.kube-system.svc.default.svc.cluster.local. (91)
00:30:05.468548 lxc981c4880b17f In  IP 172.20.1.166.53 > 172.20.1.71.57990: 16875 NXDomain*- 0/1/1 (184)
00:30:05.469314 lxcbc5692d6cb9d In  IP 172.20.1.71.57990 > 172.20.1.166.53: 58543+ [1au] A? kube-dns.kube-system.svc.cluster.local. (67)
00:30:05.469593 lxc981c4880b17f In  IP 172.20.1.166.53 > 172.20.1.71.57990: 58543*- 1/0/1 A 10.96.0.10 (121)

See the search process? It tries the longer names first, gets NXDOMAIN, then finds the right one.

For external domains, it’s even more queries:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- nslookup -debug google.com

This generates 4 failed queries before finally resolving google.com!

Optimizing DNS with NodeLocal DNS Cache

All those queries add latency. Let’s deploy NodeLocal DNS to cache queries on each node:

wget https://github.com/kubernetes/kubernetes/raw/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

kubedns=`kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}`
domain='cluster.local'
localdns='169.254.20.10'

sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml
kubectl apply -f nodelocaldns.yaml

This creates a DaemonSet — one DNS cache pod per node:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pod -n kube-system -l k8s-app=node-local-dns -owide
NAME                   READY   STATUS    RESTARTS   AGE   IP               NODE      NOMINATED NODE   READINESS GATES
node-local-dns-mr4vz   1/1     Running   0          66s   192.168.10.100   k8s-ctr   <none>           <none>
node-local-dns-zbprb   1/1     Running   0          66s   192.168.10.101   k8s-w1    <none>           <none>

But here’s the problem — pods still query 10.96.0.10 (CoreDNS), not the local cache. We need Cilium’s help!

The Power of LocalRedirectPolicy

Enable LocalRedirectPolicy in Cilium:

(⎈|HomeLab:N/A) root@k8s-ctr:~# helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
--version 1.17.6 \
--set localRedirectPolicy=true

Apply the policy:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.17.6/examples/kubernetes-local-redirect/node-local-dns-lrp.yaml

Check what this did:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it -n kube-system ds/cilium -c cilium-agent -- cilium-dbg lrp list
LRP namespace   LRP name       FrontendType                Matching Service
kube-system     nodelocaldns   clusterIP + all svc ports   kube-system/kube-dns
                |              10.96.0.10:53/UDP -> 172.20.0.55:53(kube-system/node-local-dns-b7hxt),
                |              10.96.0.10:53/TCP -> 172.20.0.55:53(kube-system/node-local-dns-b7hxt),

This is brilliant! Cilium now intercepts traffic to 10.96.0.10:53 and redirects it to the local node-local-dns pod. No network hop!

Let’s test it:

(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- nslookup www.google.com

Watch the node-local-dns logs:

kubectl -n kube-system logs -l k8s-app=node-local-dns -f
[INFO] 172.20.0.47:58903 - 49909 "A IN www.google.com.default.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd 151 0.000107619s
[INFO] 172.20.0.47:57480 - 28113 "A IN www.google.com.localdomain. udp 44 false 512" NOERROR qr,aa,rd 86 0.000037123s

And CoreDNS logs:

kubectl -n kube-system logs -l k8s-app=kube-dns -f
[INFO] 172.20.0.55:45542 - 52596 "A IN www.google.com.default.svc.cluster.local. tcp 58 false 65535" NXDOMAIN qr,aa,rd 151 0.00515458s

Notice:

  • Queries hit node-local-dns first (UDP)
  • Cache misses forward to CoreDNS (TCP)
  • Responses are cached locally

The second time you query the same domain, it comes from cache — no CoreDNS hit at all!

What We’ve Learned

Let me summarize the key behaviors we’ve observed:

Routing and Masquerading:

  • Pod-to-pod within cluster: Direct routing, no NAT
  • Pod-to-Kubernetes-nodes: Direct routing, no NAT
  • Pod-to-external: SNAT to node IP (unless configured otherwise)

DNS Resolution:

  • Without NodeLocal DNS: Every query goes across the network to CoreDNS
  • With NodeLocal DNS alone: Still goes to CoreDNS (just different config)
  • With NodeLocal DNS + LocalRedirectPolicy: eBPF magic redirects to local cache

The beauty of Cilium is that all of this happens in the kernel with eBPF. No iptables rules, no userspace proxies. Just pure kernel-level packet handling.

When you combine these features — native routing where possible, intelligent SNAT where needed, and local DNS caching — you get a network that’s both fast and flexible. And the best part? It’s all observable with tools like Hubble, so you can see exactly what’s happening to every packet.

That’s the power of eBPF-based networking with Cilium. Every optimization happens at the lowest possible level, giving you the best possible performance while maintaining full visibility and control.

Cilium

--

--

Sigrid Jin
Sigrid Jin

Written by Sigrid Jin

876 followers
1K following

twitter.com/@realsigridjin

Help

Status

About

Careers

Press

Blog

Store

Privacy

Rules

Terms

Text to speech