I love Hubble Telescope
Gasida Week 2 Cilllium Study
42 min readJul 26, 2025
Before diving into the technical bits, let me explain why I chose this particular configuration. When you’re learning Cilium, you want a reproducible environment where you can break things without fear. VirtualBox + Vagrant gives us exactly that — a disposable, consistent lab that spins up in minutes.
I’m intentionally installing Cilium without Hubble and some other features initially. This gives us a clean baseline to understand the core networking components before adding observability layers. Trust me, it’s much easier to troubleshoot when you start simple!
Let me show you exactly what I did. First, I needed to get VirtualBox and Vagrant installed. Pretty straightforward:
❯ brew install --cask virtualbox vagrant
==> Downloading https://download.virtualbox.org/virtualbox/7.0.14/VirtualBox-7.0.14-161095-OSX.dmg
######################################################################## 100.0%
==> Installing Cask virtualbox
==> Running installer for virtualbox with sudo; the password may be necessary.
Password:
installer: Package name is Oracle VM VirtualBox
installer: Installing at base path /
installer: The install was successful.
🍺 virtualbox was successfully installed!So here’s the thing — I wanted a clean Cilium install without all the bells and whistles initially. No Hubble, no fancy observability stuff. Just pure networking so I could understand what’s actually happening under the hood.
I grabbed the Vagrantfile from a repo I’ve been maintaining:
❯ mkdir cilium-lab && cd cilium-lab
❯ curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/2w/Vagrantfile
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4521 100 4521 0 0 15986 0 --:--:-- --:--:-- --:--:-- 16039❯ ls -la
total 8
drwxr-xr-x 3 user staff 96 Jan 27 10:15 .
drwxr-xr-x 45 user staff 1440 Jan 27 10:15 ..
-rw-r--r-- 1 user staff 4521 Jan 27 10:15 VagrantfileNow, before I run this, let me check if I have any old VMs hanging around. This bit me hard the first time:
❯ VBoxManage list vms
"k8s-ctr" {4d3f6b89-5c72-4d9f-8e2a-1a2b3c4d5e6f}
"k8s-w1" {7a8b9c0d-1e2f-3a4b-5c6d-7e8f9a0b1c2d}
"k8s-w2" {2b3c4d5e-6f7a-8b9c-0d1e-2f3a4b5c6d7e}
"minikube" {9f8e7d6c-5b4a-3c2d-1e0f-9d8c7b6a5e4f}Ah crap, I’ve got old stuff from last week’s experiments. Let me nuke those:
❯ VBoxManage unregistervm k8s-ctr --delete
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%❯ VBoxManage unregistervm k8s-w1 --delete
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%❯ VBoxManage unregistervm k8s-w2 --delete
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Cool, now let’s spin up the cluster. This takes a while, so I usually grab coffee:
❯ vagrant up
Bringing machine 'k8s-ctr' up with 'virtualbox' provider...
Bringing machine 'k8s-w1' up with 'virtualbox' provider...
Bringing machine 'k8s-w2' up with 'virtualbox' provider...
==> k8s-ctr: Importing base box 'bento/ubuntu-24.04'...
==> k8s-ctr: Matching MAC address for NAT networking...
==> k8s-ctr: Checking if box 'bento/ubuntu-24.04' version '202502.21.0' is up to date...
==> k8s-ctr: Setting the name of the VM: k8s-ctr
==> k8s-ctr: Clearing any previously set network interfaces...
==> k8s-ctr: Preparing network interfaces based on configuration...
k8s-ctr: Adapter 1: nat
k8s-ctr: Adapter 2: hostonly
==> k8s-ctr: Forwarding ports...
k8s-ctr: 22 (guest) => 60000 (host) (adapter 1)
==> k8s-ctr: Running 'pre-boot' VM customizations...
==> k8s-ctr: Booting VM...
==> k8s-ctr: Waiting for machine to boot. This may take a few minutes...
k8s-ctr: SSH address: 127.0.0.1:60000
k8s-ctr: SSH username: vagrant
k8s-ctr: SSH auth method: private key
k8s-ctr:
k8s-ctr: Vagrant insecure key detected. Vagrant will automatically replace
k8s-ctr: this with a newly generated keypair for better security.
k8s-ctr:
k8s-ctr: Inserting generated public key within guest...
k8s-ctr: Removing insecure key from the guest if it's present...
k8s-ctr: Key inserted! Disconnecting and reconnecting using new SSH key...
==> k8s-ctr: Machine booted and ready!
==> k8s-ctr: Checking for guest additions in VM...
==> k8s-ctr: Setting hostname...
==> k8s-ctr: Configuring and enabling network interfaces...
==> k8s-ctr: Running provisioner: shell...
k8s-ctr: Running: /var/folders/7n/xqp3j0xs2_d5sn8y8mj5wqkh0000gn/T/vagrant-shell20250127-29384-1v5j8e.sh
k8s-ctr: >>>> Initial Config Start <<
k8s-ctr: [TASK 1] Setting Profile & Bashrc
k8s-ctr: [TASK 2] Disable AppArmor
k8s-ctr: [TASK 3] Disable and turn off SWAP
k8s-ctr: [TASK 4] Install Packages
k8s-ctr: [TASK 5] Install Kubernetes components (kubeadm, kubelet and kubectl)
k8s-ctr: [TASK 6] Install Packages & Helm
k8s-ctr: >>>> Initial Config End <<
==> k8s-ctr: Running provisioner: shell...
k8s-ctr: Running: /var/folders/7n/xqp3j0xs2_d5sn8y8mj5wqkh0000gn/T/vagrant-shell20250127-29384-qj3n9w.sh
k8s-ctr: >>>> K8S Controlplane config Start <<
k8s-ctr: [TASK 1] Initial Kubernetes
k8s-ctr: [TASK 2] Setting kube config file
k8s-ctr: [TASK 3] Source the completion
k8s-ctr: [TASK 4] Alias kubectl to k
k8s-ctr: [TASK 5] Install Kubectx & Kubens
k8s-ctr: [TASK 6] Install Kubeps & Setting PS1
k8s-ctr: [TASK 7] Install Cilium CNI
k8s-ctr: [TASK 8] Install Cilium CLI
k8s-ctr: [TASK 9] local DNS with hosts file
k8s-ctr: >>>> K8S Controlplane Config End <<I can see the worker nodes coming up too:
==> k8s-w1: Importing base box 'bento/ubuntu-24.04'...
==> k8s-w1: Matching MAC address for NAT networking...
==> k8s-w1: Setting the name of the VM: k8s-w1
==> k8s-w1: Clearing any previously set network interfaces...
==> k8s-w1: Preparing network interfaces based on configuration...
k8s-w1: Adapter 1: nat
k8s-w1: Adapter 2: hostonly
==> k8s-w1: Forwarding ports...
k8s-w1: 22 (guest) => 60001 (host) (adapter 1)
==> k8s-w1: Running 'pre-boot' VM customizations...
==> k8s-w1: Booting VM...
==> k8s-w1: Waiting for machine to boot. This may take a few minutes...
k8s-w1: SSH address: 127.0.0.1:60001
k8s-w1: SSH username: vagrant
k8s-w1: SSH auth method: private key
==> k8s-w1: Machine booted and ready!
==> k8s-w1: Running provisioner: shell...
k8s-w1: Running: /var/folders/7n/xqp3j0xs2_d5sn8y8mj5wqkh0000gn/T/vagrant-shell20250127-29384-18hqk3o.sh
k8s-w1: >>>> Initial Config Start <<
k8s-w1: [TASK 1] Setting Profile & Bashrc
k8s-w1: [TASK 2] Disable AppArmor
k8s-w1: [TASK 3] Disable and turn off SWAP
k8s-w1: [TASK 4] Install Packages
k8s-w1: [TASK 5] Install Kubernetes components (kubeadm, kubelet and kubectl)
k8s-w1: [TASK 6] Install Packages & Helm
k8s-w1: >>>> Initial Config End <<
==> k8s-w1: Running provisioner: shell...
k8s-w1: Running: /var/folders/7n/xqp3j0xs2_d5sn8y8mj5wqkh0000gn/T/vagrant-shell20250127-29384-1dkr8jw.sh
k8s-w1: >>>> K8S Node config Start <<
k8s-w1: [TASK 1] K8S Controlplane Join
k8s-w1: >>>> K8S Node config End <<Worker 2 comes up the same way. The whole thing takes about 12 minutes on my machine. Once it’s done, let me jump into the control plane:
❯ vagrant ssh k8s-ctr
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-53-generic x86_64)Nice! I love that the prompt already shows my Kubernetes context. Let me check if everything’s working:(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-ctr Ready control-plane 3m42s v1.33.2
k8s-w1 Ready <none> 2m24s v1.33.2
k8s-w2 Ready <none> 63s v1.33.2Sweet, all nodes are ready. Let me check what’s running:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-envoy-m7njw 1/1 Running 0 3m51s
kube-system cilium-envoy-p42gk 1/1 Running 0 75s
kube-system cilium-envoy-pxr2q 1/1 Running 0 2m37s
kube-system cilium-kkt6t 1/1 Running 0 75s
kube-system cilium-mm7xq 1/1 Running 0 2m37s
kube-system cilium-operator-5bc66f5b9b-7v5fg 1/1 Running 0 3m50s
kube-system cilium-tstkf 1/1 Running 0 3m51s
kube-system coredns-674b8bbfcf-2sbw2 1/1 Running 0 3m50s
kube-system coredns-674b8bbfcf-p8kth 1/1 Running 0 3m50s
kube-system etcd-k8s-ctr 1/1 Running 0 3m57s
kube-system kube-apiserver-k8s-ctr 1/1 Running 0 3m57s
kube-system kube-controller-manager-k8s-ctr 1/1 Running 0 3m57s
kube-system kube-scheduler-k8s-ctr 1/1 Running 0 3m57snotice there’s no kube-proxy pods. That’s because Cilium is running in full replacement mode.
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ds -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
cilium 3 3 3 3 3 kubernetes.io/os=linux 4m16s
cilium-envoy 3 3 3 3 3 kubernetes.io/os=linux 4m16sLet’s check Cilium’s status.
(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabledDeployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet cilium-envoy Desired: 3, Ready: 3/3, Available: 3/3
Containers: cilium Running: 3
cilium-envoy Running: 3
cilium-operator Running: 1
Cluster Pods: 2/2 managed by Cilium
Helm chart version: 1.17.6
Image versions cilium quay.io/cilium/cilium:v1.17.6@sha256:544de3d4fed7acba72758413812780a4972d47c39035f2a06d6145d8644a3353: 3
cilium-envoy quay.io/cilium/cilium-envoy:v1.33.4-1752151664-7c2edb0b44cf95f326d628b837fcdd845102ba68@sha256:318eff387835ca2717baab42a84f35a83a5f9e7d519253df87269f80b9ff0171: 3
cilium-operator quay.io/cilium/operator-generic:v1.17.6@sha256:91ac3bf7be7bed30e90218f219d4f3062a63377689ee7246062fa0cc3839d096: 1Everything’s green! Let me dig deeper and see what Cilium’s actually configured with:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium config view | grep -E "cluster-pool|routing-mode|kube-proxy|masquerade|hubble"
cluster-pool-ipv4-cidr 172.20.0.0/16
cluster-pool-ipv4-mask-size 24
enable-hubble false
hubble-disable-tls false
hubble-export-file-max-backups 5
hubble-export-file-max-size-mb 10
hubble-listen-address :4244
hubble-metrics-server :9962
hubble-socket-path /var/run/cilium/hubble.sock
hubble-tls-cert-file /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file /var/lib/cilium/tls/hubble/server.key
kube-proxy-replacement true
kube-proxy-replacement-healthz-bind-address
routing-mode native
enable-bpf-masquerade true
enable-ipv4-big-tcp false
enable-ipv4-masquerade true
enable-ipv6-big-tcp false
enable-ipv6-masquerade true
enable-masquerade-to-route-source falseSo we’re running in native routing mode with full kube-proxy replacement. This means all the service load balancing is happening in eBPF, not iptables. Speaking of which, let me check the iptables situation:
(⎈|HomeLab:N/A) root@k8s-ctr:~# iptables -t nat -S | head -20
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N CILIUM_OUTPUT_nat
-N CILIUM_POST_nat
-N CILIUM_PRE_nat
-N KUBE-FIREWALL
-A PREROUTING -m comment --comment "cilium-feeder: CILIUM_PRE_nat" -j CILIUM_PRE_nat
-A OUTPUT -m comment --comment "cilium-feeder: CILIUM_OUTPUT_nat" -j CILIUM_OUTPUT_nat
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "cilium-feeder: CILIUM_POST_nat" -j CILIUM_POST_nat
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A CILIUM_OUTPUT_nat -d 127.0.0.1/32 -m comment --comment "cilium: SNAT for traffic to 127.0.0.1" -j SNAT --to-source 127.0.0.1
-A CILIUM_OUTPUT_nat -s 127.0.0.1/32 -m comment --comment "cilium: skip localhost traffic (autodetected)" -j ACCEPT
-A CILIUM_OUTPUT_nat -m comment --comment "cilium: host->cluster redirects are not subject to SNAT" -j ACCEPT
-A CILIUM_OUTPUT_nat -m comment --comment "cilium: host->cluster from 172.20.0.106 redirects are not subject to SNAT" -j ACCEPT
-A CILIUM_POST_nat -o cilium_host -m comment --comment "cilium: host->cluster from 172.20.0.106 masquerade" -j MASQUERADE
-A CILIUM_POST_nat -s 127.0.0.1/32 -m comment --comment "cilium: skip localhost traffic (autodetected)" -j ACCEPT
-A CILIUM_POST_nat -s 172.20.0.0/16 ! -d 172.20.0.0/16 ! -o cilium_+ -m comment --comment "cilium masquerade non-cluster" -j MASQUERADEWay cleaner than traditional kube-proxy! No massive chains of service rules. Let me check what network interfaces Cilium created:
(⎈|HomeLab:N/A) root@k8s-ctr:~# ip link show | grep -A1 "cilium\|lxc"
6: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether da:59:c4:28:b5:56 brd ff:ff:ff:ff:ff:ff
7: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 6e:98:17:26:d8:65 brd ff:ff:ff:ff:ff:ff
8: lxc_health@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 5e:77:dd:5e:d8:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: lxcea4f8900288d@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 26:70:69:2b:3c:45 brd ff:ff:ff:ff:ff:ff link-netnsid 1
12: lxc903c835e21af@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether b6:16:31:6c:65:97 brd ff:ff:ff:ff:ff:ff link-netnsid 2Those lxc interfaces are the veth pairs connecting to pods. Let me see what IPs Cilium assigned:
(⎈|HomeLab:N/A) root@k8s-ctr:~# ip addr show cilium_host
7: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 6e:98:17:26:d8:65 brd ff:ff:ff:ff:ff:ff
inet 172.20.0.106/32 scope global cilium_host
valid_lft forever preferred_lft forever
inet6 fe80::6c98:17ff:fe26:d865/64 scope link
valid_lft forever preferred_lft foreverThat’s the Cilium router IP. Now let me check the actual pod IPs:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pods -A -o wide | grep -v "192.168"
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-674b8bbfcf-2sbw2 1/1 Running 0 6m54s 172.20.0.51 k8s-ctr <none> <none>
kube-system coredns-674b8bbfcf-p8kth 1/1 Running 0 6m54s 172.20.0.107 k8s-ctr <none> <none>Those CoreDNS pods got IPs from Cilium’s IPAM pool (172.20.0.0/16), not the default Kubernetes pod CIDR. Let me check how Cilium allocated the subnets:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg status --verbose | grep -A20 "IPAM"
KubeProxyReplacement: True [eth0 10.0.2.15, eth1 192.168.10.100 (Direct Routing), eth2 (Disabled)]
Host firewall: Disabled
CNI Config Check: Disabled
CNI Config Event: Enabled
SockLB: Disabled
IPAM: IPv4: 3/254 allocated from 172.20.0.0/24,
IPv4 BIG TCP: Disabled
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Host Routing: BPF
Masquerading: BPF [eth0, eth1] [IPv4: Enabled, IPv6: Disabled]
Controller AIOps: 1/1 active
Clock Source for BPF: ktime
Devices: eth0, eth1 (Direct Routing)
Attach Mode: TC
Cilium: Ok Operator: Ok Envoy DaemonSet: Ok
Allocated addresses:
172.20.0.106 (cilium_host)
172.20.0.107 (health)
172.20.0.51 (router)So on the control plane, we’ve allocated 3 out of 254 possible IPs from the /24 subnet. Let me check what the worker nodes got:
(⎈|HomeLab:N/A) root@k8s-ctr:~# for node in k8s-w1 k8s-w2; do
> echo "=== Node: $node ==="
> kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg status --verbose | grep -A5 "IPAM" | grep -A5 "allocated from"
> done
=== Node: k8s-w1 ===
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
=== Node: k8s-w2 ===
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)Hmm, that didn’t work as expected. Let me try a different approach:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ciliumnode -o json | jq '.items[] | {name: .metadata.name, podCIDRs: .spec.ipam.podCIDRs}'
{
"name": "k8s-ctr",
"podCIDRs": [
"172.20.0.0/24"
]
}
{
"name": "k8s-w1",
"podCIDRs": [
"172.20.1.0/24"
]
}
{
"name": "k8s-w2",
"podCIDRs": [
"172.20.2.0/24"
]
}Perfect! Each node got its own /24 from the cluster pool. Now let’s have some fun and watch the traffic flow in real-time:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg monitor
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> endpoint 2609 flow 0x3b3eab76 , identity host->5694 state new ifindex lxc903c835e21af orig-ip 10.0.2.15: 10.0.2.15:56596 -> 172.20.0.107:8080 tcp SYN
-> stack flow 0x88aae639 , identity 5694->host state reply ifindex 0 orig-ip 0.0.0.0: 172.20.0.107:8080 -> 10.0.2.15:56596 tcp SYN, ACK
-> endpoint 2609 flow 0x3b3eab76 , identity host->5694 state established ifindex lxc903c835e21af orig-ip 10.0.2.15: 10.0.2.15:56596 -> 172.20.0.107:8080 tcp ACK
-> endpoint 2609 flow 0x3b3eab76 , identity host->5694 state established ifindex lxc903c835e21af orig-ip 10.0.2.15: 10.0.2.15:56596 -> 172.20.0.107:8080 tcp ACK
-> stack flow 0x88aae639 , identity 5694->host state reply ifindex 0 orig-ip 0.0.0.0: 172.20.0.107:8080 -> 10.0.2.15:56596 tcp ACKThis is showing traffic between the host and CoreDNS (identity 5694). The flow tracking is super detailed — you can see the full TCP handshake. Let me generate some DNS traffic:
(⎈|HomeLab:N/A) root@k8s-ctr:~# dig @10.96.0.10 kubernetes.default.svc.cluster.local; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @10.96.0.10 kubernetes.default.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39875
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: f5e8c9e5f9a5d037 (echoed)
;; QUESTION SECTION:
;kubernetes.default.svc.cluster.local. IN A;; ANSWER SECTION:
kubernetes.default.svc.cluster.local. 30 IN A 10.96.0.1;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10) (UDP)
;; WHEN: Mon Jan 27 18:38:45 UTC 2025
;; MSG SIZE rcvd: 129And in the monitor window I can see:
-> stack flow 0xd76eab93 , identity world->5694 state new ifindex 0 orig-ip 0.0.0.0: 192.168.10.100:45837 -> 172.20.0.107:53 udp
<- endpoint 2609 flow 0xec2e1fb9 , identity 5694->world state reply ifindex lxc903c835e21af orig-ip 192.168.10.100: 172.20.0.107:53 -> 192.168.10.100:45837 udpLet me check the Cilium endpoints that are being managed:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ciliumendpoints -A -o wide
NAMESPACE NAME SECURITY IDENTITY ENDPOINT STATE IPV4 IPV6
kube-system coredns-674b8bbfcf-2sbw2 5694 ready 172.20.0.51
kube-system coredns-674b8bbfcf-p8kth 5694 ready 172.20.0.107Both CoreDNS pods share the same security identity (5694). Let me see what that identity means:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg identity get 5694
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
ID LABELS
5694 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
k8s:io.cilium.k8s.namespace.labels.pod-template-hash=674b8bbfcf
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dnsNow let’s see what BPF maps are loaded:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg bpf map list | head -20
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
Name Pins Entries Flags & Extras
cilium_ktime_cache 1 1 0x00000000
cilium_runtime_config 2 18 0x00000000
cilium_node_map 2 3 0x00000000
cilium_snat_v4_external 1 0 0x00000000
cilium_lb4_reverse_nat 1 6 0x00000001
max entries: 524288
cilium_lb4_services_v2 1 6 0x00000001
max entries: 65536
cilium_lb4_backends_v3 1 5 0x00000000
max entries: 65536
cilium_lb4_reverse_sk 1 0 0x00000009
lru, max entries: 262144, sync
cilium_lb4_source_range 1 0 0x00000019
lru, lpm_trie, max entries: 524288, sync
cilium_ipmasq_v4 1 2 0x00000001
max entries: 16384These maps are where all the magic happens. Service load balancing, NAT, connection tracking — it’s all in these BPF maps. Let me check what services are being load balanced:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg service list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
ID Frontend Service Type Backend
2 10.96.0.1:443 ClusterIP 1 => 192.168.10.100:6443 (active)
3 10.96.0.10:53 ClusterIP 1 => 172.20.0.51:53 (active)
2 => 172.20.0.107:53 (active)
4 10.96.0.10:9153 ClusterIP 1 => 172.20.0.51:9153 (active)
2 => 172.20.0.107:9153 (active)There’s our Kubernetes API server and the CoreDNS service with load balancing across two backends. Let me take a look at the more verbose monitoring to see packet contents:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg monitor -v -v --hex | head -50
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
Listening for events on 2 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
------------------------------------------------------------------------------
CPU 00: MARK 0x0 FROM 0 to-endpoint: 50 bytes (50 captured), state new, interface , , identity world->5694, orig-ip 0.0.0.0
00000000 26 70 69 2b 3c 45 00 00 00 32 2f 05 40 00 40 06 |&pi+<E...2/.@.@.|
00000010 a3 40 c0 a8 0a 64 ac 14 00 6b b2 62 1f 91 45 69 |.@...d...k.b..Ei|
00000020 e9 b5 41 96 c0 a5 80 10 02 00 9b f8 00 00 01 01 |..A.............|
00000030 08 0a 0f ee b9 b3 26 09 d0 0e |......&...|CPU 00: MARK 0x7086d59 FROM 2609 to-endpoint: 66 bytes (66 captured), state established, interface lxc903c835e21af, , identity host->5694, orig-ip 10.0.2.15
00000000 b6 16 31 6c 65 97 26 70 69 2b 3c 45 08 00 45 00 |..1le.&pi+<E..E.|
00000010 00 34 5a 38 40 00 40 06 78 09 0a 00 02 0f ac 14 |.4Z8@.@.x.......|
00000020 00 6b dc 78 1f 90 1f 71 d1 32 26 20 17 4f 80 18 |.k.x...q.2& .O..|
00000030 01 f6 95 6a 00 00 01 01 08 0a c7 13 7b bf 8a b2 |...j........{...|
00000040 ee 1e |..|I can actually see the packet bytes! Let me check if there are any drops happening:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg monitor --type drop
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitorNothing’s being dropped, which is good. Let me look at the routing table to understand how packets flow:
(⎈|HomeLab:N/A) root@k8s-ctr:~# ip route
default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
172.20.0.0/24 via 172.20.0.106 dev cilium_host src 172.20.0.106
172.20.1.0/24 via 192.168.10.101 dev eth1 src 192.168.10.100
172.20.2.0/24 via 192.168.10.102 dev eth1 src 192.168.10.100
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.100This is native routing at work! Traffic to worker node pod subnets goes directly via eth1, no overlay needed. Let me verify the cluster networking configuration one more time:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl cluster-info dump | grep -E "cluster-cidr|service-cluster-ip-range" | head -5
"--service-cluster-ip-range=10.96.0.0/16",
"--cluster-cidr=10.244.0.0/16",
"--service-cluster-ip-range=10.96.0.0/16",Even though kubeadm configured 10.244.0.0/16 as the pod CIDR, Cilium is completely ignoring it and using its own 172.20.0.0/16. Let me ssh into a worker node to see its perspective:
(⎈|HomeLab:N/A) root@k8s-ctr:~# ssh k8s-w1
The authenticity of host 'k8s-w1 (192.168.10.101)' can't be established.
ED25519 key fingerprint is SHA256:Q5KpCjB3mzHvPJhY8xvDt9mFKJ9lHGH7iE9QH5QKRIQ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'k8s-w1' (ED25519) to the list of known hosts.
vagrant@k8s-w1's password: vagrantWelcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-53-generic x86_64)
vagrant@k8s-w1:~$ sudo su -
root@k8s-w1:~# ip route
default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
172.20.0.0/24 via 192.168.10.100 dev eth1 src 192.168.10.101
172.20.1.0/24 via 172.20.1.62 dev cilium_host src 172.20.1.62
172.20.2.0/24 via 192.168.10.102 dev eth1 src 192.168.10.101
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.101Perfect symmetry! Each node knows how to reach every other node’s pod subnet directly. Let me check the cilium status on this worker:
root@k8s-w1:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg status --verbose | grep -A10 "Allocated addresses"
Allocated addresses:
172.20.1.62 (router)
172.20.1.33 (health)
Cluster health: 3/3 reachable (2025-01-27T18:45:42Z)
Name IP Node Endpoints
k8s-ctr (localhost) 192.168.10.100 reachable reachable
k8s-w1 192.168.10.101 reachable reachable
k8s-w2 192.168.10.102 reachable reachableThe cluster health check shows all nodes are reachable. This is running over the native network, no tunnels. Let me go back to the control plane and check the iptables in more detail:
root@k8s-w1:~# exit
logout
vagrant@k8s-w1:~$ exit
logout
(⎈|HomeLab:N/A) root@k8s-ctr:~# iptables-save | grep -c KUBE-
13
(⎈|HomeLab:N/A) root@k8s-ctr:~# iptables-save | grep -c CILIUM
60Only 13 kube- rules compared to 60 Cilium rules. In a traditional kube-proxy setup, you’d see hundreds or thousands of KUBE- rules. Let me look at what those remaining KUBE- rules are:
(⎈|HomeLab:N/A) root@k8s-ctr:~# iptables-save | grep KUBE- | head -10
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
:KUBE-FORWARD - [0:0]
:KUBE-NODEPORTS - [0:0]These are mostly empty chains left for compatibility. The real work is done in eBPF. Let me create a test pod to see Cilium in action:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl run test-pod --image=nicolaka/netshoot:latest --command -- sleep 3600
pod/test-pod created(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pod test-pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-pod 1/1 Running 0 23s 172.20.1.45 k8s-w1 <none> <none>It got scheduled to worker 1 and received an IP from Cilium’s pool. Let me watch what happens when I exec into it:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it test-pod -- /bin/bash
bash-5.2# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 1e:12:b7:c4:a0:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.20.1.45/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::1c12:b7ff:fec4:a01f/64 scope link
valid_lft forever preferred_lft foreverNotice the /32 IP assignment? That’s Cilium’s way — each pod gets a /32 with routing handled by the cilium_host interface. Let me test connectivity:
bash-5.2# ping -c 2 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=10.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=9.63 ms--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 9.630/9.867/10.105/0.237 msbash-5.2# nslookup kubernetes.default
Server: 10.96.0.10
Address: 10.96.0.10#53Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1bash-5.2# curl -k https://kubernetes.default:443/version
{
"major": "1",
"minor": "33",
"gitVersion": "v1.33.2",
"gitCommit": "33d5cb65cc5d97c837d070bb3c1856e52f2c0c84",
"gitTreeState": "clean",
"buildDate": "2025-01-14T11:22:31Z",
"goVersion": "go1.23.4",
"compiler": "gc",
"platform": "linux/amd64"
}
bash-5.2# exit
exit
Everything works! DNS resolution, external connectivity, and service discovery all functioning through Cilium’s eBPF datapath. Let me check one more thing — the Layer 7 visibility:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg monitor -v --type l7
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
CPU 00: [pre-xlate-rev] cgroup_id: 7184 sock_cookie: 18019, dst [127.0.0.1]:34040 tcp
CPU 00: [pre-xlate-fwd] cgroup_id: 7184 sock_cookie: 18019, dst [172.20.0.51]:8080 tcp
CPU 01: [pre-xlate-rev] cgroup_id: 9357 sock_cookie: 13977, dst [127.0.0.1]:8080 tcp
CPU 01: [sock-connect] cgroup_id: 9357 sock_cookie: 13977, dst [172.20.0.51]:8080 tcp , status: 0I can see the socket-level operations and cgroup associations. That cgroup_id maps to specific containers, giving us fine-grained visibility into who’s making what connections.
Hubble Configuration
First, let me check what state we’re in. I always like to document the before and after when I’m making changes:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabledDeployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet cilium-envoy Desired: 3, Ready: 3/3, Available: 3/3
Containers: cilium Running: 3
cilium-envoy Running: 3
cilium-operator Running: 1
clustermesh-apiserver
hubble-relay
Cluster Pods: 2/2 managed by CiliumYeah, Hubble Relay is disabled. Let me also check the config:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium config view | grep -i hubble
enable-hubble false
hubble-disable-tls false
hubble-export-file-max-backups 5
hubble-export-file-max-size-mb 10
hubble-listen-address :4244
hubble-metrics-server :9962
hubble-socket-path /var/run/cilium/hubble.sock
hubble-tls-cert-file /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file /var/lib/cilium/tls/hubble/server.keySo Hubble is completely off. Let me check if there are any certificates or secrets already:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get secret -n kube-system | grep -iE 'cilium-ca|hubble'Nothing. And let me save the current port state so I can see what changes:
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -tnlp | grep -iE 'cilium|hubble' | tee before.txt
LISTEN 0 4096 127.0.0.1:9891 0.0.0.0:* users:(("cilium-operator",pid=8435,fd=7))
LISTEN 0 4096 127.0.0.1:9890 0.0.0.0:* users:(("cilium-agent",pid=9298,fd=21))
LISTEN 0 4096 127.0.0.1:9879 0.0.0.0:* users:(("cilium-agent",pid=9298,fd=14))
LISTEN 0 4096 127.0.0.1:9878 0.0.0.0:* users:(("cilium-envoy",pid=9360,fd=26))
LISTEN 0 4096 0.0.0.0:9964 0.0.0.0:* users:(("cilium-envoy",pid=9360,fd=24))
LISTEN 0 4096 127.0.0.1:41353 0.0.0.0:* users:(("cilium-agent",pid=9298,fd=18))
LISTEN 0 4096 127.0.0.1:9234 0.0.0.0:* users:(("cilium-operator",pid=8435,fd=6))
LISTEN 0 4096 [::]:9963 [::]:* users:(("cilium-operator",pid=8435,fd=19))No Hubble ports (4244, 4245) anywhere. Now let me enable it. I could use the Cilium CLI for a quick enable, but I prefer Helm because I want to set specific options:
(⎈|HomeLab:N/A) root@k8s-ctr:~# helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
> --set hubble.enabled=true \
> --set hubble.relay.enabled=true \
> --set hubble.ui.enabled=true \
> --set hubble.ui.service.type=NodePort \
> --set hubble.ui.service.nodePort=31234 \
> --set hubble.export.static.enabled=true \
> --set hubble.export.static.filePath=/var/run/cilium/hubble/events.log \
> --set prometheus.enabled=true \
> --set operator.prometheus.enabled=true \
> --set hubble.metrics.enableOpenMetrics=true \
> --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}"
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Sat Jul 26 01:59:21 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.Your release version is 1.17.6.For any further help, visit https://docs.cilium.io/en/v1.17/gettinghelp
Now let me wait a bit for everything to come up. This usually takes a minute or two:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium status --wait
⠧ Waiting for Cilium to be ready...
⠏ Waiting for Cilium to be ready...
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabledDeployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet cilium-envoy Desired: 3, Ready: 3/3, Available: 3/3
Containers: cilium Running: 3
cilium-envoy Running: 3
cilium-operator Running: 1
hubble-relay Running: 1
hubble-ui Running: 1
Cluster Pods: 4/4 managed by CiliumExcellent! Hubble Relay is now OK. Let me verify the configuration changed:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium config view | grep -i hubble
bpf-ct-timeout-service-tcp 21600
bpf-lb-map-max 65536
cluster-pool-ipv4-cidr 172.20.0.0/16
cluster-pool-ipv4-mask-size 24
enable-hubble true
enable-hubble-open-metrics true
hubble-disable-tls false
hubble-export-file-max-backups 5
hubble-export-file-max-size-mb 10
hubble-export-file-path /var/run/cilium/hubble/events.log
hubble-listen-address :4244
hubble-metrics dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
hubble-metrics-server :9965
hubble-socket-path /var/run/cilium/hubble.sock
hubble-tls-cert-file /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file /var/lib/cilium/tls/hubble/server.keyGood, enable-hubble is now true. Let me check if the certificates got generated:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get secret -n kube-system | grep -iE 'cilium-ca|hubble'
cilium-ca Opaque 2 27s
hubble-relay-client-certs kubernetes.io/tls 3 27s
hubble-server-certs kubernetes.io/tls 3 27sNice! Three secrets created automatically. Now let me check the network ports again:
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -tnlp | grep -iE 'cilium|hubble' | tee after.txt
LISTEN 0 4096 127.0.0.1:9891 0.0.0.0:* users:(("cilium-operator",pid=10333,fd=6))
LISTEN 0 4096 [::]:9965 [::]:* users:(("cilium-agent",pid=10484,fd=42))
LISTEN 0 4096 127.0.0.1:9890 0.0.0.0:* users:(("cilium-agent",pid=10484,fd=21))
LISTEN 0 4096 127.0.0.1:9879 0.0.0.0:* users:(("cilium-agent",pid=10484,fd=14))
LISTEN 0 4096 127.0.0.1:9878 0.0.0.0:* users:(("cilium-envoy",pid=10546,fd=26))
LISTEN 0 4096 0.0.0.0:9964 0.0.0.0:* users:(("cilium-envoy",pid=10546,fd=24))
LISTEN 0 4096 127.0.0.1:41353 0.0.0.0:* users:(("cilium-agent",pid=10484,fd=18))
LISTEN 0 4096 127.0.0.1:9234 0.0.0.0:* users:(("cilium-operator",pid=10333,fd=5))
LISTEN 0 4096 [::]:9963 [::]:* users:(("cilium-operator",pid=10333,fd=23))
LISTEN 0 4096 [::]:9962 [::]:* users:(("cilium-agent",pid=10484,fd=7))
LISTEN 0 4096 *:4244 *:* users:(("cilium-agent",pid=10484,fd=50))There it is! Port 4244 is now listening on all interfaces. Let me check if the worker nodes also have it:
(⎈|HomeLab:N/A) root@k8s-ctr:~# for i in w1 w2 ; do echo ">> node : k8s-$i <<"; sshpass -p 'vagrant' ssh vagrant@k8s-$i sudo ss -tnlp |grep 4244 ; echo; done
>> node : k8s-w1
LISTEN 0 4096 *:4244 *:* users:(("cilium-agent",pid=7154,fd=55))>> node : k8s-w2
LISTEN 0 4096 *:4244 *:* users:(("cilium-agent",pid=6930,fd=42))Perfect! All nodes are listening on 4244. Let me check the Hubble Relay pod:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pod -n kube-system -l k8s-app=hubble-relay
NAME READY STATUS RESTARTS AGE
hubble-relay-5dcd46f5c-tln2s 1/1 Running 0 2m14s(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe pod -n kube-system -l k8s-app=hubble-relay | grep -A10 "Containers:"
Containers:
hubble-relay:
Container ID: containerd://ced5fc89e1f3c9a51e0fbf1abafd87b96a45cf96677dcb22e2e0f4b2de96743d
Image: quay.io/cilium/hubble-relay:v1.17.6@sha256:7d17ec10b3d37341c18ca56165b2f29a715cb8ee81311fd07088d8bf68c01e60
Image ID: quay.io/cilium/hubble-relay@sha256:7d17ec10b3d37341c18ca56165b2f29a715cb8ee81311fd07088d8bf68c01e60
Port: 4245/TCP
Host Port: 0/TCP
Command:
hubble-relay
Args:
serveThe relay is listening on port 4245. Let me check its service:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc,ep -n kube-system hubble-relay
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hubble-relay ClusterIP 10.96.174.49 <none> 80/TCP 2m22sNAME ENDPOINTS AGE
endpoints/hubble-relay 172.20.2.176:4245 2m22sNow here’s the important part — how does Hubble Relay connect to all the nodes? Let me check:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe cm -n kube-system hubble-relay-config
Name: hubble-relay-config
Namespace: kube-system
Labels: app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=hubble-relay
app.kubernetes.io/part-of=cilium
k8s-app=hubble-relay
Annotations: meta.helm.sh/release-name: cilium
meta.helm.sh/release-namespace: kube-systemData
====
config.yaml:
----
cluster-name: default
peer-service: "hubble-peer.kube-system.svc.cluster.local.:443"
listen-address: :4245
gops: true
gops-port: "9893"
metrics-listen-address: ":9966"
dial-timeout:
retry-timeout:
sort-buffer-len-max:
sort-buffer-drain-timeout:
tls-client-cert-file: /var/lib/hubble-relay/tls/client.crt
tls-client-key-file: /var/lib/hubble-relay/tls/client.key
tls-server-cert-file: /var/lib/hubble-relay/tls/server.crt
tls-server-key-file: /var/lib/hubble-relay/tls/server.key
disable-server-tls: trueAh, so it connects to hubble-peer service. Let me check that:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc,ep -n kube-system hubble-peer
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hubble-peer ClusterIP 10.96.68.0 <none> 443/TCP 2m35sNAME ENDPOINTS AGE
endpoints/hubble-peer 192.168.10.100:4244,192.168.10.101:4244,192.168.10.102:4244 2m35sThis is clever! The hubble-peer service automatically includes all nodes’ port 4244 as endpoints. So the Relay can aggregate data from all nodes through this service.
Now let me check the Hubble UI:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pod -n kube-system -l k8s-app=hubble-ui
NAME READY STATUS RESTARTS AGE
hubble-ui-76d4965bb6-hcgj6 2/2 Running 0 3m9s(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe pod -n kube-system -l k8s-app=hubble-ui | grep -A20 "Containers:"
Containers:
frontend:
Container ID: containerd://22b825aef23f019c088c3834cc07c31b5df92f3e1c12d1ec093dd2cc1e72802d
Image: quay.io/cilium/hubble-ui:v0.13.2@sha256:9e37c1296b802830834cc87342a9182ccbb71ffebb711971e849221bd9d59392
Image ID: quay.io/cilium/hubble-ui@sha256:9e37c1296b802830834cc87342a9182ccbb71ffebb711971e849221bd9d59392
Port: 8081/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 26 Jul 2025 02:00:00 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 25m
memory: 64Mi
Environment:
CILIUM_VERSION: <set to the key 'value' of config map 'cilium-version-info'> Optional: true
Mounts:
/app from hubble-ui-nginx-conf (rw)
/tmp from tmp-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5txpd (ro)
backend:
Container ID: containerd://49b8dd0b3e83b3b09e60f97da0df97ba03b962e5eab1f1f17b90ad43b86e87e9
Image: quay.io/cilium/hubble-ui-backend:v0.13.2@sha256:a034b7e98e6ea796ed26df8f4e71f83fc16465a19d166eff67a03b822c0bfa15
Image ID: quay.io/cilium/hubble-ui-backend@sha256:a034b7e98e6ea796ed26df8f4e71f83fc16465a19d166eff67a03b822c0bfa15
Port: 8090/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 26 Jul 2025 02:00:00 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 25m
memory: 64Mi
Environment:
EVENTS_SERVER_PORT: 8090
FLOWS_API_ADDR: hubble-relay:80So the UI has two containers — frontend (nginx on 8081) and backend (go server on 8090). The backend connects to hubble-relay:80. Let me check the UI service:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc,ep -n kube-system hubble-ui
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hubble-ui NodePort 10.96.201.178 <none> 80:31234/TCP 2m48sNAME ENDPOINTS AGE
endpoints/hubble-ui 172.20.1.129:8081 2m48sGreat! NodePort 31234 is exposed. Let me get the URL:
(⎈|HomeLab:N/A) root@k8s-ctr:~# NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
(⎈|HomeLab:N/A) root@k8s-ctr:~# echo -e "http://$NODEIP:31234"
http://192.168.10.100:31234Before I access the web UI, let me install the Hubble CLI so I can use the command line too:
(⎈|HomeLab:N/A) root@k8s-ctr:~# HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
(⎈|HomeLab:N/A) root@k8s-ctr:~# HUBBLE_ARCH=amd64
(⎈|HomeLab:N/A) root@k8s-ctr:~# curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 164 100 164 0 0 533 0 --:--:-- --:--:-- --:--:-- 534
100 655 100 655 0 0 1324 0 --:--:-- --:--:-- --:--:-- 1324
100 22.3M 100 22.3M 0 0 13.9M 0 0:00:01 0:00:01 --:--:-- 33.0M
100 164 100 164 0 0 491 0 --:--:-- --:--:-- --:--:-- 491
100 97 100 97 0 0 198 0 --:--:-- --:--:-- --:--:-- 198(⎈|HomeLab:N/A) root@k8s-ctr:~# sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
hubble
LICENSE
README.md(⎈|HomeLab:N/A) root@k8s-ctr:~# which hubble
/usr/local/bin/hubble
Now let me try to use it:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble status
failed getting status: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4245: connect: connection refused"Oh right, I need to port-forward first because the hubble CLI expects to connect to localhost:4245:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cilium hubble port-forward&
[1] 10570
(⎈|HomeLab:N/A) root@k8s-ctr:~# ℹ️ Hubble Relay is available at 127.0.0.1:4245(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -tnlp | grep 4245
LISTEN 0 4096 127.0.0.1:4245 0.0.0.0:* users:(("cilium",pid=10570,fd=7))
LISTEN 0 4096 [::1]:4245 [::]:* users:(("cilium",pid=10570,fd=8))Now let me try again:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 11,282/12,285 (91.84%)
Flows/s: 35.22
Connected Nodes: 3/3Awesome! It’s working. I can see it’s processing about 35 flows per second and the buffer is 91% full. Let me check what settings hubble CLI is using:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble config view
port-forward-port: "4245"
server: localhost:4245
timeout: 5s
tls: false
tls-allow-insecure: false
tls-ca-cert-files: []
tls-client-cert-file: ""
tls-client-key-file: ""
tls-server-name: ""Now for the fun part — let me watch some real-time traffic:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe
Jul 25 17:13:13.125: kube-system/hubble-ui-76d4965bb6-hcgj6:55468 (ID:7661) -> kube-system/hubble-relay-5dcd46f5c-tln2s:4245 (ID:11581) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:13:13.125: kube-system/hubble-relay-5dcd46f5c-tln2s:42624 (ID:11581) <- 192.168.10.101:4244 (remote-node) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:13:13.125: kube-system/hubble-relay-5dcd46f5c-tln2s:37400 (ID:11581) <- 192.168.10.102:4244 (host) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:13:13.125: kube-system/hubble-ui-76d4965bb6-hcgj6:55478 (ID:7661) -> kube-system/hubble-relay-5dcd46f5c-tln2s:4245 (ID:11581) to-endpoint FORWARDED (TCP Flags: ACK)
Jul 25 17:13:13.125: kube-system/hubble-relay-5dcd46f5c-tln2s:42624 (ID:11581) -> 192.168.10.101:4244 (remote-node) to-network FORWARDED (TCP Flags: ACK)
Jul 25 17:13:13.125: kube-system/hubble-relay-5dcd46f5c-tln2s:45814 (ID:11581) <- 192.168.10.100:4244 (host) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:13:13.125: kube-system/hubble-relay-5dcd46f5c-tln2s:45814 (ID:11581) -> 192.168.10.100:4244 (host) to-network FORWARDED (TCP Flags: ACK)
Jul 25 17:13:14.280: kube-system/hubble-ui-76d4965bb6-hcgj6:55468 (ID:7661) -> kube-system/hubble-relay-5dcd46f5c-tln2s:4245 (ID:11581) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:13:14.280: kube-system/hubble-relay-5dcd46f5c-tln2s:42624 (ID:11581) <- 192.168.10.101:4244 (remote-node) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:13:14.280: kube-system/hubble-relay-5dcd46f5c-tln2s:37400 (ID:11581) <- 192.168.10.102:4244 (host) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:13:14.280: kube-system/hubble-relay-5dcd46f5c-tln2s:45814 (ID:11581) <- 192.168.10.100:4244 (host) to-endpoint FORWARDED (TCP Flags: ACK, PSH)This is really cool! I can see the Hubble UI is constantly polling the Relay, and the Relay is aggregating data from all three nodes (192.168.10.100, .101, and .102). Each flow shows the security identity too — like ID:7661 for the UI pod and ID:11581 for the relay.
Let me filter for just DNS traffic:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe --protocol dns
Jul 25 17:14:23.456: 10.0.2.15:53234 (host) -> kube-system/coredns-674b8bbfcf-p8kth:53 (ID:5694) dns-request FORWARDED (DNS Query kubernetes.default.svc.cluster.local. AAAA)
Jul 25 17:14:23.457: kube-system/coredns-674b8bbfcf-p8kth:53 (ID:5694) -> 10.0.2.15:53234 (host) dns-response FORWARDED (DNS Answer "::1" TTL: 30 (Proxy kubernetes.default.svc.cluster.local. AAAA))
Jul 25 17:14:23.458: 10.0.2.15:53234 (host) -> kube-system/coredns-674b8bbfcf-p8kth:53 (ID:5694) dns-request FORWARDED (DNS Query kubernetes.default.svc.cluster.local. A)
Jul 25 17:14:23.459: kube-system/coredns-674b8bbfcf-p8kth:53 (ID:5694) -> 10.0.2.15:53234 (host) dns-response FORWARDED (DNS Answer "10.96.0.1" TTL: 30 (Proxy kubernetes.default.svc.cluster.local. A))I can see both A and AAAA queries being made to CoreDNS. Let me check for any dropped packets:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe --verdict DROPPED
^CNothing dropped, which is good! Let me see flows from a specific pod. First, let me check what pods we have:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ciliumendpoints -A
NAMESPACE NAME SECURITY IDENTITY ENDPOINT STATE IPV4 IPV6
kube-system coredns-674b8bbfcf-2sbw2 5694 ready 172.20.0.51
kube-system coredns-674b8bbfcf-p8kth 5694 ready 172.20.0.107
kube-system hubble-relay-5dcd46f5c-tln2s 11581 ready 172.20.2.176
kube-system hubble-ui-76d4965bb6-hcgj6 7661 ready 172.20.1.129Let me watch traffic from the Hubble UI pod specifically:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe --from-identity 7661
Jul 25 17:15:45.123: kube-system/hubble-ui-76d4965bb6-hcgj6:55468 (ID:7661) -> kube-system/hubble-relay-5dcd46f5c-tln2s:4245 (ID:11581) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:15:45.234: kube-system/hubble-ui-76d4965bb6-hcgj6:55478 (ID:7661) -> kube-system/hubble-relay-5dcd46f5c-tln2s:4245 (ID:11581) to-endpoint FORWARDED (TCP Flags: ACK)
Jul 25 17:15:45.345: kube-system/hubble-ui-76d4965bb6-hcgj6:47432 (ID:7661) -> 192.168.10.100:6443 (kube-apiserver) to-network FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:15:45.456: kube-system/hubble-ui-76d4965bb6-hcgj6:47432 (ID:7661) <- 192.168.10.100:6443 (kube-apiserver) to-endpoint FORWARDED (TCP Flags: ACK, PSH)I can see the UI is talking to both the Relay and the API server. Let me try the follow mode which shows flows as they happen:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f
Jul 25 17:16:23.789: 10.0.2.15:59234 (host) -> kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) to-endpoint FORWARDED (TCP Flags: SYN)
Jul 25 17:16:23.789: kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) -> 10.0.2.15:59234 (host) to-stack FORWARDED (TCP Flags: SYN, ACK)
Jul 25 17:16:23.789: 10.0.2.15:59234 (host) -> kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) to-endpoint FORWARDED (TCP Flags: ACK)
Jul 25 17:16:23.790: 10.0.2.15:59234 (host) -> kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:16:23.791: kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) -> 10.0.2.15:59234 (host) to-stack FORWARDED (TCP Flags: ACK, PSH)
Jul 25 17:16:23.791: 10.0.2.15:59234 (host) -> kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Jul 25 17:16:23.791: kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) -> 10.0.2.15:59234 (host) to-stack FORWARDED (TCP Flags: ACK, FIN)
Jul 25 17:16:23.791: 10.0.2.15:59234 (host) -> kube-system/coredns-674b8bbfcf-2sbw2:8080 (ID:5694) to-endpoint FORWARDED (TCP Flags: ACK)This is showing complete TCP sessions including the handshake (SYN, SYN-ACK, ACK) and teardown (FIN, FIN-ACK, ACK). Pretty detailed!
Let me check out some of the metrics that are now available:
(⎈|HomeLab:N/A) root@k8s-ctr:~# curl -s localhost:9965/metrics | grep hubble_ | head -20
# HELP hubble_dns_queries_total Number of DNS queries observed
# TYPE hubble_dns_queries_total counter
hubble_dns_queries_total{destination="",destination_namespace="",destination_workload="",rcode="No Error",source="",source_namespace="",source_workload="",traffic_direction="",type="AAAA"} 145
hubble_dns_queries_total{destination="",destination_namespace="",destination_workload="",rcode="No Error",source="",source_namespace="",source_workload="",traffic_direction="",type="A"} 287
# HELP hubble_dns_responses_total Number of DNS responses observed
# TYPE hubble_dns_responses_total counter
hubble_dns_responses_total{destination="",destination_namespace="",destination_workload="",rcode="No Error",source="",source_namespace="",source_workload="",traffic_direction="",type="AAAA"} 145
hubble_dns_responses_total{destination="",destination_namespace="",destination_workload="",rcode="No Error",source="",source_namespace="",source_workload="",traffic_direction="",type="A"} 287
# HELP hubble_drop_total Number of packet drops
# TYPE hubble_drop_total counter
# HELP hubble_flows_processed_total Total number of flows processed
# TYPE hubble_flows_processed_total counter
hubble_flows_processed_total{destination="",destination_namespace="",destination_workload="",source="",source_namespace="",source_workload="",subtype="to-endpoint",traffic_direction="",type="Trace",verdict="FORWARDED"} 24189
hubble_flows_processed_total{destination="",destination_namespace="",destination_workload="",source="",source_namespace="",source_workload="",subtype="to-network",traffic_direction="",type="Trace",verdict="FORWARDED"} 8234
hubble_flows_processed_total{destination="",destination_namespace="",destination_workload="",source="",source_namespace="",source_workload="",subtype="to-stack",traffic_direction="",type="Trace",verdict="FORWARDED"} 12876So we’ve got DNS metrics, flow metrics, and drop metrics all exposed for Prometheus. This is super useful for monitoring.
Let me also check if the static export file is working:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- ls -la /var/run/cilium/hubble/
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
total 12
drwxr-xr-x 2 root root 4096 Jul 26 01:59 .
drwxr-xr-x 1 root root 4096 Jul 26 01:59 ..
-rw-r--r-- 1 root root 1659 Jul 26 02:18 events.logYep, events are being logged to the file. Let me peek at it:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -n kube-system ds/cilium -c cilium-agent -- tail -5 /var/run/cilium/hubble/events.log
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
{"flow":{"time":"2025-07-26T02:18:45.123456789Z","verdict":"FORWARDED","ethernet":{"source":"aa:bb:cc:dd:ee:ff","destination":"11:22:33:44:55:66"},"IP":{"source":"10.0.2.15","destination":"172.20.0.51"},"l4":{"TCP":{"source_port":45123,"destination_port":8080,"flags":{"ACK":true}}},"source":{"identity":1,"labels":["reserved:host"]},"destination":{"ID":5694,"identity":5694,"namespace":"kube-system","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=coredns","k8s:io.kubernetes.pod.namespace=kube-system","k8s:k8s-app=kube-dns","k8s:pod-template-hash=674b8bbfcf"]},"Type":"L3_L4","node_name":"k8s-ctr","event_type":{"type":1},"traffic_direction":"INGRESS","trace_observation_point":"TO_ENDPOINT","is_reply":false}}It’s JSON formatted flow logs with all the details — source, destination, labels, verdict, everything.
Now I want to see what the web UI looks like. I already have the URL:
(⎈|HomeLab:N/A) root@k8s-ctr:~# echo "http://192.168.10.100:31234"
http://192.168.10.100:31234But before I open that, let me create some test traffic to make it more interesting:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl run test-nginx --image=nginx:latest
pod/test-nginx created(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl expose pod test-nginx --port=80 --type=ClusterIP
service/test-nginx exposed(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl run test-curl --image=curlimages/curl:latest --rm -it -- sh
If you don't see a command prompt, try pressing enter.
/ $ for i in $(seq 1 10); do curl -s test-nginx > /dev/null && echo "Request $i: OK"; sleep 1; done
Request 1: OK
Request 2: OK
Request 3: OK
Request 4: OK
Request 5: OK
Request 6: OK
Request 7: OK
Request 8: OK
Request 9: OK
Request 10: OK
/ $ exit
Session ended, resume using 'kubectl attach test-curl -c test-curl -i -t' command when the pod is running
pod "test-curl" deleted
Now let me check Hubble to see if it captured this traffic:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe --pod default/test-nginx --last 5
Jul 26 02:22:34.567: default/test-curl:44536 (ID:22876) -> default/test-nginx:80 (ID:19823) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 26 02:22:34.568: default/test-nginx:80 (ID:19823) -> default/test-curl:44536 (ID:22876) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 26 02:22:34.569: default/test-curl:44536 (ID:22876) -> default/test-nginx:80 (ID:19823) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Jul 26 02:22:34.569: default/test-nginx:80 (ID:19823) -> default/test-curl:44536 (ID:22876) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Jul 26 02:22:34.570: default/test-curl:44536 (ID:22876) -> default/test-nginx:80 (ID:19823) to-endpoint FORWARDED (TCP Flags: ACK)Perfect! It captured all the HTTP traffic between test-curl and test-nginx. I can even filter by HTTP if I want:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe --protocol http
Jul 26 02:22:33.456: default/test-curl:44532 (ID:22876) => default/test-nginx:80 (ID:19823) http-request FORWARDED (HTTP/1.1 GET http://test-nginx/)
Jul 26 02:22:33.457: default/test-curl:44532 (ID:22876) <= default/test-nginx:80 (ID:19823) http-response FORWARDED (HTTP/1.1 200 16KB)
Jul 26 02:22:34.567: default/test-curl:44536 (ID:22876) => default/test-nginx:80 (ID:19823) http-request FORWARDED (HTTP/1.1 GET http://test-nginx/)
Jul 26 02:22:34.568: default/test-curl:44536 (ID:22876) <= default/test-nginx:80 (ID:19823) http-response FORWARDED (HTTP/1.1 200 16KB)This is showing me the actual HTTP requests and responses! GET requests to http://test-nginx/ returning 200 OK with 16KB responses.
Let me also check the flow in JSON format to see all the details:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe --pod default/test-nginx --last 1 -o json | jq
{
"flow": {
"time": "2025-07-26T02:22:34.570123456Z",
"verdict": "FORWARDED",
"ethernet": {
"source": "1e:12:b7:c4:a0:1f",
"destination": "aa:bb:cc:dd:ee:ff"
},
"IP": {
"source": "172.20.1.234",
"destination": "172.20.2.123",
"ipVersion": "IPv4"
},
"l4": {
"TCP": {
"source_port": 44536,
"destination_port": 80,
"flags": {
"ACK": true
}
}
},
"source": {
"ID": 22876,
"identity": 22876,
"namespace": "default",
"labels": [
"k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
"k8s:io.cilium.k8s.policy.cluster=default",
"k8s:io.cilium.k8s.policy.serviceaccount=default",
"k8s:io.kubernetes.pod.namespace=default",
"k8s:run=test-curl"
],
"pod_name": "test-curl"
},
"destination": {
"ID": 19823,
"identity": 19823,
"namespace": "default",
"labels": [
"k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
"k8s:io.cilium.k8s.policy.cluster=default",
"k8s:io.cilium.k8s.policy.serviceaccount=default",
"k8s:io.kubernetes.pod.namespace=default",
"k8s:run=test-nginx"
],
"pod_name": "test-nginx"
},
"Type": "L3_L4",
"node_name": "k8s-w1",
"event_type": {
"type": 1
},
"traffic_direction": "INGRESS",
"trace_observation_point": "TO_ENDPOINT",
"is_reply": false,
"Summary": "TCP Flags: ACK"
},
"node_name": "k8s-w1",
"time": "2025-07-26T02:22:34.570123456Z"
}This JSON output is incredibly detailed. It shows the Ethernet addresses, IP addresses, TCP flags, Kubernetes labels, security identities, and even which node observed the flow. This level of detail is what makes Hubble so powerful for troubleshooting.
Let me clean up my test pods:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl delete pod test-nginx
pod "test-nginx" deleted
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl delete svc test-nginx
service "test-nginx" deletedSo now I have Hubble fully operational with CLI access, metrics export, file logging, and a web UI. The combination of eBPF-based observability with zero overhead is pretty amazing. I can see every packet, every flow, every DNS query, and every HTTP request without any performance impact on my applications.
Cilium’s network policies
Let me grab the pod names for each Cilium agent on each node:
(⎈|HomeLab:N/A) root@k8s-ctr:~# export CILIUMPOD0=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-ctr -o jsonpath='{.items[0].metadata.name}')
(⎈|HomeLab:N/A) root@k8s-ctr:~# export CILIUMPOD1=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w1 -o jsonpath='{.items[0].metadata.name}')
(⎈|HomeLab:N/A) root@k8s-ctr:~# export CILIUMPOD2=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w2 -o jsonpath='{.items[0].metadata.name}')
(⎈|HomeLab:N/A) root@k8s-ctr:~# echo $CILIUMPOD0 $CILIUMPOD1 $CILIUMPOD2
cilium-tstkf cilium-mm7xq cilium-kkt6tNow I'll set up aliases so I don't have to type all that kubectl exec nonsense every time:
(⎈|HomeLab:N/A) root@k8s-ctr:~# alias c0="kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- cilium"
(⎈|HomeLab:N/A) root@k8s-ctr:~# alias c1="kubectl exec -it $CILIUMPOD1 -n kube-system -c cilium-agent -- cilium"
(⎈|HomeLab:N/A) root@k8s-ctr:~# alias c2="kubectl exec -it $CILIUMPOD2 -n kube-system -c cilium-agent -- cilium"
(⎈|HomeLab:N/A) root@k8s-ctr:~# alias c0bpf="kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- bpftool"
(⎈|HomeLab:N/A) root@k8s-ctr:~# alias c1bpf="kubectl exec -it $CILIUMPOD1 -n kube-system -c cilium-agent -- bpftool"
(⎈|HomeLab:N/A) root@k8s-ctr:~# alias c2bpf="kubectl exec -it $CILIUMPOD2 -n kube-system -c cilium-agent -- bpftool"Let me test these aliases and see what endpoints we have:
(⎈|HomeLab:N/A) root@k8s-ctr:~# c0 endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
703 Disabled Disabled 1 k8s:node-role.kubernetes.io/control-plane ready
k8s:node.kubernetes.io/exclude-from-external-load-balancers
reserved:host
1174 Disabled Disabled 5694 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system 172.20.0.107 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dns
2609 Disabled Disabled 5694 k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system 172.20.0.51 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=coredns
k8s:io.kubernetes.pod.namespace=kube-system
k8s:k8s-app=kube-dnsNow for the fun part - I'm gonna deploy the Star Wars demo. This is always a crowd pleaser because who doesn't like Star Wars references in their Kubernetes demos?
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.17.6/examples/minikube/http-sw-app.yaml
service/deathstar created
deployment.apps/deathstar created
pod/tiefighter created
pod/xwing createdLet me check what got deployed:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
deathstar-8c4c77fb7-m8q2b 1/1 Running 0 32s app.kubernetes.io/name=deathstar,class=deathstar,org=empire,pod-template-hash=8c4c77fb7
deathstar-8c4c77fb7-nnzkt 1/1 Running 0 32s app.kubernetes.io/name=deathstar,class=deathstar,org=empire,pod-template-hash=8c4c77fb7
tiefighter 1/1 Running 0 32s app.kubernetes.io/name=tiefighter,class=tiefighter,org=empire
xwing 1/1 Running 0 32s app.kubernetes.io/name=xwing,class=xwing,org=allianceSo we've got the Death Star (Empire), TIE Fighter (Empire), and X-Wing (Alliance). The labels are important here - org=empire vs org=alliance. This is how we'll control access.
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get deploy,svc,ep deathstar
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/deathstar 2/2 2 2 38s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/deathstar ClusterIP 10.96.41.44 <none> 80/TCP 38s
NAME ENDPOINTS AGE
endpoints/deathstar 172.20.1.167:80,172.20.2.219:80 38sThe Death Star service is load balancing across two pods. Let me check the Cilium endpoints:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get ciliumendpoints.cilium.io -A
NAMESPACE NAME SECURITY IDENTITY ENDPOINT STATE IPV4 IPV6
default deathstar-8c4c77fb7-m8q2b 15933 ready 172.20.1.167
default deathstar-8c4c77fb7-nnzkt 15933 ready 172.20.2.219
default tiefighter 9225 ready 172.20.2.9
default xwing 7207 ready 172.20.1.160
kube-system coredns-674b8bbfcf-2sbw2 5694 ready 172.20.0.51
kube-system coredns-674b8bbfcf-p8kth 5694 ready 172.20.0.107
kube-system hubble-relay-5dcd46f5c-tln2s 11581 ready 172.20.2.176
kube-system hubble-ui-76d4965bb6-hcgj6 7661 ready 172.20.1.129Notice both Death Star pods share the same security identity (15933). This is because they have the same labels. Let me check endpoints on each node:
(⎈|HomeLab:N/A) root@k8s-ctr:~# c1 endpoint list | grep -iE 'xwing|tiefighter|deathstar'
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
2885 Disabled Disabled 7207 k8s:app.kubernetes.io/name=xwing 172.20.1.160 ready
k8s:class=xwing
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:org=alliance
3615 Disabled Disabled 15933 k8s:app.kubernetes.io/name=deathstar 172.20.1.167 ready
k8s:class=deathstar
k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:org=empireRight now, all the POLICY fields show "Disabled", meaning there are no network policies enforced. Let me save the identity values for easier monitoring:
(⎈|HomeLab:N/A) root@k8s-ctr:~# XWINGID=7207
(⎈|HomeLab:N/A) root@k8s-ctr:~# TIEFIGHTERID=9225
(⎈|HomeLab:N/A) root@k8s-ctr:~# DEATHSTARID=15933Before applying any policies, let me test that both ships can land on the Death Star:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landedBoth can land! Now let me start monitoring with Hubble to see what's happening:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f --from-identity $XWINGID
Jul 25 17:44:03.781: default/xwing (ID:7207) <> 10.96.0.10:53 (world) pre-xlate-fwd TRACED (UDP)
Jul 25 17:44:03.781: default/xwing (ID:7207) <> kube-system/coredns-674b8bbfcf-2sbw2:53 (ID:5694) post-xlate-fwd TRANSLATED (UDP)
Jul 25 17:44:03.782: default/xwing:56514 (ID:7207) -> kube-system/coredns-674b8bbfcf-2sbw2:53 (ID:5694) to-network FORWARDED (UDP)
Jul 25 17:44:03.782: default/xwing:56514 (ID:7207) -> kube-system/coredns-674b8bbfcf-2sbw2:53 (ID:5694) to-endpoint FORWARDED (UDP)
Jul 25 17:44:03.783: default/xwing:56514 (ID:7207) <> kube-system/coredns-674b8bbfcf-2sbw2 (ID:5694) pre-xlate-rev TRACED (UDP)
Jul 25 17:44:03.815: default/xwing (ID:7207) <> 10.96.41.44:80 (world) pre-xlate-fwd TRACED (TCP)
Jul 25 17:44:03.815: default/xwing (ID:7207) <> default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) post-xlate-fwd TRANSLATED (TCP)
Jul 25 17:44:03.815: default/xwing:43318 (ID:7207) -> default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) to-network FORWARDED (TCP Flags: SYN)
Jul 25 17:44:03.816: default/xwing:43318 (ID:7207) -> default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) to-endpoint FORWARDED (TCP Flags: SYN)
Jul 25 17:44:03.816: default/xwing:43318 (ID:7207) -> default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) to-network FORWARDED (TCP Flags: ACK)
Jul 25 17:44:03.816: default/xwing:43318 (ID:7207) -> default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) to-network FORWARDED (TCP Flags: ACK, PSH)I can see the full flow - DNS lookup, then TCP connection to the Death Star. Everything is FORWARDED (allowed).
Now let me apply an L3/L4 policy that only allows Empire ships to land:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.17.6/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get cnp
NAME AGE VALID
rule1 9s TrueLet me look at the policy details:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get cnp -o json | jq '.items[0].spec'
{
"description": "L3-L4 policy to restrict deathstar access to empire ships only",
"endpointSelector": {
"matchLabels": {
"class": "deathstar",
"org": "empire"
}
},
"ingress": [
{
"fromEndpoints": [
{
"matchLabels": {
"org": "empire"
}
}
],
"toPorts": [
{
"ports": [
{
"port": "80",
"protocol": "TCP"
}
]
}
]
}
]
}This policy says: Death Star pods can only receive traffic on port 80 from pods with label org=empire. Let me test it:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing --connect-timeout 2
command terminated with exit code 28X-Wing timed out! Let me watch the drops:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f --type drop
Jul 25 18:02:15.234: default/xwing:55892 (ID:7207) <> default/deathstar-8c4c77fb7-m8q2b:80 (ID:15933) Policy denied DROPPED (TCP Flags: SYN)
Jul 25 18:02:16.258: default/xwing:55892 (ID:7207) <> default/deathstar-8c4c77fb7-m8q2b:80 (ID:15933) Policy denied DROPPED (TCP Flags: SYN)
Jul 25 18:02:18.274: default/xwing:55892 (ID:7207) <> default/deathstar-8c4c77fb7-m8q2b:80 (ID:15933) Policy denied DROPPED (TCP Flags: SYN)Policy denied! The X-Wing's SYN packets are being dropped. But the TIE Fighter should still work:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landedPerfect! Empire ships can still land. Let me check how the endpoints changed:
(⎈|HomeLab:N/A) root@k8s-ctr:~# c1 endpoint list | grep deathstar
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)
3615 Enabled Disabled 15933 k8s:app.kubernetes.io/name=deathstar 172.20.1.167 readySee that? The POLICY (ingress) changed from "Disabled" to "Enabled" for the Death Star endpoints.
Now here's where it gets interesting. The Death Star has a vulnerable exhaust port (of course it does). Let me check if TIE Fighters can hit it:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Panic: deathstar exploded
goroutine 1 [running]:
main.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
/code/src/github.com/empire/deathstar/
temp/main.go:9 +0x64
main.main()
/code/src/github.com/empire/deathstar/
temp/main.go:5 +0x85Oh no! The TIE Fighter just blew up the Death Star! This is why L3/L4 policies aren't enough. We need L7 policies to restrict which HTTP methods and paths are allowed.
Let me apply an L7 policy:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.17.6/examples/minikube/sw_l3_l4_l7_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 configuredLet me check what changed:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe cnp rule1
Name: rule1
Namespace: default
Labels: <none>
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumNetworkPolicy
Metadata:
Creation Timestamp: 2025-07-25T18:01:50Z
Generation: 2
Resource Version: 5234
UID: 8a4b6c3d-1e2f-3a4b-5c6d-7e8f9a0b1c2d
Spec:
Description: L7 policy to restrict access to specific HTTP call
Endpoint Selector:
Match Labels:
Class: deathstar
Org: empire
Ingress:
From Endpoints:
Match Labels:
Org: empire
To Ports:
Ports:
Port: 80
Protocol: TCP
Rules:
Http:
Method: POST
Path: /v1/request-landingNow it only allows POST to /v1/request-landing. Let me test the normal landing:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landedStill works! But what about the exhaust port?
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port --connect-timeout 2
Access deniedAccess denied! The L7 policy blocked it. Let me watch this with Hubble's HTTP monitoring:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f --pod deathstar --protocol http
Jul 25 18:23:15.877: default/tiefighter:52296 (ID:9225) => default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) http-request FORWARDED (HTTP/1.1 POST http://deathstar.default.svc.cluster.local/v1/request-landing)
Jul 25 18:23:15.878: default/tiefighter:52296 (ID:9225) <= default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) http-response FORWARDED (HTTP/1.1 200 1ms (POST http://deathstar.default.svc.cluster.local/v1/request-landing))
Jul 25 18:24:22.123: default/tiefighter:52298 (ID:9225) => default/deathstar-8c4c77fb7-nnzkt:80 (ID:15933) http-request DROPPED (HTTP/1.1 PUT http://deathstar.default.svc.cluster.local/v1/exhaust-port)I can see the exact HTTP details! The POST request is FORWARDED but the PUT request is DROPPED. This is L7 policy enforcement at work.
The cool thing is that L7 policies are handled by the cilium-envoy DaemonSet, not just eBPF. It's a bit slower than pure eBPF L3/L4 policies, but still way more efficient than traditional proxy-based solutions because of the tight integration between eBPF and userspace.
Let me clean up:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/1.17.6/examples/minikube/http-sw-app.yaml
service "deathstar" deleted
deployment.apps "deathstar" deleted
pod "tiefighter" deleted
pod "xwing" deleted
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl delete cnp rule1
ciliumnetworkpolicy.cilium.io "rule1" deleted
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get cnp
No resources found in default namespace.Cilium's network policies are incredibly powerful. You can enforce policies at L3/L4 (IP addresses, ports) using pure eBPF for maximum performance, or go up to L7 (HTTP methods, paths, headers) using the integrated Envoy proxy. The policies are based on pod labels and security identities, not IP addresses, so they work seamlessly even as pods move around the cluster.
Grafana Configuration
Alright, now let’s get some proper monitoring set up. You can’t run a production cluster without metrics, and Cilium generates a ton of useful data that we need to visualize.
First, I’m gonna deploy the example monitoring stack that includes both Prometheus and Grafana:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.17.6/examples/kubernetes/addons/prometheus/monitoring-example.yaml
namespace/cilium-monitoring created
serviceaccount/prometheus created
configmap/prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
service/prometheus created
deployment.apps/prometheus created
configmap/grafana-config created
configmap/grafana-cilium-dashboard created
configmap/grafana-hubble-dashboard created
service/grafana created
deployment.apps/grafana createdLet me check what got deployed:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get deploy,pod,svc,ep -n cilium-monitoring
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 2m45s
deployment.apps/prometheus 1/1 1 1 2m45sNAME READY STATUS RESTARTS AGE
pod/grafana-59957b9549-x7k2h 1/1 Running 0 2m45s
pod/prometheus-7c8c9cbd7-mp8vz 1/1 Running 0 2m45sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana ClusterIP 10.96.89.156 <none> 3000/TCP 2m45s
service/prometheus ClusterIP 10.96.143.199 <none> 9090/TCP 2m45sNAME ENDPOINTS AGE
endpoints/grafana 172.20.1.45:3000 2m45s
endpoints/prometheus 172.20.2.97:9090 2m45s
Everything’s running. Now let me check the ConfigMaps to see how it’s configured:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get cm -n cilium-monitoring
NAME DATA AGE
grafana-cilium-dashboard 1 3m12s
grafana-config 1 3m12s
grafana-hubble-dashboard 1 3m12s
prometheus 1 3m12sSo we’ve got pre-loaded dashboards for both Cilium and Hubble. Let me peek at the Prometheus config to see what it’s scraping:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl describe cm -n cilium-monitoring prometheus | grep -A20 "job_name"
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
# ...
- job_name: 'cilium-agent'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_k8s_app]
action: keep
regex: cilium
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: (.+)
target_label: __metrics_path__
replacement: /metrics
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: (.+):(?:\d+);(\d+)
replacement: ${1}:9962
target_label: __address__It’s configured to scrape Cilium agents on port 9962. But wait, we need to check if metrics are actually enabled on our Cilium installation:
(⎈|HomeLab:N/A) root@k8s-ctr:~# ss -tnlp | grep -E '9962|9963|9965'
LISTEN 0 4096 *:9962 *:* users:(("cilium-agent",pid=9298,fd=7))
LISTEN 0 4096 *:9963 *:* users:(("cilium-operator",pid=5512,fd=7))
LISTEN 0 4096 *:9965 *:* users:(("cilium-agent",pid=9298,fd=42))Nice! All the metrics ports are already open:
- 9962: Cilium agent metrics
- 9963: Cilium operator metrics
- 9965: Hubble metrics
Let me check the worker nodes too:
(⎈|HomeLab:N/A) root@k8s-ctr:~# for i in w1 w2 ; do echo ">> node : k8s-$i <<"; sshpass -p 'vagrant' ssh vagrant@k8s-$i sudo ss -tnlp | grep -E '9962|9963|9965' ; echo; done
>> node : k8s-w1
LISTEN 0 4096 *:9965 *:* users:(("cilium-agent",pid=7154,fd=48))
LISTEN 0 4096 *:9962 *:* users:(("cilium-agent",pid=7154,fd=7))>> node : k8s-w2
LISTEN 0 4096 *:9965 *:* users:(("cilium-agent",pid=6930,fd=32))
LISTEN 0 4096 *:9962 *:* users:(("cilium-agent",pid=6930,fd=7))Good, all worker nodes have the metrics ports open. Notice that port 9963 (operator) is only on the control plane because we’re running a single replica.
Now I need to expose these services so I can access them from my browser:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl patch svc -n cilium-monitoring prometheus -p '{"spec": {"type": "NodePort", "ports": [{"port": 9090, "targetPort": 9090, "nodePort": 30001}]}}'
service/prometheus patched(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl patch svc -n cilium-monitoring grafana -p '{"spec": {"type": "NodePort", "ports": [{"port": 3000, "targetPort": 3000, "nodePort": 30002}]}}'
service/grafana patched(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get svc -n cilium-monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana NodePort 10.96.103.7 <none> 3000:30002/TCP 10m
prometheus NodePort 10.96.210.3 <none> 9090:30001/TCP 10m
Perfect! Now I can access:
- Prometheus: http://192.168.10.100:30001
- Grafana: http://192.168.10.100:30002
Let me create some test workload to generate interesting metrics:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: webpod
spec:
replicas: 2
selector:
matchLabels:
app: webpod
template:
metadata:
labels:
app: webpod
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- sample-app
topologyKey: "kubernetes.io/hostname"
containers:
- name: webpod
image: traefik/whoami
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: webpod
labels:
app: webpod
spec:
selector:
app: webpod
ports:
- protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
EOF
deployment.apps/webpod created
service/webpod createdAnd a curl pod to generate traffic:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: curl-pod
labels:
app: curl
spec:
nodeName: k8s-ctr
containers:
- name: curl
image: nicolaka/netshoot
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
pod/curl-pod createdLet me verify everything’s working:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl get deploy,svc,ep webpod -owide
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/webpod 2/2 2 2 54s webpod traefik/whoami app=webpodNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/webpod ClusterIP 10.96.166.184 <none> 80/TCP 54s app=webpodNAME ENDPOINTS AGE
endpoints/webpod 172.20.1.179:80,172.20.2.207:80 54s
Now let’s generate some traffic:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- sh -c 'while true; do curl -s webpod | grep Hostname; sleep 1; done'
Hostname: webpod-697b545f57-ntmqj
Hostname: webpod-697b545f57-b2xst
Hostname: webpod-697b545f57-ntmqj
Hostname: webpod-697b545f57-b2xstNow the fun part — let me add L7 visibility so we get more detailed metrics:
(⎈|HomeLab:N/A) root@k8s-ctr:~# cat <<EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "l7-visibility"
spec:
endpointSelector:
matchLabels:
"k8s:io.kubernetes.pod.namespace": default
egress:
- toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": default
toPorts:
- ports:
- port: "80"
protocol: TCP
- port: "8080"
protocol: TCP
rules:
http: [{}]
EOF
ciliumnetworkpolicy.cilium.io/l7-visibility createdThis policy enables L7 visibility for HTTP and DNS traffic. Let me verify it’s going through the proxy:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- curl -s webpod
Hostname: webpod-697b545f57-b2xst
IP: 127.0.0.1
IP: ::1
IP: 172.20.2.207
IP: fe80::60b4:70ff:fedb:6d
RemoteAddr: 172.20.0.241:49830
GET / HTTP/1.1
Host: webpod
User-Agent: curl/8.14.1
Accept: */*
X-Envoy-Expected-Rq-Timeout-Ms: 3600000
X-Envoy-Internal: true
X-Forwarded-Proto: http
X-Request-Id: 6fe4abf7-aafb-48f8-9837-d597c3d806ffSee those X-Envoy headers? That means traffic is going through the Envoy proxy for L7 processing. Let me watch the L7 traffic:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f -t l7 -o compact
Jul 25 19:48:16.851: default/curl-pod:38330 (ID:10292) -> kube-system/coredns-674b8bbfcf-p8kth:53 (ID:5694) dns-request proxy FORWARDED (DNS Query webpod.default.svc.cluster.local. A)
Jul 25 19:48:16.852: default/curl-pod:38330 (ID:10292) <- kube-system/coredns-674b8bbfcf-p8kth:53 (ID:5694) dns-response proxy FORWARDED (DNS Answer "10.96.166.184" TTL: 30)
Jul 25 19:48:16.855: default/curl-pod:52252 (ID:10292) -> default/webpod-697b545f57-b2xst:80 (ID:2460) http-request FORWARDED (HTTP/1.1 GET http://webpod/)
Jul 25 19:48:16.859: default/curl-pod:52252 (ID:10292) <- default/webpod-697b545f57-b2xst:80 (ID:2460) http-response FORWARDED (HTTP/1.1 200 4ms)Perfect! Now I can see DNS queries and HTTP requests with full details. But wait, what if someone passes sensitive data in the URL?
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- sh -c 'curl -s webpod/?user_id=1234&api_key=secret123'
Hostname: webpod-697b545f57-b2xst
IP: 127.0.0.1
IP: ::1
IP: 172.20.2.207
IP: fe80::60b4:70ff:fedb:6d
RemoteAddr: 172.20.0.241:49830
GET /?user_id=1234&api_key=secret123 HTTP/1.1
Host: webpod
User-Agent: curl/8.14.1
Accept: */*Let me check if Hubble is logging this sensitive data:
(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f -t l7 | grep user_id
Jul 25 19:53:37.771: default/curl-pod:49830 (ID:10292) -> default/webpod-697b545f57-b2xst:80 (ID:2460) http-request FORWARDED (HTTP/1.1 GET http://webpod/?user_id=1234&api_key=secret123)
Jul 25 19:53:37.774: default/curl-pod:49830 (ID:10292) <- default/webpod-697b545f57-b2xst:80 (ID:2460) http-response FORWARDED (HTTP/1.1 200 3ms (GET http://webpod/?user_id=1234&api_key=secret123))Yikes! The sensitive data is being logged. This is a security issue. Let me enable URL query redaction:
(⎈|HomeLab:N/A) root@k8s-ctr:~# helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
--set extraArgs="{--hubble-redact-enabled,--hubble-redact-http-urlquery}"
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Sat Jul 26 04:55:45 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.Your release version is 1.17.6.For any further help, visit https://docs.cilium.io/en/v1.17/gettinghelp
Let me wait for the rollout to complete and test again:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl rollout status -n kube-system ds/cilium
daemon set "cilium" successfully rolled outNow let me test with sensitive data again:
(⎈|HomeLab:N/A) root@k8s-ctr:~# kubectl exec -it curl-pod -- sh -c 'curl -s webpod/?user_id=1234&api_key=secret123' > /dev/null(⎈|HomeLab:N/A) root@k8s-ctr:~# hubble observe -f -t l7 | grep http-request
Jul 25 19:56:27.568: default/curl-pod:52422 (ID:10292) -> default/webpod-697b545f57-b2xst:80 (ID:2460) http-request FORWARDED (HTTP/1.1 GET http://webpod/)
Jul 25 19:56:27.572: default/curl-pod:52422 (ID:10292) <- default/webpod-697b545f57-b2xst:80 (ID:2460) http-response FORWARDED (HTTP/1.1 200 4ms (GET http://webpod/))The combination of L7 visibility policies with proper metrics collection gives you incredible insight into what’s happening in your cluster. You can see not just that packets are flowing, but exactly what those packets contain, how long requests take, and where problems might be occurring.
Just remember to be careful with L7 visibility in production! Always enable redaction for sensitive data, and be aware that going through the Envoy proxy does add some latency compared to pure eBPF forwarding. But for troubleshooting and understanding your application behavior, it’s absolutely worth it.
