Kubernetes Security with Cilium and Tetragon: A Comprehensive Lab Study

https://tetragon.io/docs/overview/

Setting Up the Lab Environment

We’ll build a three-node Kubernetes cluster using Vagrant to explore Cilium’s security capabilities. The environment consists of one control plane node (k8s-ctr) and two worker nodes (k8s-w1, k8s-w2), which provides enough complexity to properly demonstrate inter-node communication, encryption, and distributed security policies.

mkdir cilium-lab && cd cilium-lab
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/8w/Vagrantfile
vagrant up
vagrant ssh k8s-ctr

Once the cluster is operational, we install Cilium with a comprehensive feature set. Each configuration option serves a specific purpose in our security lab. The native routing mode eliminates unnecessary packet encapsulation overhead, while kube-proxy replacement means all networking happens through eBPF programs in the kernel. Hubble provides deep visibility into network flows, and Prometheus integration enables metrics collection for security monitoring.

helm repo add cilium https://helm.cilium.io
helm repo update
helm install cilium cilium/cilium --version 1.18.1 --namespace kube-system \
  --set k8sServiceHost=auto \
  --set ipam.mode="cluster-pool" \
  --set k8s.requireIPv4PodCIDR=true \
  --set ipv4NativeRoutingCIDR=10.244.0.0/16 \
  --set routingMode=native \
  --set autoDirectNodeRoutes=true \
  --set endpointRoutes.enabled=true \
  --set kubeProxyReplacement=true \
  --set bpf.masquerade=true \
  --set installNoConntrackIptablesRules=true \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.ui.service.type=NodePort \
  --set hubble.ui.service.nodePort=30003 \
  --set prometheus.enabled=true \
  --set operator.prometheus.enabled=true

The installation exposes several interfaces for monitoring and debugging:

• Prometheus at http://192.168.10.100:30001 for metrics
• Grafana at http://192.168.10.100:30002 for visualization
• Hubble UI at http://192.168.10.100:30003 for real-time network flow observation

Understanding Cilium’s Identity-Based Security Model

The cornerstone of Cilium’s security architecture is its identity system, which fundamentally changes how we think about network security in Kubernetes. Instead of relying on ephemeral IP addresses that change whenever pods restart or scale, Cilium assigns stable identities based on pod labels. This approach aligns perfectly with Kubernetes’ declarative model where labels define the desired state.

Let’s examine how identities work in practice. First, check the current Cilium endpoints in the kube-system namespace:

kubectl get ciliumendpoints -n kube-system

The output reveals something interesting:

NAME                              SECURITY IDENTITY   ENDPOINT STATE   IPV4           IPV6
coredns-674b8bbfcf-7v8gz          14735               ready            172.20.0.138   
coredns-674b8bbfcf-sbftb          14735               ready            172.20.0.128   
hubble-relay-fdd49b976-fps92      15715               ready            172.20.0.32    
hubble-ui-655f947f96-cmncv        12634               ready            172.20.0.8

Notice how both CoreDNS pods share security identity 14735 despite having different IP addresses. This happens because they have identical security-relevant labels. From a security perspective, these pods are equivalent — they’re both instances of the same service and should have identical network access rights.

Now let’s observe what happens when we modify labels on running pods. This demonstrates Cilium’s dynamic identity management:

kubectl label pods -n kube-system -l k8s-app=kube-dns study=8w

After adding the label, check the identity list directly from the Cilium agent:

kubectl exec -it -n kube-system ds/cilium -- cilium identity list

The pods transition through a “waiting-for-identity” state while Cilium recalculates the appropriate identity based on the new label combination. This process happens automatically whenever labels change, ensuring security policies immediately reflect the current state without manual intervention.

Cilium also maintains special reserved identities for common communication patterns:

kubectl exec -it -n kube-system ds/cilium -- cilium identity list | grep reserved

You’ll see entries like:

• reserved:host (identity 1) - represents the host namespace, used when pods communicate with node services
• reserved:world (identity 2) - represents everything outside the cluster
• reserved:unmanaged (identity 3) - for endpoints not managed by Cilium
• reserved:health (identity 4) - for health checking endpoints
• reserved:init (identity 5) - for pods in initialization phase

These reserved identities simplify policy creation for common scenarios like allowing pods to reach external services or communicate with the host.

DNS-Based Security Policies

Traditional IP-based network policies quickly become unmaintainable when dealing with external services. Cloud providers use massive IP ranges that change frequently. CDNs serve content from different IPs based on geographic location. Services like GitHub or AWS have thousands of IP addresses. DNS-based policies solve this problem elegantly by allowing you to specify destinations by domain name.

Let’s create a test pod to demonstrate DNS policies:

cat << EOF > dns-sw-app.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mediabot
  labels:
    org: empire
    class: mediabot
    app: mediabot
spec:
  containers:
  - name: mediabot
    image: quay.io/cilium/json-mock:v1.3.8@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603
EOF

kubectl apply -f dns-sw-app.yaml
kubectl wait pod/mediabot --for=condition=Ready

Initially, this pod can communicate with any external service. Verify this by testing connectivity to different GitHub domains:

kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
# Output: HTTP/2 200

kubectl exec mediabot -- curl -I -s --max-time 5 https://support.github.com | head -1
# Output: HTTP/2 200

Now we’ll apply a DNS policy that restricts external access to only api.github.com. This policy demonstrates several important concepts:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
EOF

The policy has two critical components. First, it allows DNS queries to CoreDNS (port 53) — without this, the pod couldn’t resolve any domain names. Second, it permits connections only to api.github.com. The DNS proxy component of Cilium intercepts DNS responses, extracts the IP addresses, and temporarily adds them to an allow list.

Test the policy enforcement:

kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
# Still works: HTTP/2 200

kubectl exec mediabot -- curl -I -s --max-time 5 https://support.github.com | head -1
# Now times out - connection blocked

To understand how this works internally, let’s examine Cilium’s DNS proxy behavior. First, enable Hubble port forwarding and observe the traffic:

cilium hubble port-forward&
hubble observe --pod mediabot

When the pod makes a DNS query, you’ll see Cilium intercept it, resolve the domain, and cache the results. The FQDN cache maintains these IP-to-domain mappings:

export CILIUMPOD1=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w1 -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $CILIUMPOD1 -n kube-system -c cilium-agent -- cilium fqdn cache list

The cache output shows entries like:

Endpoint   Source   FQDN              TTL    ExpirationTime             IPs
1234       lookup   api.github.com.   30     2024-03-15 14:23:45 UTC   140.82.121.5

The default TTL is 30 seconds, after which Cilium removes the IPs from the allowed list. The pod must make a new DNS query to refresh the cache. This ensures that IP address changes are automatically handled without policy updates.

DNS policies also support wildcards for more flexible rules:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn-wildcard"
spec:
  endpointSelector:
    matchLabels:
      org: empire
  egress:
  - toFQDNs:
    - matchPattern: "*.github.com"
EOF

This allows access to any GitHub subdomain, useful when services use multiple domains for different functions like api.github.com, gist.github.com, and raw.githubusercontent.com.

Transparent Encryption with WireGuard

In production environments, especially in public clouds or shared infrastructure, encrypting pod-to-pod traffic across nodes is essential. Cilium integrates WireGuard to provide transparent encryption with minimal performance overhead compared to traditional IPsec solutions. WireGuard’s modern cryptographic design and kernel implementation make it ideal for high-throughput container networks.

First, verify your kernel supports WireGuard:

grep -E 'CONFIG_WIREGUARD=m' /boot/config-$(uname -r)

Enable WireGuard encryption in Cilium:

helm upgrade cilium cilium/cilium --version 1.18.1 --namespace kube-system --reuse-values \
  --set encryption.enabled=true \
  --set encryption.type=wireguard

kubectl -n kube-system rollout restart ds/cilium

After the rollout completes, verify the encryption status:

kubectl exec -it -n kube-system ds/cilium -- cilium encrypt status

The output shows:

Encryption: Wireguard                 
Interface: cilium_wg0
Public key: o4ISkG+hN88144j0FCwTEo0NtfvLnID3/q48eJ1pfVQ=
Number of peers: 2

Each node generates its own WireGuard keypair and automatically exchanges public keys with other nodes. Check the WireGuard interface directly:

wg show

This displays detailed peer information:

interface: cilium_wg0
  public key: o4ISkG+hN88144j0FCwTEo0NtfvLnID3/q48eJ1pfVQ=
  private key: (hidden)
  listening port: 51871

peer: kRN8cL3p2Jcn6k9rFQNmH8eZ2xQ8vWxGr5YJ1xV3wRc=
  endpoint: 192.168.10.101:51871
  allowed ips: 10.244.1.0/24
  latest handshake: 12 seconds ago
  transfer: 1.23 MiB received, 2.45 MiB sent

All inter-node pod traffic now flows through WireGuard tunnels on UDP port 51871. Importantly, pod-to-pod traffic on the same node remains unencrypted since it never leaves the host — there’s no security benefit to encrypting localhost traffic, and avoiding it improves performance.

Monitor the encrypted traffic with tcpdump:

tcpdump -eni any udp port 51871

You’ll see UDP packets between nodes containing the encrypted pod traffic. The actual pod communication is completely transparent — applications don’t need any changes to benefit from encryption.

TLS Interception and Deep Packet Inspection

Sometimes you need visibility into HTTPS traffic for security scanning, compliance auditing, or debugging. Cilium can perform transparent TLS interception by acting as a man-in-the-middle with your own certificate authority. This capability is crucial for enforcing application-layer policies on encrypted traffic or detecting data exfiltration attempts.

First, configure Cilium for TLS interception mode:

cat << EOF > tls-config.yaml
tls:
  readSecretsOnlyFromSecretsNamespace: true
  secretsNamespace:
    name: cilium-secrets
  secretSync:
    enabled: true
EOF

helm upgrade cilium cilium/cilium --version 1.18.1 --namespace kube-system --reuse-values -f tls-config.yaml
kubectl -n kube-system rollout restart deploy/cilium-operator
kubectl -n kube-system rollout restart ds/cilium

Create your own Certificate Authority:

openssl genrsa -des3 -out myCA.key 2048
# Enter password: qwe123

openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.crt
# Country: KR
# State: Seoul  
# City: Seoul
# Organization: cloudneta
# Unit: study
# Common Name: cloudneta.net

Generate a certificate for httpbin.org (the site we’ll intercept):

openssl genrsa -out internal-httpbin.key 2048

openssl req -new -key internal-httpbin.key -out internal-httpbin.csr
# Common Name must be: httpbin.org
openssl x509 -req -days 360 -in internal-httpbin.csr -CA myCA.crt -CAkey myCA.key -CAcreateserial -out internal-httpbin.crt -sha256

Create Kubernetes secrets with these certificates:

kubectl create secret tls httpbin-tls-data -n kube-system --cert=internal-httpbin.crt --key=internal-httpbin.key

Configure the pod to trust your CA:

kubectl cp myCA.crt default/mediabot:/usr/local/share/ca-certificates/myCA.crt
kubectl exec mediabot -- update-ca-certificates

Create a CA bundle for Cilium to use:

kubectl cp default/mediabot:/etc/ssl/certs/ca-certificates.crt ca-certificates.crt
kubectl create secret generic tls-orig-data -n kube-system --from-file=ca.crt=./ca-certificates.crt

Apply the TLS visibility policy:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes-tls-inspection/l7-visibility-tls.yaml

Test HTTPS interception:

kubectl exec -it mediabot -- curl -sL 'https://httpbin.org/headers' -v

In the verbose output, you’ll see your internal certificate being presented instead of httpbin’s real certificate. Cilium transparently intercepts the TLS handshake, presents your certificate to the client, establishes a separate TLS connection to the real server, and proxies the decrypted traffic. This allows full visibility into HTTPS communications while maintaining encryption on both sides of the proxy.

gRPC Traffic Control and Method-Level Security

gRPC has become the standard for microservice communication, running over HTTP/2 with protocol buffers for efficient serialization. Cilium understands the gRPC protocol and can enforce policies at the individual RPC method level, providing fine-grained access control that’s impossible with traditional network policies.

Deploy the Cloud City Door Manager demonstration application:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes-grpc/cc-door-app.yaml

Verify the deployment:

kubectl get pods,svc

You should see:

NAME                      READY   STATUS    RESTARTS   AGE
pod/cc-door-mgr           1/1     Running   0          30s
pod/terminal-87           1/1     Running   0          30s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
service/cc-door-mgr  ClusterIP   10.96.123.45   <none>        50051/TCP

Test the available gRPC methods without any restrictions:

kubectl exec terminal-87 -- python3 /cloudcity/cc_door_client.py GetName 1
# Output: Door name: Door-1

kubectl exec terminal-87 -- python3 /cloudcity/cc_door_client.py GetLocation 1
# Output: Door location: Level-1
kubectl exec terminal-87 -- python3 /cloudcity/cc_door_client.py SetAccessCode 1 999
# Output: Access code set successfully

All methods work without restrictions. Now apply a policy that selectively allows certain gRPC methods:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "rule1"
spec:
  description: L7 policy to restrict gRPC methods
  endpointSelector:
    matchLabels:
      app: cc-door-mgr
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: public-terminal
    toPorts:
    - ports:
      - port: "50051"
        protocol: TCP
      rules:
        http:
        - method: "POST"
          path: "/cloudcity.DoorManager/GetName"
        - method: "POST"
          path: "/cloudcity.DoorManager/GetLocation"
        - method: "POST"
          path: "/cloudcity.DoorManager/RequestMaintenance"
EOF

The policy explicitly allows three methods: GetName, GetLocation, and RequestMaintenance. Notice how gRPC methods are represented as HTTP POST requests to paths following the pattern /package.Service/Method.

Test the policy enforcement:

kubectl exec terminal-87 -- python3 /cloudcity/cc_door_client.py GetLocation 1
# Still works: Door location: Level-1

kubectl exec terminal-87 -- python3 /cloudcity/cc_door_client.py SetAccessCode 1 999
# Blocked with error: grpc._channel._InactiveRpcError: StatusCode.PERMISSION_DENIED

The key difference from traditional packet filtering is that Cilium returns a proper gRPC error code (StatusCode.PERMISSION_DENIED) rather than silently dropping packets. This makes debugging much easier for developers who see clear error messages in their application logs.

Monitor the gRPC traffic with Hubble:

hubble observe --pod terminal-87 --protocol grpc

You’ll see detailed gRPC method calls with their status codes, making it easy to audit API usage and troubleshoot access issues.

Host Firewall Configuration

Protecting pods is only part of the security equation — the Kubernetes nodes themselves need protection. Cilium’s host firewall extends network policy concepts to the host namespace, controlling access to node services like SSH, kubelet API, and host network services.

Enable the host firewall feature:

helm upgrade cilium cilium/cilium --version 1.18.1 --namespace kube-system --reuse-values \
  --set hostFirewall.enabled=true

Verify activation:

kubectl exec -n kube-system ds/cilium -- cilium-dbg status | grep 'Host firewall'
# Output: Host firewall:            Enabled

Label nodes to apply specific firewall rules:

kubectl label node k8s-w1 node-access=ssh

Before applying restrictive policies, use audit mode to understand current traffic patterns. First, find the host endpoint ID:

export CILIUM_POD=$(kubectl get pods -n kube-system --field-selector spec.nodeName=k8s-w1 -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it -n kube-system $CILIUM_POD -- cilium endpoint list | grep reserved:host

Output shows:

ENDPOINT   POLICY    IDENTITY   LABELS  IPv4           STATUS
2488       Disabled  1          reserved:host  192.168.10.101  ready

Enable audit mode for the host endpoint (using ID 2488 from above):

kubectl exec -it -n kube-system $CILIUM_POD -- cilium-dbg endpoint config 2488 PolicyAuditMode=Enabled

Apply a host firewall policy allowing only SSH and ICMP:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "demo-host-policy"
spec:
  description: "Allow only SSH and ICMP"
  nodeSelector:
    matchLabels:
      node-access: ssh
  ingress:
  - fromEntities:
    - cluster
  - toPorts:
    - ports:
      - port: "22"
        protocol: TCP
  - icmps:
    - fields:
      - type: EchoRequest
        family: IPv4
EOF

Monitor what traffic would be affected in audit mode:

kubectl exec -it -n kube-system $CILIUM_POD -- cilium-dbg monitor -t policy-verdict --related-to 2488

Create a test client container to verify connectivity:

docker run -d --rm --name client --cap-add=NET_ADMIN --network kind nicolaka/netshoot tail -f /dev/null
docker exec -it client ssh root@k8s-w1
# Works in audit mode

The monitor shows both allowed and would-be-denied traffic. Once satisfied with the policy, enforce it:

kubectl exec -it -n kube-system $CILIUM_POD -- cilium-dbg endpoint config 2488 PolicyAuditMode=Disabled

Now only SSH (port 22) and ICMP echo requests are allowed. Attempts to connect to other ports are immediately dropped.

Tetragon Runtime Security and Monitoring

While Cilium secures network communications, Tetragon provides runtime security by monitoring and enforcing policies on process execution, file access, and system calls. Built on eBPF like Cilium, Tetragon can detect and prevent attacks that network policies alone cannot stop.

Install Tetragon:

helm install tetragon cilium/tetragon -n kube-system --version 1.1.0
kubectl rollout status -n kube-system ds/tetragon -w

Deploy a sample application for monitoring:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.18.1/examples/minikube/http-sw-app.yaml

Apply a network monitoring policy to track all connections:

cat << EOF | kubectl apply -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "networking"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    return: false
  - call: "tcp_close"
    syscall: false
    return: false
  - call: "tcp_sendmsg"
    syscall: false
    return: false
EOF

Start monitoring process execution in real-time:

POD=$(kubectl -n kube-system get pods -l 'app.kubernetes.io/name=tetragon' -o name --field-selector spec.nodeName=$(kubectl get pod xwing -o jsonpath='{.spec.nodeName}'))
kubectl exec -ti -n kube-system $POD -c tetragon -- tetra getevents -o compact --pods xwing

In another terminal, execute commands in the monitored pod:

kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'

Tetragon captures the complete execution chain:

🚀 process default/xwing /usr/bin/bash -c "curl https://ebpf.io/applications/#tetragon"
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
🔌 connect default/xwing /usr/bin/curl tcp 10.244.1.5:38462 -> 104.198.14.52:443
📤 sendmsg default/xwing /usr/bin/curl tcp 10.244.1.5:38462 -> 104.198.14.52:443 2048 bytes
🔌 close   default/xwing /usr/bin/curl tcp 10.244.1.5:38462 -> 104.198.14.52:443
💥 exit    default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 0

Apply a file monitoring policy to track sensitive file access:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml

This policy monitors access to sensitive files like /etc/shadow and /etc/passwd. Test it:

kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'

Tetragon logs show:

🚀 process default/xwing /usr/bin/bash -c "cat /etc/shadow"
🚀 process default/xwing /usr/bin/cat /etc/shadow
📚 read    default/xwing /usr/bin/cat /etc/shadow
💥 exit    default/xwing /usr/bin/cat /etc/shadow 0

Now escalate to enforcement mode, which actively blocks the action:

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring_enforce.yaml

Try accessing the sensitive file again:

kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'
# Output: command terminated with exit code 137

The process is immediately killed with SIGKILL (exit code 137 = 128 + 9). The enforcement happens in the kernel before the file read occurs, preventing any data leakage.

Container Escape Detection and Prevention

Container escapes are among the most serious security threats in Kubernetes. Tetragon can detect and prevent various escape techniques. Let’s simulate a real attack scenario.

Deploy a privileged pod that an attacker might compromise:

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: sith-infiltrator
  labels:
    org: empire
spec:
  containers:
  - name: sith-infiltrator
    image: busybox
    command: ["sleep", "3600"]
    securityContext:
      privileged: true
EOF

Apply a policy to detect namespace changes, which are common in container escapes:

cat << EOF | kubectl apply -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "sys-setns"
spec:
  kprobes:
  - call: "sys_setns"
    syscall: true
    args:
    - index: 0
      type: "int"
    - index: 1
      type: "int"
EOF

Monitor Tetragon events while attempting an escape:

POD=$(kubectl -n kube-system get pods -l 'app.kubernetes.io/name=tetragon' -o name --field-selector spec.nodeName=$(kubectl get pod sith-infiltrator -o jsonpath='{.spec.nodeName}'))
kubectl exec -ti -n kube-system $POD -c tetragon -- tetra getevents -o compact --pods sith-infiltrator

In another terminal, attempt to escape to the host:

kubectl exec -it sith-infiltrator -- /bin/sh
nsenter -t 1 -a bash

Tetragon detects seven process_kprobe events, each representing entry into a different namespace:

• cgroup namespace
• ipc namespace
• uts namespace
• net namespace
• pid namespace
• mnt namespace
• time namespace

This pattern is a clear indicator of container escape. Now prevent the attack by blocking writes to critical directories:

cat << EOF | kubectl apply -f -
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-write-etc-kubernetes-manifests"
spec:
  podSelector:
    matchLabels:
     org: empire
  kprobes:
  - call: "security_file_permission"
    syscall: false
    args:
    - index: 0
      type: "file"
    - index: 1
      type: "int"
    selectors:
    - matchArgs:      
      - index: 0
        operator: "Prefix"
        values:
        - "/etc/kubernetes/manifests"
      matchActions:
        - action: Override
          argError: -1
EOF

This policy prevents writing to /etc/kubernetes/manifests, where static pod definitions could create backdoors. Test the protection:

kubectl exec -it sith-infiltrator -- /bin/sh
cd /etc/kubernetes/manifests/
cat << EOF > hack-latest.yaml
apiVersion: v1
kind: Pod
metadata:
  name: hack-latest
  namespace: doesnt-exist
spec:
  containers:
  - name: hack-latest
    image: sublimino/hack:latest
    securityContext:
      privileged: true
EOF
# Output: Permission denied

Even with root privileges in the container, the write is blocked at the kernel level. The eBPF program intercepts the system call and returns an error before any file operation occurs.

Tetragon can also detect fileless malware execution:

kubectl exec -n kube-system -ti daemonset/tetragon -c tetragon -- tetra getevents -o compact --processes curl,python

In another terminal, simulate fileless execution:

curl https://raw.githubusercontent.com/realpython/python-scripts/master/scripts/18_zipper.py | python

Tetragon captures both the network fetch and the interpreter execution, making it impossible for attackers to hide their activities.

Policy Audit Mode for Safe Production Deployment

One of Cilium’s most valuable features for production environments is policy audit mode. This allows you to test network policies without actually enforcing them, eliminating the risk of accidentally breaking critical services.

Let’s walk through a complete example using a multi-tier application. The application consists of frontend, backend-http, backend-grpc, and MySQL components. Deploy it first, then we’ll use audit mode to safely implement network segmentation.

First, identify the MySQL pod and prepare it for audit mode:

CILIUM_NAMESPACE="kube-system"
PODNAME=$(kubectl get pods -l app=mysql -o jsonpath='{.items[*].metadata.name}')
NODENAME=$(kubectl get pod -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].spec.nodeName}")
ENDPOINT=$(kubectl get cep -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].status.id}")
CILIUM_POD=$(kubectl -n "$CILIUM_NAMESPACE" get pod --all-namespaces --field-selector spec.nodeName="$NODENAME" -lk8s-app=cilium -o jsonpath='{.items[*].metadata.name}')

Enable audit mode for the MySQL endpoint:

kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
    cilium-dbg endpoint config "$ENDPOINT" PolicyAuditMode=Enabled

Verify audit mode is active:

kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
    cilium-dbg endpoint get "$ENDPOINT" -o jsonpath='{[*].spec.options.PolicyAuditMode}'
# Output: Enabled

Apply an initial deny-all policy to MySQL:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: mysql-deny-all
spec:
  endpointSelector:
    matchLabels:
      app: mysql
  ingress:
  - {}
EOF

The - {} at the end creates a deny-all rule. However, because audit mode is enabled, traffic continues to flow normally. The application remains fully functional while we analyze traffic patterns.

Test the application by clicking HTTP and gRPC buttons in the web interface. Then examine the audit logs:

kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
  hubble observe flows -t policy-verdict --last 10

The output shows traffic marked as “AUDITED”:

Sep  5 13:39:39.508: default/backend-http-5859bf58f-gr8nq:35152 -> default/mysql-647d757f9d-xzqxv:3306 policy-verdict:none INGRESS AUDITED
Sep  5 13:39:39.794: default/backend-grpc-6955f774c9-pcj6k:38440 -> default/mysql-647d757f9d-xzqxv:3306 policy-verdict:none INGRESS AUDITED

For more detailed analysis, output JSON format to see all labels:

kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
  hubble observe flows -t policy-verdict -o json | jq ".flow.source.labels" -c | sort | uniq

This reveals exactly which services are accessing MySQL:

["k8s:app=backend-grpc","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"]
["k8s:app=backend-http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"]

Test frontend access to MySQL:

kubectl exec deployment/frontend -- nc -vz mysql 3306
# Output: mysql (10.96.14.118:3306) open

In audit mode, even frontend can connect. Check the logs again:

kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
  hubble observe flows -t policy-verdict --last 5

You’ll see frontend access is also audited. Based on this information, create a proper policy allowing only backend services:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: mysql-allow-backend
spec:
  endpointSelector:
    matchLabels:
      app: mysql
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: backend-http
    - matchLabels:
        app: backend-grpc
    toPorts:
    - ports:
      - port: "3306"
        protocol: TCP
EOF

Now disable audit mode to enforce the policy:

kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
    cilium-dbg endpoint config "$ENDPOINT" PolicyAuditMode=Disabled

Test the enforcement. Backend services still work:

kubectl exec deployment/backend-http -- nc -vz mysql 3306
# Output: Connection successful

But frontend is now blocked:

kubectl exec deployment/frontend -- nc -vz mysql 3306
# No response - connection blocked

Check Hubble for the verdict:

kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
  hubble observe flows -t policy-verdict --last 10

Backend connections show “ALLOWED” while frontend shows “DENIED”:

Sep  5 14:17:06.559: default/backend-grpc-6955f774c9-pcj6k:38898 -> default/mysql-647d757f9d-xzqxv:3306 policy-verdict:L3-L4 INGRESS ALLOWED
Sep  5 14:17:52.005: default/frontend-74647b95d9-lnvss:35443 -> default/mysql-647d757f9d-xzqxv:3306 policy-verdict:none INGRESS DENIED

This workflow demonstrates how audit mode enables safe, data-driven policy implementation. You observe actual traffic patterns, refine policies based on real data, and only enforce them when confident they won’t disrupt services.

Comprehensive Security Architecture

Combining Cilium and Tetragon creates a defense-in-depth security architecture for Kubernetes. Each layer addresses specific threats and works together to provide comprehensive protection.

At the network layer, Cilium provides identity-based segmentation that’s resilient to IP address changes. DNS policies handle external service dependencies without maintaining IP lists. WireGuard encryption protects data in transit between nodes. TLS interception enables deep packet inspection when required for compliance or security scanning. Application-aware policies for protocols like gRPC provide fine-grained access control at the method level. Host firewalls extend protection to the nodes themselves.

At the runtime layer, Tetragon monitors and enforces policies on process behavior. It tracks process execution chains, file access patterns, network connections, and system calls. More importantly, it can actively prevent malicious actions like container escapes, unauthorized file access, or suspicious process behavior. The enforcement happens in the kernel through eBPF, making it extremely difficult to bypass.

The audit mode capability is critical for production deployment. It allows you to test policies without risk, observe actual traffic patterns, and iteratively refine rules based on real data. This eliminates the guesswork and risk traditionally associated with network policy implementation.

Performance remains excellent because both tools leverage eBPF for kernel-level processing. Unlike traditional security solutions that operate in userspace with significant overhead, eBPF programs run at native kernel speeds. This makes it practical to enable comprehensive security monitoring and enforcement even in high-throughput production environments.

Integration with Kubernetes is seamless. Both tools understand Kubernetes concepts natively — pods, services, namespaces, and labels. Policies use familiar Kubernetes constructs and can be managed through standard kubectl commands or GitOps workflows. Observability through Hubble and Tetragon provides deep visibility into both network flows and runtime behavior, essential for troubleshooting and security auditing.

The combination addresses the full attack lifecycle. Network policies prevent unauthorized connections and lateral movement. DNS policies control external communications. Encryption protects data in transit. Runtime monitoring detects suspicious behavior. Enforcement policies block attacks in real-time. Together, they implement a true zero-trust architecture where nothing is implicitly trusted and everything is continuously verified.

For production deployment, start with monitoring and gradually add enforcement. Begin with Hubble to understand network flows. Add network policies in audit mode. Enable Tetragon monitoring to baseline normal behavior. Implement runtime policies for obvious threats. Gradually tighten controls as you gain confidence and understanding of your application’s requirements.

This layered approach provides resilience against various attack vectors. Even if an attacker compromises a pod, they face network segmentation that limits lateral movement. If they attempt privilege escalation or container escape, Tetragon detects and blocks it. If they try to exfiltrate data, DNS policies and TLS inspection provide visibility and control. Each layer adds security without depending entirely on any single control.

The eBPF foundation ensures these capabilities scale with your cluster. Whether you’re running ten pods or ten thousand, the per-packet and per-syscall overhead remains minimal. This efficiency, combined with the granular control and deep visibility provided by Cilium and Tetragon, makes them ideal for securing modern cloud-native applications.