Building a Kubernetes Cluster From Scratch — Part 1 and Part 2

31 min readJan 10, 2026

See the repository if you want to follow my tutorial! https://github.com/sigridjineth/k8s-hard-way

The first part

This is the first part of a comprehensive tutorial series where we will build a fully functional Kubernetes cluster without using any automation tools like kubeadm or kubespray. By the end of this series, you will have manually installed and configured every single component that makes up a Kubernetes cluster, giving you deep insight into how all the pieces fit together.

I spent about a week going through this process myself, and I can tell you that it was both frustrating and enlightening. There were moments when I questioned why anyone would do this manually when tools exist to automate everything. But once the cluster came to life and I could actually trace the path of a request from kubectl all the way to a running container, it all made sense. The understanding you gain from this exercise is something you simply cannot get from running a single command that sets everything up for you.

If you have been working with Kubernetes for a while, you have probably used managed services like EKS, GKE, or AKS. Maybe you have spun up local clusters with minikube or kind. These tools are fantastic for getting things done quickly, but they hide an enormous amount of complexity from you.

Here is the thing: when something breaks in production, and it will break eventually, you need to understand what is actually happening under the hood. Why is that pod stuck in Pending? Why can pods on one node not talk to pods on another node? Why is the API server rejecting your requests with a cryptic certificate error?

When you build a cluster by hand, you touch every certificate, every configuration file, every systemd unit. You see exactly how the API server authenticates requests, how the scheduler decides where to place pods, how kubelet communicates with the container runtime. This knowledge becomes invaluable when you need to troubleshoot real problems.

There is another practical benefit too. If you ever work in an air-gapped environment where you cannot pull images from the internet or use managed services, knowing how to set up Kubernetes manually becomes a necessity rather than a nice-to-have skill.

By the end of this tutorial series, we will have a working Kubernetes cluster with the following components:

One control plane node running etcd, kube-apiserver, kube-controller-manager, and kube-scheduler. In production you would want multiple control plane nodes for high availability, but for learning purposes a single node is sufficient.

Two worker nodes running containerd as the container runtime, kubelet as the node agent, and kube-proxy for network proxying.

A jumpbox that serves as our administration host. This is where we will generate certificates, create configuration files, and run kubectl commands. Think of it as a bastion host that provides secure access to the cluster.

The network setup uses a private network in the 192.168.10.0/24 range for communication between nodes. Each worker node gets its own pod CIDR: node-0 gets 10.200.0.0/24 and node-1 gets 10.200.1.0/24. Services will use the 10.32.0.0/24 range.

Before we begin, let me be clear about what you need. This tutorial is written for a MacBook Pro, and I have tested it on both Intel and Apple Silicon machines. The instructions should work on either, though I will point out any differences when they matter.

You will need at least 16GB of RAM on your machine. We are going to run four virtual machines simultaneously, and while each one is not huge, they add up. If you have only 8GB, you might be able to squeeze by with reduced memory allocations, but I would not recommend it.

You should have at least 20GB of free disk space. The base VM images plus all the Kubernetes binaries will eat through storage faster than you might expect.

I am assuming you are comfortable with the command line and have a basic understanding of networking concepts like IP addresses, subnets, and routing. You should know what TCP/IP is and have at least a vague idea of how TLS certificates work. If terms like “private key” and “certificate authority” sound completely foreign, you might want to do some background reading first.

You should also have some familiarity with Kubernetes concepts. I am not going to explain what a pod is or why you would use a deployment. If you have never used kubectl before, I would suggest playing with minikube first to get the basics down.

Let us start by installing the tools we need on your Mac. We will use Homebrew for package management because it makes everything so much easier.

If you do not have Homebrew installed yet, open Terminal and run this:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Follow the prompts to complete the installation. If you are on Apple Silicon, Homebrew installs to /opt/homebrew, so you may need to add it to your PATH. The installer will tell you exactly what to do.

Now let us install VirtualBox. This is our virtualization platform that will run the virtual machines:

brew install --cask virtualbox

On macOS, you will likely see a prompt about allowing system extensions. Go to System Settings, then Privacy and Security, and approve the Oracle extension. You may need to reboot your Mac after this.

Once VirtualBox is installed, verify it works:

VBoxManage --version

You should see something like 7.0.14r161095 or similar. The exact version does not matter much as long as it is relatively recent.

Next, install Vagrant. This tool lets us define our virtual machine configuration in code and spin up multiple VMs with a single command:

brew install --cask vagrant

Verify the installation:

vagrant --version

You should see Vagrant 2.4.1 or something close to that.

A quick note for Apple Silicon users: VirtualBox added support for ARM-based Macs starting with version 7.0. However, the ARM support is still maturing, and you might encounter occasional quirks. If you run into serious issues, an alternative is to use UTM or VMware Fusion, but those would require modifying the instructions throughout this tutorial. For most people, VirtualBox works fine.

Let us create a dedicated directory for this project. I like to keep all my lab work organized:

mkdir -p ~/kubernetes-the-hard-way
cd ~/kubernetes-the-hard-way

This directory will contain our Vagrantfile, initialization scripts, and eventually all the certificates and configuration files we generate.

Before we create the Vagrantfile, let me explain what we are going to build and why.

We need four machines total. The jumpbox is a lightweight machine that we will use for administration. It does not need much horsepower since it just runs command-line tools. We will give it 2 CPU cores and 1.5GB of RAM.

The server machine is our control plane node. It will run etcd, the API server, and the other control plane components. These are memory-hungry processes, so we allocate 2 CPU cores and 2GB of RAM.

The two worker nodes, node-0 and node-1, will run our actual workloads. Each gets 2 CPU cores and 2GB of RAM. In a real cluster you would probably have more workers and give them more resources, but for learning purposes this is plenty.

All four machines will run Debian 12, which is a stable and well-supported Linux distribution. Kubernetes works on many Linux flavors, but Debian gives us a clean and predictable environment.

Now, about networking. Vagrant automatically creates a NAT network for each VM so they can access the internet. This network uses the 10.0.2.0/24 range, and every VM gets 10.0.2.15 as its first network interface. This might seem odd since they all have the same IP, but it works because each VM has its own isolated NAT network.

The second network interface is where the real action happens. We create a private network in the 192.168.10.0/24 range where all our VMs can talk to each other directly. The jumpbox gets 192.168.10.10, the server gets 192.168.10.100, and the workers get 192.168.10.101 and 192.168.10.102.

Now let us create the Vagrantfile that defines our virtual machines. Create a new file:

cat > Vagrantfile << 'EOF'
BOX_IMAGE = "bento/debian-12"
BOX_VERSION = "202502.21.0"Vagrant.configure("2") do |config|config.vm.define "jumpbox" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/k8s-hardway"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "jumpbox"
      vb.cpus = 2
      vb.memory = 1536
      vb.linked_clone = true
    end
    subconfig.vm.hostname = "jumpbox"
    subconfig.vm.network "private_network", ip: "192.168.10.10"
    subconfig.vm.network "forwarded_port", guest: 22, host: 2210, auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "init.sh"
  endconfig.vm.define "server" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/k8s-hardway"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "server"
      vb.cpus = 2
      vb.memory = 2048
      vb.linked_clone = true
    end
    subconfig.vm.hostname = "server"
    subconfig.vm.network "private_network", ip: "192.168.10.100"
    subconfig.vm.network "forwarded_port", guest: 22, host: 2200, auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "init.sh"
  endconfig.vm.define "node-0" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/k8s-hardway"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "node-0"
      vb.cpus = 2
      vb.memory = 2048
      vb.linked_clone = true
    end
    subconfig.vm.hostname = "node-0"
    subconfig.vm.network "private_network", ip: "192.168.10.101"
    subconfig.vm.network "forwarded_port", guest: 22, host: 2201, auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "init.sh"
  endconfig.vm.define "node-1" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/k8s-hardway"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "node-1"
      vb.cpus = 2
      vb.memory = 2048
      vb.linked_clone = true
    end
    subconfig.vm.hostname = "node-1"
    subconfig.vm.network "private_network", ip: "192.168.10.102"
    subconfig.vm.network "forwarded_port", guest: 22, host: 2202, auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "init.sh"
  endend
EOF

Let me walk through what this file does because there are some important details.

The BOX_IMAGE specifies which base image to use. We use bento/debian-12, which is a well-maintained Debian image from the Bento project. The version pin ensures we all use the same image and avoid surprises.
The linked_clone setting is a nice optimization. Instead of creating a full copy of the base disk for each VM, VirtualBox creates a differencing disk that only stores changes. This saves disk space and speeds up VM creation significantly.
The nicpromisc2 setting puts the second network interface in promiscuous mode. This allows it to see all network traffic on the virtual network, which is necessary for our cluster networking to work properly.
We disable the synced folder because we do not need it and it can cause issues on some setups. We will transfer files between machines using scp instead.

Each VM has a forwarded port for SSH. This means you can SSH to localhost:2210 to reach the jumpbox, localhost:2200 for the server, and so on. Vagrant uses these for its own SSH connections.

Each VM needs some initial configuration when it first boots. We handle this with a shell script that Vagrant runs automatically. Create the init.sh file:

cat > init.sh << 'EOF'
#!/usr/bin/env bash

echo ">>> Starting initial configuration <<<"

echo "[1/7] Configuring shell environment"
echo 'alias vi=vim' >> /etc/profile
echo 'export HISTTIMEFORMAT="%F %T "' >> /etc/profile
ln -sf /usr/share/zoneinfo/UTC /etc/localtime

echo "[2/7] Disabling AppArmor"
systemctl stop apparmor 2>/dev/null
systemctl disable apparmor 2>/dev/null

echo "[3/7] Disabling swap"
swapoff -a
sed -i '/swap/s/^/#/' /etc/fstab

echo "[4/7] Installing required packages"
apt-get update -qq
apt-get install -y -qq tree git jq curl wget vim sshpass net-tools dnsutils > /dev/null 2>&1

echo "[5/7] Setting root password"
echo "root:kubernetes" | chpasswd

echo "[6/7] Configuring SSH"
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
systemctl restart sshd

echo "[7/7] Configuring /etc/hosts"
cat >> /etc/hosts << HOSTS
192.168.10.10  jumpbox
192.168.10.100 server server.kubernetes.local
192.168.10.101 node-0 node-0.kubernetes.local
192.168.10.102 node-1 node-1.kubernetes.local
HOSTS

echo ">>> Initial configuration complete <<<"
EOF
chmod +x init.sh

Let me explain each step because these settings are important for Kubernetes to work correctly.

Disabling AppArmor might seem like a security regression, and in production you would want to keep it enabled and properly configured. For this learning exercise, we disable it to avoid potential conflicts with container runtimes. AppArmor can interfere with how containerd manages containers if not configured correctly.

Disabling swap is a hard requirement for Kubernetes. The kubelet refuses to start if swap is enabled. The reason is that Kubernetes needs to accurately track memory usage for scheduling decisions, and swap makes this unpredictable. When a node runs low on memory, Kubernetes wants to evict pods rather than have them silently slow down due to swapping.

The packages we install are utilities we will need throughout the tutorial. jq is essential for parsing JSON output from kubectl and etcdctl. sshpass lets us script SSH connections with passwords, which is convenient for automation. The networking tools help with debugging.

Setting a root password and enabling password authentication makes it easy to SSH between machines. In production you would use key-based authentication only, but for a lab environment this is much more convenient.

The /etc/hosts entries let us refer to machines by name instead of IP address. This is simpler than setting up a DNS server and works perfectly for our small cluster.

Now we are ready to create our virtual machines. Make sure you are in the project directory and run:

vagrant up

This command reads the Vagrantfile, downloads the base image if needed, creates all four VMs, and runs the initialization script on each one. The first run takes a while because it needs to download the Debian image, which is several hundred megabytes. Subsequent runs are much faster thanks to the linked clone feature.

You should see output for each VM as it boots and runs the provisioning script. If everything goes well, after about 5–10 minutes you will have four running virtual machines.

Check the status of your VMs. If any VM shows as not running, check the VirtualBox application for error messages. Common issues include not enough RAM, VirtualBox extensions not approved in system settings, or conflicts with other virtualization software.

$ vagrant status

Current machine states:

jumpbox                   running (virtualbox)
server                    running (virtualbox)
node-0                    running (virtualbox)
node-1                    running (virtualbox)

The jumpbox is our command center for the rest of this tutorial. Let us connect to it:

vagrant ssh jumpbox

You are now inside the jumpbox VM. Since we configured the init script to add entries to /etc/hosts, you can ping the other machines by name:

ping -c 3 server
ping -c 3 node-0
ping -c 3 node-1

All three should respond successfully. If any of them fail, check that the VMs are running and that the private network was set up correctly.

While we are here, let us verify the system configuration:

cat /etc/os-release | grep PRETTY_NAME

You should see Debian GNU/Linux 12 (bookworm).

Check that swap is disabled:

swapon --show

This should produce no output, indicating swap is off.

Verify the network configuration:

ip addr show eth1

You should see 192.168.10.10 as the IP address for the jumpbox.

Now let us test SSH connectivity to the other machines. From the jumpbox:

ssh root@server hostname

Enter the password “kubernetes” when prompted. You should see “server” printed as the hostname. Test the worker nodes too:

ssh root@node-0 hostname
ssh root@node-1 hostname

If all three commands work, your lab environment is set up correctly.

Typing the password every time gets old fast. Let us set up SSH keys so we can connect without passwords.

On the jumpbox, generate an SSH key pair:

ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519

The ed25519 algorithm is modern and secure. The -N “” flag sets an empty passphrase, which is fine for a lab environment.

Now copy the public key to all other machines:

ssh-copy-id -o StrictHostKeyChecking=no root@server
ssh-copy-id -o StrictHostKeyChecking=no root@node-0
ssh-copy-id -o StrictHostKeyChecking=no root@node-1

Enter the password “kubernetes” for each one. After this, you can SSH without a password:

ssh root@server "echo 'SSH key auth working'"
ssh root@node-0 "echo 'SSH key auth working'"
ssh root@node-1 "echo 'SSH key auth working'"

All three should print the message without asking for a password.

Throughout this tutorial, we will need to reference information about our machines: their IP addresses, hostnames, and which pod CIDR they should use. Let us create a configuration file that captures all this information.

On the jumpbox:

cat > ~/machines.txt << 'EOF'
192.168.10.100 server server.kubernetes.local
192.168.10.101 node-0 node-0.kubernetes.local 10.200.0.0/24
192.168.10.102 node-1 node-1.kubernetes.local 10.200.1.0/24
EOF

The format is: IP address, short hostname, FQDN, and pod CIDR (for worker nodes only). We will reference this file in later scripts to avoid hardcoding values everywhere.

Before we finish this part, let us download all the binaries we will need. This takes a while, so it is better to do it now while we discuss what each component does.

First, let us set up a directory structure:

mkdir -p ~/downloads
cd ~/downloads

Now determine your architecture. On the jumpbox:

ARCH=$(uname -m)
if [ "$ARCH" = "x86_64" ]; then
  ARCH="amd64"
elif [ "$ARCH" = "aarch64" ]; then
  ARCH="arm64"
fi
echo "Architecture: $ARCH"

Set the Kubernetes version we will use:

KUBE_VERSION="v1.32.0"
ETCD_VERSION="v3.5.17"
CONTAINERD_VERSION="2.0.1"
RUNC_VERSION="v1.2.4"
CNI_VERSION="v1.6.2"

Download the Kubernetes control plane binaries:

echo "Downloading kube-apiserver..."
curl -sLO "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kube-apiserver"

echo "Downloading kube-controller-manager..."
curl -sLO "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kube-controller-manager"

echo "Downloading kube-scheduler..."
curl -sLO "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kube-scheduler"

Download kubectl, which we will use for cluster administration:

echo "Downloading kubectl..."
curl -sLO "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kubectl"

Download the worker node binaries:

echo "Downloading kubelet..."
curl -sLO "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kubelet"

echo "Downloading kube-proxy..."
curl -sLO "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kube-proxy"

Download etcd, the distributed key-value store that Kubernetes uses for all cluster state:

echo "Downloading etcd..."
curl -sLO "https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz"
tar -xzf etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz
mv etcd-${ETCD_VERSION}-linux-${ARCH}/etcd .
mv etcd-${ETCD_VERSION}-linux-${ARCH}/etcdctl .
rm -rf etcd-${ETCD_VERSION}-linux-${ARCH}*

Download containerd, the container runtime:

echo "Downloading containerd..."
curl -sLO "https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/containerd-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gz"
mkdir containerd-temp
tar -xzf containerd-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gz -C containerd-temp
mv containerd-temp/bin/* .
rm -rf containerd-temp containerd-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gz

Download runc, the low-level container runtime that containerd uses:

echo "Downloading runc..."
curl -sLO "https://github.com/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.${ARCH}"
mv runc.${ARCH} runc

Download the CNI plugins for container networking:

echo "Downloading CNI plugins..."
curl -sLO "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz"
mkdir cni-plugins
tar -xzf cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz -C cni-plugins
rm cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz

Download crictl, a command-line tool for interacting with CRI-compatible container runtimes:

echo "Downloading crictl..."
CRICTL_VERSION="v1.32.0"
curl -sLO "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz"
tar -xzf crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz
rm crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz

Make all binaries executable:

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl
chmod +x kubelet kube-proxy
chmod +x etcd etcdctl
chmod +x containerd containerd-shim-runc-v2 containerd-stress ctr
chmod +x runc crictl

Verify we have everything:

ls -la

You should see all the binaries we downloaded. Let us also check that they run:

./kubectl version --client
./etcdctl version

Both should print version information without errors.

Let me briefly explain what each of these components does, because understanding their roles is key to understanding Kubernetes as a whole.

The kube-apiserver is the front door to your cluster. Every interaction with Kubernetes, whether from kubectl, the kubelet, or external systems, goes through the API server. It validates requests, authenticates users, and stores the resulting state in etcd.
The kube-controller-manager runs a collection of controllers, each responsible for maintaining a particular aspect of cluster state. The deployment controller watches for deployment changes and creates or updates replica sets. The replica set controller ensures the right number of pods are running. The node controller monitors node health. There are many more, but they all follow the same pattern: watch for changes, compare current state to desired state, and take action to reconcile differences.
The kube-scheduler watches for newly created pods that have no node assigned and selects a node for them to run on. It considers resource requirements, affinity rules, taints and tolerations, and many other factors when making scheduling decisions.
The kubelet is the agent that runs on every worker node. It receives pod specifications from the API server and ensures the containers described in those specs are running and healthy. It works with the container runtime to actually start and stop containers.
The kube-proxy runs on every node and maintains network rules that allow communication to pods from inside and outside the cluster. When you create a Service, kube-proxy sets up the rules that route traffic to the right pods.
etcd is a distributed key-value store that holds all cluster state. Every object you create in Kubernetes, every pod, every service, every secret, is stored in etcd. It uses the Raft consensus algorithm to maintain consistency across multiple nodes, though in our setup we will run just a single etcd instance.
containerd is a container runtime that manages the complete container lifecycle: pulling images, creating containers, attaching storage, setting up networking, and so on. It implements the Container Runtime Interface (CRI) that Kubernetes uses to interact with container runtimes.
runc is a lower-level tool that actually creates and runs containers according to the OCI (Open Container Initiative) specification. containerd uses runc under the hood.
The CNI plugins handle container networking. When kubelet creates a pod, it calls a CNI plugin to set up the network namespace, assign an IP address, and configure routing.

At this point, your lab environment is fully set up. You have four virtual machines that can communicate with each other, SSH keys configured for password-less access, and all the Kubernetes binaries downloaded and ready to deploy.

Let us do a final verification before we move on. Exit from the jumpbox if you are still connected, then verify you can reach it:

exit
vagrant ssh jumpbox -c "echo 'Environment ready'"

You should see “Environment ready” printed.

The Second Part

Building a Kubernetes Cluster From Scratch: Setting Up etcd and Control Plane

In Part 1, we created our lab environment with four virtual machines and downloaded all the Kubernetes binaries. Now comes the real work: setting up the certificate infrastructure and bootstrapping the control plane.

This part is long and dense. We will generate over a dozen certificates, create configuration files for every component, and bring up etcd and the three control plane services. Take your time working through this. If something does not work, go back and check the previous steps carefully. Certificate issues are the number one cause of problems during Kubernetes installation, and a single typo can cause hours of debugging.

Before we start generating certificates, let me explain why Kubernetes needs so many of them.

Kubernetes uses TLS certificates for two purposes: encryption and authentication. Every connection between components is encrypted using TLS, which prevents eavesdropping. But more importantly, certificates prove identity. When the kubelet connects to the API server, it presents a certificate that says “I am the kubelet for node-0.” The API server checks that this certificate was signed by a trusted authority before accepting the connection.

This is called mutual TLS, or mTLS. In regular TLS, like when you browse a website, only the server proves its identity. In mTLS, both sides present certificates and verify each other. This is critical for security in a distributed system like Kubernetes where many components communicate over the network.

Each certificate contains a Common Name (CN) and optionally an Organization (O) field. Kubernetes uses these fields for identity. The CN becomes the username, and the O becomes the group membership. For example, a certificate with CN=system:kube-scheduler tells the API server that requests using this certificate come from the scheduler component.

Here is what we need to create.

A Certificate Authority (CA) that signs all other certificates. Every component trusts certificates signed by this CA.
An admin client certificate for cluster administration. This is what kubectl will use.
A kubelet client certificate for each worker node. These identify the kubelets to the API server.
Client certificates for kube-controller-manager, kube-scheduler, and kube-proxy. Each component authenticates to the API server with its own certificate.
A server certificate for the API server. This proves to clients that they are talking to the real API server.
A key pair for signing service account tokens. This is not a certificate per se, but the API server uses it to sign and verify tokens.

First, connect to the jumpbox where we will do all our work:

vagrant ssh jumpbox

Create a directory for our certificates and configuration files:

mkdir -p ~/certs ~/configs ~/units
cd ~/certs

We will generate all certificates in the certs directory, configuration files in configs, and systemd unit files in units.

The Certificate Authority is the root of trust for our cluster. Every other certificate will be signed by the CA, and every component will be configured to trust certificates signed by it.

First, create the CA configuration file:

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

This configuration defines a signing profile called “kubernetes” that we will use for all certificates. The expiry is set to 8760 hours, which is one year. In production you might want longer-lived CA certificates and shorter-lived component certificates, but for learning purposes one year is fine.

Now create the CA certificate signing request:

cat > ca-csr.json << EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Oregon"
    }
  ]
}
EOF

The CN (Common Name) is “Kubernetes” and the O (Organization) is also “Kubernetes”. The location fields do not matter for functionality but are required by the X.509 standard.

We need the cfssl tools to generate certificates. Install them:

curl -sLO https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_amd64
curl -sLO https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_amd64
chmod +x cfssl_1.6.5_linux_amd64 cfssljson_1.6.5_linux_amd64
sudo mv cfssl_1.6.5_linux_amd64 /usr/local/bin/cfssl
sudo mv cfssljson_1.6.5_linux_amd64 /usr/local/bin/cfssljson

# If you are on Apple Silicon and your VMs are ARM-based, use the arm64 versions instead:curl -sLO https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssl_1.6.5_linux_arm64
curl -sLO https://github.com/cloudflare/cfssl/releases/download/v1.6.5/cfssljson_1.6.5_linux_arm64
chmod +x cfssl_1.6.5_linux_arm64 cfssljson_1.6.5_linux_arm64
sudo mv cfssl_1.6.5_linux_arm64 /usr/local/bin/cfssl
sudo mv cfssljson_1.6.5_linux_arm64 /usr/local/bin/cfssljson

Verify cfssl is working:

cfssl version

Now generate the CA certificate and private key:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

This creates three files: ca.pem (the certificate), ca-key.pem (the private key), and ca.csr (the certificate signing request, which we do not need). Let us verify:

ls -la ca*

You should see ca-key.pem, ca.pem, and ca.csr along with the JSON files we created.

You can examine the CA certificate:

openssl x509 -in ca.pem -text -noout | head -20

You will see the certificate details including the subject (CN=Kubernetes) and that it is a CA certificate.

The admin certificate is what we will use with kubectl to manage the cluster. It needs to be part of the system:masters group, which has full cluster admin privileges.

cat > admin-csr.json << EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:masters",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

Notice that the Organization (O) is set to “system:masters”. This is a special group in Kubernetes that has unrestricted access to the cluster. The CN is “admin”, which will be the username.

Generate the certificate:

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin

This creates admin.pem and admin-key.pem.

Each kubelet needs its own certificate. The certificate identifies which node the kubelet is running on. Kubernetes uses a specific naming convention: the CN must be system:node:<nodename> and the O must be system:nodes.

For node-0:

cat > node-0-csr.json << EOF
{
  "CN": "system:node:node-0",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:nodes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=node-0,node-0.kubernetes.local,192.168.10.101 \
  -profile=kubernetes \
  node-0-csr.json | cfssljson -bare node-0

The hostname flag adds Subject Alternative Names (SANs) to the certificate. These are the names and IP addresses that are valid for this certificate. The kubelet certificate needs the node’s hostname and IP address.

For node-1:

cat > node-1-csr.json << EOF
{
  "CN": "system:node:node-1",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:nodes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=node-1,node-1.kubernetes.local,192.168.10.102 \
  -profile=kubernetes \
  node-1-csr.json | cfssljson -bare node-1

The kube-controller-manager authenticates to the API server using its own certificate:

cat > kube-controller-manager-csr.json << EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

Generate Kube-Proxy Client Certificate:

cat > kube-proxy-csr.json << EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:node-proxier",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy

Generate Scheduler Client Certificate:

cat > kube-scheduler-csr.json << EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler

The API server certificate is special because it needs many Subject Alternative Names. Clients connect to the API server using various names and IP addresses, and the certificate must be valid for all of them.

cat > kubernetes-csr.json << EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

Now we need to specify all the SANs. The API server needs to be reachable by:

The hostname “server” and “server.kubernetes.local”
The IP address 192.168.10.100
The internal cluster IP 10.32.0.1 (the first IP in the service CIDR)
The localhost addresses for internal communication
The special name “kubernetes” and its variations.

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=10.32.0.1,192.168.10.100,server,server.kubernetes.local,127.0.0.1,localhost,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes

You can verify the SANs were included:

openssl x509 -in kubernetes.pem -text -noout | grep -A1 "Subject Alternative Name"

Service accounts use a different authentication mechanism. Instead of certificates, they use tokens signed by a key pair. The API server signs tokens with the private key, and components verify tokens using the public key.

cat > service-account-csr.json << EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Portland",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Oregon"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account

Let us make sure we generated everything:

ls -la *.pem

You should see these files:

admin-key.pem
admin.pem
ca-key.pem
ca.pem
kube-controller-manager-key.pem
kube-controller-manager.pem
kube-proxy-key.pem
kube-proxy.pem
kube-scheduler-key.pem
kube-scheduler.pem
kubernetes-key.pem
kubernetes.pem
node-0-key.pem
node-0.pem
node-1-key.pem
node-1.pem
service-account-key.pem
service-account.pem

That is 18 .pem files total (9 certificates and 9 private keys).

Now we need to copy the appropriate certificates to each machine. Each machine only gets the certificates it needs.

Copy certificates to the server (control plane):

scp ca.pem ca-key.pem kubernetes.pem kubernetes-key.pem \
  service-account.pem service-account-key.pem \
  root@server:~/

Copy certificates to node-0:

scp ca.pem node-0.pem node-0-key.pem root@node-0:~/

Copy certificates to node-1:

scp ca.pem node-1.pem node-1-key.pem root@node-1:~/

Notice that worker nodes only get the CA certificate (to verify other certificates) and their own node certificate. They do not get any private keys except their own.

Kubernetes components use kubeconfig files to locate and authenticate to the API server. A kubeconfig file contains the cluster’s CA certificate, the API server address, and client credentials.

We need to generate kubeconfig files for kubelet (one per node), kube-proxy, kube-controller-manager, kube-scheduler, and admin.

Let us move to the configs directory:

cd ~/configs

First, copy kubectl to a location where we can use it:

sudo cp ~/downloads/kubectl /usr/local/bin/
sudo chmod +x /usr/local/bin/kubectl

Now let us set some variables we will use repeatedly:

KUBERNETES_API_SERVER="https://192.168.10.100:6443"

Generate Kubelet Kubeconfig Files:

For node-0:

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=~/certs/ca.pem \
  --embed-certs=true \
  --server=${KUBERNETES_API_SERVER} \
  --kubeconfig=node-0.kubeconfig

kubectl config set-credentials system:node:node-0 \
  --client-certificate=~/certs/node-0.pem \
  --client-key=~/certs/node-0-key.pem \
  --embed-certs=true \
  --kubeconfig=node-0.kubeconfigkubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:node:node-0 \
  --kubeconfig=node-0.kubeconfigkubectl config use-context default --kubeconfig=node-0.kubeconfig

For node-1:

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=~/certs/ca.pem \
  --embed-certs=true \
  --server=${KUBERNETES_API_SERVER} \
  --kubeconfig=node-1.kubeconfig

kubectl config set-credentials system:node:node-1 \
  --client-certificate=~/certs/node-1.pem \
  --client-key=~/certs/node-1-key.pem \
  --embed-certs=true \
  --kubeconfig=node-1.kubeconfigkubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:node:node-1 \
  --kubeconfig=node-1.kubeconfigkubectl config use-context default --kubeconfig=node-1.kubeconfig

Generate Kube-Proxy Kubeconfig:

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=~/certs/ca.pem \
  --embed-certs=true \
  --server=${KUBERNETES_API_SERVER} \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials system:kube-proxy \
  --client-certificate=~/certs/kube-proxy.pem \
  --client-key=~/certs/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfigkubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfigkubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

Generate Controller Manager Kubeconfig:

The controller manager runs on the same host as the API server, so it can use localhost.

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=~/certs/ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=~/certs/kube-controller-manager.pem \
  --client-key=~/certs/kube-controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-controller-manager.kubeconfigkubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-controller-manager \
  --kubeconfig=kube-controller-manager.kubeconfigkubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig

Generate Scheduler Kubeconfig:

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=~/certs/ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
  --client-certificate=~/certs/kube-scheduler.pem \
  --client-key=~/certs/kube-scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-scheduler.kubeconfigkubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=system:kube-scheduler \
  --kubeconfig=kube-scheduler.kubeconfigkubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig

Generate Admin Kubeconfig:

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=~/certs/ca.pem \
  --embed-certs=true \
  --server=https://127.0.0.1:6443 \
  --kubeconfig=admin.kubeconfig

kubectl config set-credentials admin \
  --client-certificate=~/certs/admin.pem \
  --client-key=~/certs/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=admin.kubeconfigkubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=admin \
  --kubeconfig=admin.kubeconfigkubectl config use-context default --kubeconfig=admin.kubeconfig

Distribute Kubeconfig Files. Copy the appropriate kubeconfig files to each machine.

To the server:

scp admin.kubeconfig kube-controller-manager.kubeconfig kube-scheduler.kubeconfig root@server:~/

To node-0:

scp node-0.kubeconfig kube-proxy.kubeconfig root@node-0:~/

To node-1:

scp node-1.kubeconfig kube-proxy.kubeconfig root@node-1:~/

Generate the Data Encryption Config: Kubernetes can encrypt secret data at rest in etcd. This means that even if someone gains access to the etcd data files, they cannot read your secrets without the encryption key.

First, generate a random encryption key:

ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo $ENCRYPTION_KEY

Save this key somewhere safe. You will need it if you ever need to recover your cluster.

Now create the encryption config:

cat > encryption-config.yaml << EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

This configuration tells Kubernetes to encrypt secrets using AES-CBC with the key we generated. The identity provider at the end is a fallback that allows reading unencrypted secrets (for backward compatibility during migration).

Copy the encryption config to the server:

scp encryption-config.yaml root@server:~/

Now we begin the actual installation. etcd is the first component we set up because everything else depends on it. etcd stores all cluster state, including pod definitions, service configurations, secrets, and more.

SSH into the server:

ssh root@server

Create the necessary directories:

mkdir -p /etc/etcd /var/lib/etcd
chmod 700 /var/lib/etcd

Copy the certificates and move them into place:

cp ca.pem kubernetes.pem kubernetes-key.pem /etc/etcd/

Move the etcd binaries (we will copy them from jumpbox first, so exit back to jumpbox):

exit

From jumpbox, copy the binaries:

scp ~/downloads/etcd ~/downloads/etcdctl root@server:~/

SSH back to server:

ssh root@server

Install the binaries:

mv etcd etcdctl /usr/local/bin/

Verify:

etcd --version
etcdctl version

Now create the systemd unit file for etcd. We are going to run etcd with HTTP for simplicity in this tutorial. In production, you would use HTTPS with client certificate authentication, but that adds complexity that can obscure the core concepts:

cat > /etc/systemd/system/etcd.service << EOF
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
  --name server \\
  --data-dir=/var/lib/etcd \\
  --listen-peer-urls http://127.0.0.1:2380 \\
  --listen-client-urls http://127.0.0.1:2379 \\
  --initial-advertise-peer-urls http://127.0.0.1:2380 \\
  --advertise-client-urls http://127.0.0.1:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster server=http://127.0.0.1:2380 \\
  --initial-cluster-state new
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

Let me explain the key options:

The name flag gives this etcd member a unique name. In a multi-node cluster, each member needs a different name.
The data-dir is where etcd stores its data. We set restrictive permissions on this directory earlier.
The listen-peer-urls is where etcd listens for communication from other etcd members. We use 127.0.0.1 because we have only one member.
The listen-client-urls is where etcd listens for client connections (from the API server).
The initial-cluster defines all members of the cluster. Since we have only one member, it just lists our server.

Start and enable etcd:

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

Check the status:

systemctl status etcd --no-pager

It should show “active (running)”. If it shows failed, check the logs:

journalctl -u etcd --no-pager -n 50

Verify etcd is working by listing members:

etcdctl member list

You should see one member. Check the cluster health:

etcdctl endpoint health

It should report as healthy.

The API server is the central hub of Kubernetes. Every other component talks to it, and it is the only component that talks directly to etcd.

First, create the directory for Kubernetes configuration:

mkdir -p /etc/kubernetes/config /var/lib/kubernetes

Move the certificates, encryption config, and kubeconfigs into place:

mv ca.pem ca-key.pem kubernetes.pem kubernetes-key.pem \
  service-account.pem service-account-key.pem \
  encryption-config.yaml /var/lib/kubernetes/

mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
mv kube-scheduler.kubeconfig /var/lib/kubernetes/
mv admin.kubeconfig /var/lib/kubernetes/

Copy the API server binary from jumpbox. Exit to jumpbox first.

From jumpbox:

scp ~/downloads/kube-apiserver ~/downloads/kube-controller-manager \
  ~/downloads/kube-scheduler ~/downloads/kubectl root@server:~/

SSH back:

ssh root@server

Install the binaries:

mv kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
chmod +x /usr/local/bin/kube-apiserver /usr/local/bin/kube-controller-manager \
  /usr/local/bin/kube-scheduler /usr/local/bin/kubectl

Now create the API server systemd unit file. This is where things get interesting because there are a lot of flags.

cat > /etc/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
  --advertise-address=192.168.10.100 \\
  --allow-privileged=true \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/log/audit.log \\
  --authorization-mode=Node,RBAC \\
  --bind-address=0.0.0.0 \\
  --client-ca-file=/var/lib/kubernetes/ca.pem \\
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --etcd-servers=http://127.0.0.1:2379 \\
  --event-ttl=1h \\
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
  --runtime-config=api/all=true \\
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \\
  --service-account-signing-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-account-issuer=https://192.168.10.100:6443 \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\
  --v=2
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

That is a lot of flags. Let me explain the important ones:

The advertise-address is the IP that will be advertised to other cluster members. This is where the API server is reachable.
The authorization-mode is set to Node,RBAC. Node authorization allows kubelets to access resources they need. RBAC enables role-based access control.
The client-ca-file specifies the CA used to verify client certificates. Any certificate signed by this CA will be accepted.
The enable-admission-plugins lists admission controllers that process requests. NamespaceLifecycle ensures you cannot create objects in non-existent namespaces. NodeRestriction limits what kubelets can modify. ServiceAccount ensures pods get service account tokens automatically.
The etcd-servers tells the API server where to find etcd.
The encryption-provider-config points to our encryption configuration for secrets.
The kubelet-certificate-authority, kubelet-client-certificate, and kubelet-client-key are used when the API server connects to kubelets (for logs, exec, etc.).
The service-account-key-file and service-account-signing-key-file are used to sign and verify service account tokens.
The service-cluster-ip-range defines the IP range for ClusterIP services. We use 10.32.0.0/24, which gives us 254 possible service IPs.
The tls-cert-file and tls-private-key-file are the API server’s own certificate and key.

Let us set up the Controller Manager. Create the systemd unit file:

cat > /etc/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
  --bind-address=0.0.0.0 \\
  --cluster-cidr=10.200.0.0/16 \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --leader-elect=true \\
  --root-ca-file=/var/lib/kubernetes/ca.pem \\
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --use-service-account-credentials=true \\
  --v=2
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

The cluster-cidr is the range used for pod IPs across the entire cluster. We use 10.200.0.0/16, which will be subdivided into smaller ranges for each node.

The cluster-signing-cert-file and cluster-signing-key-file let the controller manager sign certificates for kubelet certificate rotation.

The leader-elect flag enables leader election. In a multi-master setup, only one controller manager is active at a time. With a single master, this still works fine.

Let us continue setting up scheduler.

First, create the scheduler configuration file:

cat > /etc/kubernetes/config/kube-scheduler.yaml << EOF
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
EOF

Now create the systemd unit file:

cat > /etc/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
  --config=/etc/kubernetes/config/kube-scheduler.yaml \\
  --v=2
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

Now let us start all three services:

systemctl daemon-reload
systemctl enable kube-apiserver kube-controller-manager kube-scheduler
systemctl start kube-apiserver kube-controller-manager kube-scheduler

Wait a few seconds for them to start, then check their status:

systemctl status kube-apiserver --no-pager
systemctl status kube-controller-manager --no-pager
systemctl status kube-scheduler --no-pager

All three should show “active (running)”. If any show failed, check the logs:

journalctl -u kube-apiserver --no-pager -n 50
journalctl -u kube-controller-manager --no-pager -n 50
journalctl -u kube-scheduler --no-pager -n 50

Let us verify everything is working. First, check that the API server is responding:

kubectl cluster-info --kubeconfig /var/lib/kubernetes/admin.kubeconfig

You should see:

Kubernetes control plane is running at https://127.0.0.1:6443

Check the component statuses:

kubectl get componentstatuses --kubeconfig /var/lib/kubernetes/admin.kubeconfig

Note: The componentstatuses endpoint is deprecated in newer Kubernetes versions and may show warnings or incomplete information. A better way to verify is to check if services are responding:

kubectl get --raw='/readyz?verbose' --kubeconfig /var/lib/kubernetes/admin.kubeconfig

This should show many checks, all passing.

Check the default namespace and services:

kubectl get namespaces --kubeconfig /var/lib/kubernetes/admin.kubeconfig
kubectl get services --kubeconfig /var/lib/kubernetes/admin.kubeconfig

You should see the default, kube-system, kube-public, and kube-node-lease namespaces. The kubernetes service in the default namespace should exist, pointing to 10.32.0.1.

When the API server needs to connect to kubelets (for logs, exec, port-forward, etc.), it authenticates using its kubernetes certificate. We need to create RBAC rules that allow this.

Create a ClusterRole with the necessary permissions:

cat > /tmp/kube-apiserver-to-kubelet.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

kubectl apply -f /tmp/kube-apiserver-to-kubelet.yaml --kubeconfig /var/lib/kubernetes/admin.kubeconfig

This creates a ClusterRole that allows access to kubelet endpoints and binds it to the “kubernetes” user, which is the identity in the API server’s certificate.

Let us make sure secret encryption is working. Create a test secret:

kubectl create secret generic test-secret --from-literal=mykey=mydata \
  --kubeconfig /var/lib/kubernetes/admin.kubeconfig

Now query etcd directly to see how it is stored:

etcdctl get /registry/secrets/default/test-secret | hexdump -C | head -20

You should see “k8s:enc:aescbc:v1:key1” at the beginning of the data, indicating it is encrypted. The actual secret value should not be visible in plaintext.

Clean up the test secret:

kubectl delete secret test-secret --kubeconfig /var/lib/kubernetes/admin.kubeconfig

The control plane is ready, but we cannot run any workloads yet. There are no worker nodes registered with the cluster. If you try to run a pod now, the scheduler will not find any nodes to place it on.

Before moving on, exit from the server and verify you can reach the API server from the jumpbox:

exit

From jumpbox, let us create a kubeconfig that points to the server’s external IP:

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority=~/certs/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.100:6443 \
  --kubeconfig=~/.kube/config

kubectl config set-credentials admin \
  --client-certificate=~/certs/admin.pem \
  --client-key=~/certs/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=~/.kube/configkubectl config set-context default \
  --cluster=kubernetes-the-hard-way \
  --user=admin \
  --kubeconfig=~/.kube/configkubectl config use-context default --kubeconfig=~/.kube/config

Now test it:

kubectl get nodes

You should see “No resources found” because there are no worker nodes yet. That is expected.

kubectl get namespaces

You should see the default namespaces listed.

In Part 3, we will set up the worker nodes. We will install containerd as the container runtime, configure kubelet and kube-proxy, and set up pod networking so containers on different nodes can communicate with each other. By the end of Part 3, you will be able to run actual workloads on your cluster.

Building a Kubernetes Cluster From Scratch — Part 1 and Part 2

The first part

The Second Part

Building a Kubernetes Cluster From Scratch: Setting Up etcd and Control Plane

Written by Sigrid Jin