weekends are for leisure

Learning more Kubernetes

Tags: #Kubernetes  #Linux  #Debian 

A few months ago I got the urge to refresh and expand my Kubernetes skills. I know the basics, but have never managed a cluster. Several companies I worked at used Kubernetes in some capacity, but sadly I was never on those teams. But that’s ok! I’m no stranger to learning on my own, and it’s a great excuse to expand my homelab.

I thought about using minikube or kind to create a cluster, but wanted a “k8s on bare metal” experience because I like tinkering with real hardware. So I searched ebay and purchased three Lenovo 8th-generation Intel i3 machines (model m720s), for $60 each. They came with 8 GB of RAM and 128GB NVMEs, which should be plenty for a learning cluster.

Prepping for installation

While the machines were in transit, I read through the kubeadm cluster creation docs, and put together an installation script for Debian. I chose to install Kubernetes version 1.36 since it’s older and will let me practice cluster upgrades without having to wait for new versions to come out.

Here’s the script that prepares a Debian machine for becoming either a k8s control plane node or worker node.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

#!/bin/bash

# K8S SETUP FOR DEBIAN
# https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
# This installs the foundation for control plane nodes AND worker nodes

CONTAINERD_VERSION="2.3.1"
RUNC_VERSION="1.4.2"
CNI_VERSION="1.9.1"
K8S_VERSION="1.36"
KUBECTL_VERSION="1.36.0"

set -e
# TODO: root needs this in .bashrc
export PATH="/usr/sbin:/sbin:$PATH"

apt install neovim tmux sudo curl

# ensure swap is off
systemctl mask swap.target


# CONTAINER RUNTIME SETUP
# https://kubernetes.io/docs/setup/production-environment/container-runtimes/

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF


# As of 1.22, kubeadm defaults to systemd as a cgroup driver, so nothing for us to do regarding cgroups


# CONTAINERD SETUP
# https://github.com/containerd/containerd/releases
curl -LO https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz
curl -LO https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz.sha256sum
curl -LO https://raw.githubusercontent.com/containerd/containerd/main/containerd.service

cat containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz.sha256sum | sha256sum --check

tar Cxzvf /usr/local containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gz

# Debian doesn't seem to have this folder by default
mkdir -p /usr/local/lib/systemd/system
cp containerd.service /usr/local/lib/systemd/system/containerd.service
systemctl daemon-reload
systemctl enable --now containerd


# RUNC INSTALLATION
# https://github.com/opencontainers/runc/releases
curl -LO https://github.com/opencontainers/runc/releases/download/v${RUNC_VERSION}/runc.amd64
curl -LO https://github.com/opencontainers/runc/releases/download/v${RUNC_VERSION}/runc.sha256sum

# runc.sha256sum has lots of entries that cause problems
grep runc.amd64 runc.sha256sum | sha256sum --check
install -m 755 runc.amd64 /usr/local/sbin/runc


# CNI PLUGINS
# https://github.com/containernetworking/plugins/releases
curl -LO https://github.com/containernetworking/plugins/releases/download/v${CNI_VERSION}/cni-plugins-linux-amd64-v${CNI_VERSION}.tgz
curl -LO https://github.com/containernetworking/plugins/releases/download/v${CNI_VERSION}/cni-plugins-linux-amd64-v${CNI_VERSION}.tgz.sha256

cat cni-plugins-linux-amd64-v${CNI_VERSION}.tgz.sha256 | sha256sum --check

mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v${CNI_VERSION}.tgz


# KUBECTL INSTALLATION

apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

systemctl enable --now kubelet

After the machines arrived, I used the above script to configure the first one. Then, I created a local DNS record for my control plane endpoint in dnsmasq (not shown), and created the cluster with the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# kube-router requires setting a --pod-network-cidr
kubeadm init --control-plane-endpoint k8s.home.arpa --pod-network-cidr 10.0.0.0/16

export KUBECONFIG=/etc/kubernetes/admin.conf

# kube-router networking addon
kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml

# permit this control plane node to be my first worker node
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Note that I’m specifying a control plane endpoint. The cluster creation docs mention why this is recomended:

(Recommended) If you have plans to upgrade this single control-plane kubeadm cluster to high availability you should specify the –control-plane-endpoint to set the shared endpoint for all control-plane nodes. Such an endpoint can be either a DNS name or an IP address of a load-balancer.

Moving to HA would require at least 3 control plane nodes, and may or may not be something I dive into later.

Don’t change IPs willy-nilly

At this point, I realized I was so excited to get k8s running that I forgot to fully plan how this cluster would fit into my home network. So I took a pause, and decided to reserve 3 addresses outside of my normal DHCP range. I updated dnsmasq and got back to the control plane node.

Given that I’d just configured dnsmasq to give this machine a different IP address, I rebooted to pick up the new address. When the machine came back up, Kubernetes was not happy! From what I read, it certainly seems possible to change the IP address of an existing control plane node, but it’s quite involved. So instead of spelunking through several lengthy configuration files, I decided to simply wipe the system and reinstall.

Adding a worker

With that completed, the next task was to add a worker node. I used the same Debian shell script as above to get kubelet running, and then ran the additional command to join the node to the cluster as a worker.

1
2
3

kubeadm join --token <token> \
    <control-plane-host>:<control-plane-port> \
    --discovery-token-ca-cert-hash sha256:<hash>

Hooray! Here are the nodes:

1
2
3
4

alan@thinkpad:~/Projects/server-setup/20260521.k8s$ kubectl get nodes
NAME    STATUS   ROLES           AGE   VERSION
kube1   Ready    control-plane   24d   v1.36.1
kube2   Ready    <none>          24d   v1.36.1

I mentioned at the start of this post that I purchased three machines for this adventure, and as you can see above my cluster only has two. I’m currently using the third to experiment with hardening Linux servers, so it may join the cluster at a later date.

Now that I’ve got a cluster, I need some containers to run. But I’ll save that journey for another post. Thanks for reading.



Older Post