Raspberry Pi Kubernetes Cluster with Cilium CNI

Hits: 411

Raspberry Pi Kubernetes cluster with Cilium CNI Seems very flexible

I need to do a proof of concept with a good container network plugin (CNI) while our on prem server room is getting fixed. This is No reason to stop working. The previous kubernetes cluster used the Flannel plugin. But flannel suffers from changes due to growth and hardware variety. Flannel install templates are not idempotent, so when you run them again with new settings they utterly brake your Kubernetes Cluster! So I am trying to use Raspberry Pi  kubernetes to setup a replacement. I used Raspberry pi 4 8 gb k8s cluster

We researched  popular Kubernetes CNi plugins. Calico seems like a good option, but it only works in  a network with professional routers  that support BGP and the POC is going to be done at home.

I read up on the newest CNI called Cilium, Cilium makes use of a new secure sandbox technology available in the recent versions of the Linux Kernel called eBPF. So, Raspberry Pi Kubernetes can use Cilium, and apparently extremely large clusters too, whether ARM64 or regular Intel AMD64 style, even 32 bit. However, don’t write off using Raspberry Pi as Kubernetes nodes, as Raspberry Pi and other IOT  edge devices will are becoming more and more important, this in addition to smart cities.

MetalLB Load Balancer completes the configuration of your Raspberry Pi Kubernetes cluster

To complete the Raspberry Pi Kubernetes cluster, I added installing Metallb Load Balancer that allows you to easily expose APIs or Websites with an external IP address.

Raspberry Pi OS as the base for Kubernetes

Start by installing the 64 bit Raspberry PI OS on SSD card for each Pi in the cluster, Then boot and login to the Master. I Named the Master kubemaster and nodes some simple numbered system,  if you use the same naming for the master and nodes, you might get confused,  I did.

Login and update / upgrade the OS.

apt update && apt upgrade -y

reboot then run the upgrade again

Run “apt autoremove” if it says to

After you have upgraded install the screen app. Run the screen command so that your session and upgrades are not disrupted.

sudo apt-get install screen

Now we can start the work to install kubernetes cluster on Raspberry Pi. I used Ethernet cables, but it should work with the Wifi too. You just need to make sure that the cluster initiation is connected to the correct NIC. So record all IP addresses. I put tiny stickers on the ethernet connection on the pi, BTW you should reserve the IP addresses if your network DHCP often changes them.

Rapberry pi 4 8 gb k8s cluster

Compile and propagate the kernel with 48 bit page

Cilium expects larger pages of memory, the size of 48 bits required by Cilium is set smaller for Raspberry Pi. So, the kernel needs to be re-compiled and the kernel and module dependencies  copied to all Pi nodes. I realized this only after deploying kubernetes, when I tried to deploy the Cilium CNI, I found some errors, as follows:

root@kubemaster:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-ln87t 0/1 CrashLoopBackOff 4 (8s ago) 2m57s
kube-system cilium-operator-69b677f97c-5flmt 1/1 Running 0 2m57s
(edited)
Warning Unhealthy 8m43s (x4 over 8m49s) kubelet Startup probe failed: Get "http://127.0.0.1:9879/healthz": dial tcp 127.0.0.1:9879: connect: connection refused

https://github.com/google/tcmalloc

https://github.com/raspberrypi/linux/issues/4375#issuecomment-863931118

The discussions above had the following:

I recompiled the kernel with CONFIG_ARM64_VA_BITS_48 = y. It works.

I am currently testing it as a kubernetes worker node (k8s 1.21.1, containerd 1.4.4), with raspbian os 32 bit, armv8 core, the node runs 32/64 bit containers. I will monitor the system for a few days, if there are no problems I will install the new kernel on the other rpi3/4 nodes. I don’t seem to have noticed any performance issues or failing services.

Build Raspberry Pi Kernel

Here in the beginning of the project  is a good place to compile the kernel, although it is only required for installing Cilium. If you are using a different CNI you might be able to skip this.

To compile a new kernel, you need to:

  • download the git repo for Raspberry Pi
  • make
  • edit a file
  • make the kernel
  • install kernel by copying it
  • editing another file

Be sure to run all of this after the screen command is running. build can take a long time. I did it on the pi itself, they say that if you do the build on a faster PC that it will go faster.

So run:

screen
sudo apt install git bc bison flex libssl-dev make ncurses-devel libncurses-dev
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux
KERNEL=kernel8
make bcm2711_defconfig
This  builds a lot of stuff. On the Pi itself it takes a while. I didn’t time it, but you will be very bored if you just watch the make
If your ssh session or the like got inter  rupted, just run “screen -x” to get back to the make output. Among other stuff that it does it creates a .config file in the linux directory, this has the default kernel settings, so edit this .config file.
vi .config
In my case around line 382 in that file is the default setting of
CONFIG_ARM64_VA_BITS_39. Be sure that any reference to 39 bits is commented out. Instead set it to:
CONFIG_ARM64_VA_BITS_48=y

CONFIG_ARM64_VA_BITS=48
Now run:
make -j4 Image.gz modules dtbs
sudo make modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm64/boot/Image.gz /boot/kernel8-48bit.img

Instead of Bios Raspberry Pi OS uses /boot/config.txt

In the last command we copied the new kernel without overwriting the default kernel8.img file. So you need to tell the “bios” which kernel to boot from.  I learned that Raspberry pi OS uses the /boot/config.txt instead of Bios.   This makes it really easy to switch your projects by changing the Micro SD card.

Add the following line to your /boot/config.txt file

kernel=kernel8-48bits.imgf

Now reboot your Raspberry pi and hope that it boots.

Now you need to propagate the new kernel and it’s built modules to all of your Raspberry pi nodes.

Copy to the nodes:
/boot/kernel8-48bit.img
/usr/lib/modules/5.15.65-v8-48bits+

The exact name of the newly built 48bits modules might be different

Remember to edit the config.txt on each node.

If I had a lot of nodes or need to constantly repeat this task, I would automate with ansible.

Now you can continue towards your Kubernetes cluster on Raspberry pi.

Disable Swap File

Disabling Swap in Raspberry Pi is more effort than other Debian variants. You need disable it with dphys-swapfile and configure the swap config  to 0.

# disable swap
sudo dphys-swapfile swapoff && \
sudo dphys-swapfile uninstall && \
sudo update-rc.d dphys-swapfile remove

Now edit the config file. use your favorite editor. I use vi

vi /etc/dphys-swapfile

change line to CONF_SWAPSIZE=0

Configure Cgroups and install Containerd

Kubernetes uses cgroups to control resources. It is not optional. We are going to use the default of containerd for running kubernetes  1.25 , so it needs to be set for Containerd, so kubernetes can control stuff, especially the memory with it.

I won’t further complicate things by giving the option to use Docker instead of containerd.

Raspberry Pi OS is configured to use cgroups for most things, except for memory, Cgroup memory controlling is needed by Kubernetes. After you enable it, Containerd also  needs to know about it.

Raspberry pi uses the /boot/cmdline.txt for startup command that passes parameters to the Linux kernel.

Run:

sudo echo " cgroup_enable=memory cgroup_memory=1" >> /boot/cmdline.txt

You can check the startup command line with

cat /boot/cmdline.txt

reboot

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

The following modprobe commands will only work if you correctly copied the modules directory to the nodes. Overlay and  br_netfilter have module issues that the new modules need to be in the correct location.

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

sudo apt-get update
sudo apt-get -y install containerd
sudo apt-get -y install containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Next, edit the containerd config file that you just created

vi /etc/containerd/config.toml

# do the following

#look for the line - search for "options" - seems like line 96
#[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
#
#then add the line under
# SystemdCgroup = true
#
# should look like the following with the S of Systemd under the l from plugins, 
# the file is much more indented than here.
#
 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
   SystemdCgroup = true

Linux Kernel needs to properly load containerd to use. So, restart containerd and check that it’s ok.

sudo systemctl restart containerd
systemctl status containerd.service

Cri-O confuses the universe and won’t seemlessly work with kubernetes if you don’t set it to use the correct socket, so run:

crictl config runtime-endpoint unix:///var/run/containerd/containerd.sock

Install Kubernetes

 

Initiate the Master

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

Join the node to the Cluster

 

Unless you have DNS entries for your pis, you need to add them to your /etc/hosts file. Use correct information, not what’s below. Raspberry pi likes to assign the hostname to 127.0.1.1  . I just left that and added the other nodes with real ip addresses.

 /etc/hosts
#127.0.1.1 kubemaster
#172.21.0.47 raspberrypi2

We ran above crictl command to tell it to use the correct socket so we won’t need –cri-socket unix:///var/run/containerd/containerd.sock. As this is a POC we can safely run –token-ttl=0 which creates a token that lasts forever, consider not using it for a real environment.

From Here the instructions are different for the Nodes and Master, First we initiate the master.  

kubeadm init --token-ttl=0 --skip-phases=addon/kube-proxy

You will be instructed to use the newly created Kubernetes credentials, like the following. Also be sure to copy the “Join” command at the end of the initialization output.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

or

export KUBECONFIG=/etc/kubernetes/admin.conf

Without CNI coredns pods are stuck ! So don’t be annoyed that your coredns pod is stuck.

You won’t be able to run anything on nodes until you have Network Overlay. We are using Cilium,  as Flannel is fickle and Calico requires fancy network equipment so won’t work for our POC.

Node Install

Everything is the same as installing  the Master Control Plane including needing the compiled kernel and modules, until the kubeadm command, For joining the nodes use the copied command from the kubeadm init on the master. Although it won’t work without CNI, It’s best to connect a Node to the Cluster. When you finish installing Cilium CNI everything will start working together as a cluster.

kubeadm join ........

Finally Install the Cilium CNI

The default installation with cilium doesn’t allow for ingress and load balancer.  This is because the kube-proxy doesn’t use cilium correctly. So the kubernetes deployment needs to leave out kube-proxy. Instead of this, cilium uses it’s own methods, which likely work faster, as it is not side-carted like other CNIs, as cilium communicates directly through the kernel and does not need iptables for it’s routing like side carted ingress.

The following cilium installation  code also installs Cilium CNI.

Add Cilium https://docs.cilium.io/en/stable/

# https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=arm64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Install Helm on the master, so that you can deploy cilium

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

Now add the Cilium repo to your new Helm installation

helm repo add cilium https://helm.cilium.io/

Run the Cilium Helm deployment with correct environment variables.

USE correct api server address. This is likely your master server, unless you specified otherwise in your Raspberry Pi Kubernetes deployment. You need to use Cilium version 1.12 or above, as previously it didn’t support Egress and Load Balancing.

API_SERVER_IP=<YOUR API SERVER IP>
# Kubeadm default is 6443
API_SERVER_PORT=6443
helm install cilium cilium/cilium –version 1.12.1 \
–namespace kube-system \
–set kubeProxyReplacement=strict \
–set k8sServiceHost=${API_SERVER_IP} \
–set k8sServicePort=${API_SERVER_PORT}

 

Verify  that cilium is up and running.

kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement

or with verbose

kubectl -n kube-system exec ds/cilium -- cilium status --verbose

Deploy a stateless nginx on Your Raspberry Pi Kubernetes cluster with Cilium

We will follow the directions from the following link, it guides you into making a node port. Other docs tell you how to use a load balanacer with set IP Address.

https://docs.cilium.io/en/stable/gettingstarted/kubeproxy-free/#kubeproxy-free

kubectl apply -f https://k8s.io/examples/application/deployment.yaml

kubectl expose deployment nginx-deployment --type=NodePort --port=80

kubectl -n kube-system exec ds/cilium -- cilium service list

node_port=$(kubectl get svc nginx-deployment -o=jsonpath='{@.spec.ports[0].nodePort}')

curl 127.0.0.1:$node_port

Load Balancing and Egress for external traffic: MetalLB

When I started this project I thought that I would use the Cilium Egress feature, However, this requires a router that supports iBGP, which most of Ya’ll don’t have. So I went back to using MetalLB, which previously worked with the Flannel CNI.

Install Hubble for cilium visibility, it shows low level whats happening

 

export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ “$(uname -m)” = “aarch64″ ]; then HUBBLE_ARCH=arm64; fi
curl -L –fail –remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum –check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

Now Install Metal LB so you can use your local IP Pool without a BGP router

kubectl create secret generic -n metallb-system memberlist –from-literal=secretkey=”$(openssl rand -base64 128)”
secret/memberlist created

helm repo add metallb https://metallb.github.io/metallb

helm install metallb metallb/metallb

Now create 2 seperate yaml files and apply them. For IP Addresses use some from your own pool. It’s also wise to block them off in your router. I had first used old yaml config files for metallb and I got <pending> forever instead of an external IP address.

vi metallb-layer-2-config-home-net.yml

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
namespace: metallb-system
spec:
addresses:
- 172.21.0.60-172.21.0.65

# save and apply 
kubectl apply -f metallb-layer-2-config-home-net.yml

 

Now install the second MetalLB file to advertise the IP addresses in your LAN. Without this I was only able to access the nginx from on the kubemaster node.

 

vi metal-lb-layer-2-advert.yaml

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: example
namespace: metallb-system

# save and apply 
kubectl apply -f metal-lb-layer-2-advert.yaml

#Now you can test metallb by deploying Nginx – this is incredibly important, although we are going to only check for the default nginx site, once you have nginx up, you can configure it to proxy your APIs, or use whatever API you like , like Swagger.

git clone https://github.com/jodykpw/metallb-nginx-demo.git
cd metallb-nginx-demo/
helm install nginx-demo ./

After a few moments check that the service is up.

Check that all is fine with the following command, You should see an IP address in the “External-IP” column for nginx.

kubectl get svc

You can copy the external IP address and run curl from the command line or declare it

export SERVICE_IP=$(kubectl get svc --namespace default nginx --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
curl $SERVICE_IP

Since we ran the secnd yaml which advertises the IP addresses you should see the nginx page from anywhere on the same Lan network. So paste the external IP address into chrome.

Epilogue

After installing this also with a newer ubuntu 22. I noticed that at this point history Sep 2022 , the newest version of containerd does NOT work with kubernetes 1.25 and perhaps 1.24 . You need to use version 1.4.13 of containerd.

I downloaded containerd, runc and (runc might work just from apt) with the following

wget http://ftp.debian.org/debian/pool/main/r/runc/runc_1.1.4+ds1-1_amd64.deb ### MAYBE try to just apt install runc

wget http://ftp.debian.org/debian/pool/main/c/containerd/containerd_1.4.13~ds1-1~deb11u2_amd64.deb

 

%d bloggers like this: