Fredrick Biering

Upgrading K3s

Tue, 24 Mar 2026 00:00:00 +0000

I haven’t upgraded the K3s cluster in the homelab for quite some time. Now seemed like a good time to do a round of upgrades before starting a new project.

What has been upgraded:

Item	New version
K3s cluster	`v1.35.1+k3s1`
Calico	`v3.31.x`
ArgoCD	`3.x`
Longhorn	`1.10.x` (Helm chart version)
Sealed Secrets	`2.18.x` (Helm chart version)
step-ca	`1.29.x` (Helm chart version)
step-issuer	`1.9.9` (Helm chart version)
MetalLB	`0.15.3` (Helm chart version)
Prometheus	`28.13.x` (Helm chart version)
Loki	`6.53.x` (Helm chart version)
opentelemetry-collector	`0.146.1` (Helm chart version)
cert-manager	`1.19.x` (Helm chart version)
pve-exporter	`3.8.1`

An excerpt of the upgrade process is detailed in the following sections.

K3s

For K3s, the upgrade procedure boils down to:

Grab the latest version of the K3s installation script from https://get.k3s.io and overwrite the previous version stored in the Ansible role.

Change the release channel and version in the inventory:

--- a/inventory.ini
+++ b/inventory.ini
@@ -1,8 +1,8 @@
[all:vars]
# Channels and versions for k3s is located at
# https://update.k3s.io/v1-release/channels
-k3s_channel=v1.34
-k3s_version=v1.34.4+k3s1
+k3s_channel=v1.35
+k3s_version=v1.35.0+k3s1
k3s_install_script_src=installation_scripts/install_k3s.sh
k3s_install_script_dest=/usr/local/bin/install_k3s.sh
reinstall=false

Rerun the Ansible playbook with -e reinstall=true to trigger a rerun of the installation script using the new version. The playbook uses serial: 1 to upgrade a single node at a time, starting with the control plane nodes before proceeding to the worker nodes.

Thats it. I have yet to experience any issues with this approach and I commend the maintainers of K3s for providing a great upgrade experience.

This is during the upgrade process to v1.32.0+k3s1. The control plane nodes have been upgraded, worker nodes are in the process of being upgraded:

Calico

When upgrading Calico there was an issue with the CPU architecture of the control plane node VMs in Proxmox. I had not switched the architecture from qemu to x86-64-v2-AES, resulting in this error:

csi-node-driver-registrar This program can only be run on AMD64 processors with v2 microarchitecture support.
calico-csi This program can only be run on AMD64 processors with v2 microarchitecture support.

When it started to fail I tried rolling back the version of Calico to v3.30.x, but doing so proved more difficult (for several reasons) than just going ahead with the upgrade. I drained the control plane nodes, switched the VM CPU architecture, rebooted, and once all 3 nodes were fixed I had version v3.31.x running.

Changes to the Terraform Proxmox VM config for control plane nodes:

--- a/environments/production/terraform.tfvars
+++ b/environments/production/terraform.tfvars
@@ -7,7 +7,7 @@ vms = [
     ip                  = "10.0.3.9/24"
     memory              = 4096
     cores               = 4,
-    cpu_architecture    = "qemu64"
+    cpu_architecture    = "x86-64-v2-AES"
     bootdisk_size       = "20"
     disks               = []
     tags                = ["k8s", "control", "ubuntu-24.04"]
@@ -20,7 +20,7 @@ vms = [
     ip                  = "10.0.3.10/24"
     memory              = 4096
     cores               = 4,
-    cpu_architecture    = "qemu64"
+    cpu_architecture    = "x86-64-v2-AES"
     bootdisk_size       = "20"
     disks               = []
     tags                = ["k8s", "control", "ubuntu-24.04"]
@@ -33,7 +33,7 @@ vms = [
     ip                  = "10.0.3.11/24"
     memory              = 4096
     cores               = 4
-    cpu_architecture    = "qemu64"
+    cpu_architecture    = "x86-64-v2-AES"
     bootdisk_size       = "20"
     disks               = []
     tags                = ["k8s", "control", "ubuntu-24.04"]

ArgoCD

ArgoCD had a long chain of releases since my last upgrade, reading changelogs took most of the time. Not only was the major version of ArgoCD itself bumped from 2.x to 3.x (upgrade notes), but the Helm chart had gone through a few major releases from 7.xto 9.x. I had to follow the advice in this issue and disable redis.ha temporarily to be able to upgrade to 8.x. The upgrade from 8.x to 9.x did not cause any issues.

Longhorn

Longhorn was time-consuming since each version upgrade was preceded by an offsite backup of all volumes. I upgraded from version 1.8.x to 1.10.x. My Longhorn installation has a tendency to attach all volumes to a single worker node, which sometimes overloads the node and causes instability. It does eventually stabilize after each upgrade, but I haven’t found the root cause yet.

Prometheus

I temporarily lost some cluster metrics due to a missing port in a NetworkPolicy after upgrading the Prometheus chart from 27.x to 28.x (starting point was 26.x):

I crosschecked the traffic in Whisker and calico-flow-logs-otlphttp-exporter to identify the blocked port:

Updating the ports in the NetworkPolicy for traffic from Prometheus to the K3s VLAN fixed it:

--- a/prometheus/allow_egress_to_k3s_vlan_netpol.yaml
+++ b/prometheus/allow_egress_to_k3s_vlan_netpol.yaml
@@ -15,3 +15,4 @@ spec:
         ports:
           - 80
           - 7472
+          - 10250

Conclusion

Using Ansible to manage cluster configuration and ArgoCD to manage applications is one of the better investments I’ve made in the homelab. Most of my time spent upgrading is reading changelogs, taking backups of persistent data and updating config. I don’t really have to worry about versioning or how to apply changes.

Issue when customizing cloud-init images on Proxmox 9

Wed, 28 Jan 2026 00:00:00 +0000

I upgraded to Proxmox 9 in the homelab last year without any noticeable issues. It was only when I started the upgrade process of my Ansible playbook for the Proxmox hosts that I ran into problems. The role for creating cloud-init VM templates failed during the task which installs the qemu-guest-agent:

- name: Install qemu-guest-agent in all images in directory "{{ pvesm_local_storage_path }}/{{ pvesm_local_storage_iso_subpath }}"
  loop: "{{ iso_images_to_download }}"
  ansible.builtin.command: |
    virt-customize \
      -a "{{ pvesm_local_storage_path }}/{{ pvesm_local_storage_iso_subpath }}/{{ item.filename }}" \
      --install qemu-guest-agent

Error messages indicated DNS problems, similar to those in this forum post. There is already a GitHub issue on this in libguestfs. A suggested solution is to install dhcpcd-base prior to running virt-customize. I added it to the playbook:

- name: Install dhcpcd-base
  ansible.builtin.apt:
    name: dhcpcd-base

- name: Install qemu-guest-agent in all images in directory "{{ pvesm_local_storage_path }}/{{ pvesm_local_storage_iso_subpath }}"
  loop: "{{ iso_images_to_download }}"
  ansible.builtin.command: |
    virt-customize \
      -a "{{ pvesm_local_storage_path }}/{{ pvesm_local_storage_iso_subpath }}/{{ item.filename }}" \
      --install qemu-guest-agent

And then I had a working VM template again:

Resources:

Ingesting Calico flow logs into Loki in the homelab

Mon, 12 Jan 2026 00:00:00 +0000

I’ve written about Whisker previously in migrating from Flannel to Calico. While I like being able to debug network traffic in realtime, Whisker does not give me a historical overview:

What services has the most traffic measured by packets or bandwidth in the last 30 minutes?
What is the distribution of traffic by protocol over the last 30 minutes?

Version 3.30 of Calico OSS also introduced Goldmane, the gRPC-API powering Whisker. Goldmane uses Protobufs, which exposes a streaming endpoint to receive flow logs in realtime.

I use Loki to store logs for everything I run in my homelab, including firewall logs from OPNsense. It only makes sense to ingest flow logs into Loki as well. The protobuf payload from Goldmane contains information about the amount of bytes and packet count for each network flow. I can use this info with Metric Queries to turn the flow logs into metrics.

While researching the HTTP API for ingesting logs into Loki, I stumbled upon the documentation for the OTLP endpoint. Having no experience with OTLP, I went down the rabbit hole and learned about OTLP/HTTP, Direct to collector logging, Logs SDK and the log/slog Logging bridge. Using an open standard like OTLP/HTTP instead of targeting Loki specifically sounds like a much better idea.

This lead me to writing calico-flow-logs-otlphttp-exporter, which streams flow logs from Goldmane to anything thats compatible with OTLP/HTTP.

Note: calico-flow-logs-otlphttp-exporter is still in development, use at your own risk.

This is my setup to ingest flow logs from Goldmane into Loki in the homelab:

The OpenTelemetry Collector is not really necessary, its just there in case I want to add processors later.

graph TB
  goldmane[Goldmane]
  exporter[calico-flow-logs-otlphttp-exporter]
  otel-collector[OpenTelemetry Collector]
  loki[Loki]
  grafana[Grafana]
  exporter-->|Subscribe to flow logs streaming endpoint|goldmane
  goldmane-->|Stream flow logs|exporter
  exporter-->|Push logs using OTLP/HTTP|otel-collector
  otel-collector-->|Push logs using OTLP/HTTP|loki
  grafana-->|Consume Datasource|loki

Using Grafana to query flow logs ingested into Loki:

I also created a Grafana dashboard to see the distribution of traffic based on protocol/bandwidth over a specific time period:

I now have historical data for all network flows in the K3s cluster. The flow logs can be analyzed, visualized, and I retain all the data in my own infrastructure.

Upgrading the Proxmox cluster from 8 to 9 in the homelab

Tue, 11 Nov 2025 00:00:00 +0000

Proxmox 9 was released in August. I’ve focused the past few weeks on migrating from Flannel to Calico, and with the CNI-switch in K3s out of the way I was able to dedicate time to upgrade Proxmox.

Proxmox has a pretty nice guide for upgrading from 8 to 9. I opted for doing an in-place upgrade this time as opposed to reinstalling the entire OS. I did a mix of one-off commands and running a temporary Ansible playbook against each host.

The Proxmox cluster as it stands currently:

graph LR
  subgraph cluster[Datacenter: pve-cluster-1]
    pve2
    pve3
    pve4
  end
  subgraph pve2[Node: pve2]
    pve2_vms[VMs]
  end
  subgraph pve3[Node: pve3]
    pve3_vms[VMs]
  end
  subgraph pve4[Node: pve4]
    pve4_vms[VMs]
  end

Upgrades were done in the following order:

pve2 -> pve3 -> pve4

Upgrade process performed on each node:

Migrate all VMs to another node in the cluster
Upgrade the node to the latest 8.4.x version
Run pve8to9 --full and fix reported errors
Perform the upgrade from 8 to 9
Migrate VMs back to the upgraded node

Then the Terraform provider in all configs for Proxmox VMs was updated to the latest version.

Preparing for the upgrade

I had to fix this prior to starting (more info):

Removable bootloader found at '/boot/efi/EFI/BOOT/BOOTX64.efi', but GRUB packages not set up to update it!
Run the following command:

echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u

Then reinstall GRUB with 'apt install --reinstall grub-efi-amd64'

A reboot later it was fixed.

Performing the upgrade

Migrating VMs takes a long time for the K3s nodes dedicated to storage. Each of those VMs has a large disk reserved specifically for Longhorn:

Migrating a K3s worker VM running Longhorn before upgrading the underlying Proxmox host

I ran pve8to9 --full to identify and fix issues before starting the upgrade:

pve8to9 --full
...
= SUMMARY =

TOTAL:    48
PASSED:   39
SKIPPED:  5
WARNINGS: 1
FAILURES: 0

ATTENTION: Please check the output for detailed information!

The systemd-boot package had to be removed:

systemd-boot meta-package installed. This will cause problems on upgrades of other boot-related packages. Remove ‘systemd-boot’. See https://pve.proxmox.com/wiki/ Upgrade_from_8_to_9#sd-boot-warning for more information.

For nodes pve3 and pve4 I opted to remove the package using an Ansible role, but for pve2 (which was the initial host) I did it manually:

apt remove systemd-boot

Fixed all of the apt sources:

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list.d/*.list*

Cleaned up the Grafana apt repository sources used to install promtail for log collection:

wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee -a /etc/apt/sources.list.d/grafana.list
rm /etc/apt/sources.list.d/apt_grafana_com.list
rm /etc/apt/trusted.gpg.d/grafana.asc

Replaced privilege VM.Monitor with Sys.Audit in the Terraform provisioning user role since VM.Monitor is deprecated in Proxmox 9. Also switched to privileges listed in the bpg provider docs, even if they are a bit excessive:

roles/create-terraform-provisioning-user/vars/main.yaml:

diff --git a/roles/create-terraform-provisioning-user/vars/main.yaml b/roles/create-terraform-provisioning-user/vars/main.yaml
index 7a98193..8ca22be 100644
--- a/roles/create-terraform-provisioning-user/vars/main.yaml
+++ b/roles/create-terraform-provisioning-user/vars/main.yaml
@@ -2,4 +2,4 @@ terraform_user_token_name: proxmox-kubernetes-terraform-setup
 terraform_provider_role:
-terraform_user_token_role_privileges: "Datastore.AllocateSpace Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate VM.Monitor VM.PowerMgmt SDN.Use"
+terraform_user_token_role_privileges: "Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate VM.PowerMgmt VM.GuestAgent.Audit SDN.Use"

Made a temporary role to help prepare hosts for the upgrade:

roles/upgrade-pve8-to-pve9/tasks/main.yaml:

- name: Remove old systemd-boot package
  ansible.builtin.apt:
    name: systemd-boot
    state: absent

# https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#LVM/LVM-thin_storage_has_guest_volumes_with_autoactivation_enabled
- name: Fix LVM/LVM-thin storage has guest volumes with autoactivation enabled
  ansible.builtin.command:
    cmd: /usr/share/pve-manager/migrations/pve-lvm-disable-autoactivation --assume-yes

The role for adding apt repositories was permanently changed and now also uses the new DEB822 source format:

roles/update-apt-repositories/tasks/main.yaml:

diff --git a/roles/update-apt-repositories/tasks/main.yaml b/roles/update-apt-repositories/tasks/main.yaml
index 8e0c240..8ce693c 100644
--- a/roles/update-apt-repositories/tasks/main.yaml
+++ b/roles/update-apt-repositories/tasks/main.yaml
@@ -1,30 +1,59 @@
 ---
-# https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories
-- name: Remove pve-enterprise repository from list
+- name: Add Debian base repositories
   block:
-  - apt_repository:
-      repo: deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise
-      state: absent
-      filename: /etc/apt/sources.list.d/pve-enterprise.list
-      update_cache: false
-  - apt_repository:
-      repo: deb https://enterprise.proxmox.com/debian/ceph-quincy bookworm enterprise
-      state: absent
-      filename: /etc/apt/sources.list.d/ceph.list
-      update_cache: false
+  - ansible.builtin.deb822_repository:
+      enabled: true
+      name: debian
+      types:
+      - deb
+      - deb-src
+      uris: http://deb.debian.org/debian/
+      suites:
+      - trixie
+      - trixie-updates
+      components:
+      - main
+      - non-free-firmware
+      signed_by: /usr/share/keyrings/debian-archive-keyring.gpg
+  - ansible.builtin.deb822_repository:
+      enabled: true
+      name: debian-security
+      types:
+      - deb
+      - deb-src
+      uris: http://security.debian.org/debian-security/
+      suites:
+      - trixie-security
+      components:
+      - main
+      - non-free-firmware
+      signed_by: /usr/share/keyrings/debian-archive-keyring.gpg
 
-- name: Add pve-no-subscription repository to list
-  block:
-  - apt_repository:
-      repo: deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
-      state: present
-      filename: /etc/apt/sources.list.d/pve-enterprise.list
-      update_cache: false
-  - apt_repository:
-      repo: deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
-      state: present
-      filename: /etc/apt/sources.list.d/ceph.list
-      update_cache: false
+- name: Add Proxmox no-subscription repository
+  ansible.builtin.deb822_repository:
+    enabled: true
+    name: proxmox
+    types:
+    - deb
+    uris: http://download.proxmox.com/debian/pve
+    suites:
+    - trixie
+    components:
+    - pve-no-subscription
+    signed_by: /usr/share/keyrings/proxmox-archive-keyring.gpg
+
+- name: Add Ceph repositories
+  ansible.builtin.deb822_repository:
+    enabled: true
+    name: ceph
+    types:
+    - deb
+    uris: http://download.proxmox.com/debian/ceph-squid
+    suites:
+    - trixie
+    components:
+    - no-subscription
+    signed_by: /usr/share/keyrings/proxmox-archive-keyring.gpg
...

Then I did the actual upgrade manually:

apt dist-upgrade

Upgrading pve3 in this case

After rebooting I fixed the old apt sources:

apt modernize-sources
The following files need modernizing:
  - /etc/apt/sources.list
  - /etc/apt/sources.list.d/grafana.list

Modernizing will replace .list files with the new .sources format,
add Signed-By values where they can be determined automatically,
and save the old files into .list.bak files.

This command supports the 'signed-by' and 'trusted' options. If you
have specified other options inside [] brackets, please transfer them
manually to the output files; see sources.list(5) for a mapping.

For a simulation, respond N in the following prompt.
Rewrite 2 sources? [Y/n] Y
Modernizing /etc/apt/sources.list...
- Writing /etc/apt/sources.list.d/debian.sources

Modernizing /etc/apt/sources.list.d/grafana.list...
- Writing /etc/apt/sources.list.d/grafana.sources

Reran the playbook to remove the enterprise repo file again and verified apt worked after the changes:

apt update
Hit:1 http://deb.debian.org/debian trixie InRelease
Hit:2 http://deb.debian.org/debian trixie-updates InRelease
Hit:3 http://security.debian.org/debian-security trixie-security InRelease
Hit:4 https://apt.grafana.com stable InRelease
Hit:5 http://download.proxmox.com/debian/ceph-squid trixie InRelease
Hit:6 http://download.proxmox.com/debian/pve trixie InRelease
All packages are up to date.

I noticed “IO Pressure Stall” increasing drastically when migrating VMs to a node running version 9 from a node running version 8:

IO Pressure Stall while migrating VMs to pve2 running Proxmox 9

This was reflected in some of the VMs running on the affected Proxmox host, among them being k8s-control-5:

kubectl get nodes
NAME            STATUS                        ROLES                       AGE   VERSION
k8s-control-4   Ready                         control-plane,etcd,master   45h   v1.31.7+k3s1
k8s-control-5   NotReady                      control-plane,etcd,master   44h   v1.31.7+k3s1
k8s-control-6   Ready                         control-plane,etcd,master   44h   v1.31.7+k3s1
k8s-worker-1    Ready                         <none>                      44h   v1.31.7+k3s1
k8s-worker-2    NotReady,SchedulingDisabled   <none>                      44h   v1.31.7+k3s1
k8s-worker-3    Ready                         <none>                      44h   v1.31.7+k3s1
k8s-worker-4    Ready                         <none>                      44h   v1.31.7+k3s1

There is a Reddit thread describing a similar issue even on fresh Proxmox 9 installations. The IO Pressure Stall has since been reduced. This is the month maximum for pve2:

The IO pressure stall maximum for pve2 last month

VMs have been stable after the migration, so I’m going to just keep monitoring for now.

Upgrading Terraform provider

I’ve changed Terraform provider for Proxmox VM configuration to the bpg provider over the last few years. Upgrading to the latest version of the provider (0.86.0 as of this writing) worked without any issues:

The VM configuration after having upgraded the provider

Conclusion

The tools and guides for preparing and performing in-place upgrades of Proxmox are quite good. With the exception of the IO Pressure Stall situation, everything went smoothly.

Resources:

Replacing Flannel with Calico as CNI in the K3s cluster

Sat, 25 Oct 2025 00:00:00 +0000

Locking down network traffic in the K3s cluster in the homelab has been in the backlog for some time. While not strictly required, I’d rather have a setup more secure by default.

Since K3s ships with Flannel (which does not support NetworkPolicies) as standard I have to switch CNI. After looking at multiple options, I ended up with Calico. Their documentation has a guide specifically for K3s. One feature in particular that caught my attention was Whisker, which is a UI for network flow logs offered in version v3.30 of Calico.

The plan:

Modify the existing Ansible playbook for K3s to install Calico
Expose Whisker using an Ingress
Identify existing network flows to target with NetworkPolicies
Add NetworkPolicies to the ArgoCD configuration

Modifying the Ansible playbook

New additions to the main inventory file:

inventory.ini:

...
k3s_server_install_args="--disable servicelb --flannel-backend=none --disable-network-policy"
pod_cidr=10.42.0.0/16

The pod_cidr variable is now placed in the inventory since it is needed for the Calico configuration later.

The new role install-calico:

roles/install-calico/tasks/main.yaml:

- name: Install Calico
  become: yes
  ansible.builtin.shell: |
    k3s kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/{{ calico_version }}/manifests/operator-crds.yaml
    k3s kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/{{ calico_version }}/manifests/tigera-operator.yaml

- name: Wait until Calico crds are installed
  become: yes
  ansible.builtin.shell: k3s kubectl api-resources | grep operator.tigera.io/v1
  register: result
  until: result.stdout.find("Installation") != -1
  retries: 5
  delay: 10

- name: Template out Calico custom-resources.yaml
  ansible.builtin.template:
    src: templates/calico-custom-resources.yaml.j2
    dest: "{{ calico_custom_resources_dest }}"

- name: Apply Calico custom-resources.yaml
  become: yes
  ansible.builtin.command: k3s kubectl apply -f {{ calico_custom_resources_dest }}

- name: Enable Felix metrics
  become: yes
  ansible.builtin.command: |
    k3s kubectl patch felixconfiguration default --type merge --patch '{"spec":{"prometheusMetricsEnabled": true}}'

roles/install-calico/templates/calico-custom-resources.yaml.j2:

# This section includes base Calico installation configuration.
# For more information, see: https://docs.tigera.io/calico/latest/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    bgp: Disabled
    ipPools:
    - name: default-ipv4-ippool
      blockSize: 26
      cidr: {{ pod_cidr }}
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled
      nodeSelector: all()
  typhaMetricsPort: 9093
---

# This section configures the Calico API server.
# For more information, see: https://docs.tigera.io/calico/latest/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}

---

# Configures the Calico Goldmane flow aggregator.
apiVersion: operator.tigera.io/v1
kind: Goldmane
metadata:
  name: default

---

# Configures the Calico Whisker observability UI.
apiVersion: operator.tigera.io/v1
kind: Whisker
metadata:
  name: default

roles/install-calico/vars/main.yaml:

calico_version: v3.30.2
calico_custom_resources_dest: /tmp/calico-custom-resources.yaml

Output after recreating the VMs of the nodes in Proxmox and reinstalling K3s using the new playbook:

kubectl get pods -n calico-system
NAME                                      READY   STATUS    RESTARTS     AGE
calico-kube-controllers-f979cd594-pdwl7   1/1     Running   0            58d
calico-node-grhjn                         1/1     Running   0            51d
calico-node-hgw28                         1/1     Running   2 (8d ago)   51d
calico-node-nhtwk                         1/1     Running   0            51d
calico-node-qxm97                         1/1     Running   1 (8d ago)   51d
calico-node-tgf2v                         1/1     Running   0            51d
calico-node-tzxgc                         1/1     Running   0            51d
calico-node-vwczl                         1/1     Running   0            51d
calico-typha-588c846999-8jn4f             1/1     Running   0            53d
calico-typha-588c846999-mkzjb             1/1     Running   0            53d
calico-typha-588c846999-sfpmm             1/1     Running   0            8d
csi-node-driver-9jnqc                     2/2     Running   0            58d
csi-node-driver-nwwhs                     2/2     Running   0            58d
csi-node-driver-r72m4                     2/2     Running   2 (8d ago)   58d
csi-node-driver-s6rf4                     2/2     Running   0            58d
csi-node-driver-sd2g8                     2/2     Running   0            58d
csi-node-driver-vvndc                     2/2     Running   0            58d
csi-node-driver-z62lp                     2/2     Running   0            58d
goldmane-64584db4cf-26bdg                 1/1     Running   0            58d
whisker-98dff96db-t78nq                   2/2     Running   0            58d

Exposing Whisker

With Calico up and running I added an Ingress and a NetworkPolicy to the ArgoCD configuration to expose Whisker:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-traefik-to-whisker
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          app.kubernetes.io/name: traefik
    ports:
    - port: 8081
      protocol: TCP
  podSelector:
    matchLabels:
      app.kubernetes.io/name: whisker
  policyTypes:
  - Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: step-issuer
    cert-manager.io/issuer-group: certmanager.step.sm
    cert-manager.io/issuer-kind: StepClusterIssuer
  name: whisker
spec:
  rules:
  - host: whisker.homelab.fredrickb.com
    http:
      paths:
      - backend:
          service:
            name: whisker
            port:
              number: 8081
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - whisker.homelab.fredrickb.com
    secretName: whisker-tls-cert

After applying changes with ArgoCD:

Whisker running in the homelab

Locking down network traffic

I repeated the following steps until all network traffic was covered by a NetworkPolicy:

Identify traffic for NetworkPolicies using Whisker
Label known IP-ranges outside of K3s using GlobalNetworkSets/NetworkSets
Dry-run NetworkPolicies before enforcing them using StagedGlobalNetworkPolicies/ StagedNetworkPolicies

I created GlobalNetworkSets for systems external to the cluster such as Proxmox and GitHub Actions self-hosted runner hosts:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: proxmox-hosts
  labels:
    name: proxmox-hosts
    node-exporter: "true"
    process-exporter: "true"
    promtail: "true"
spec:
  nets:
    - 10.0.2.0/24

apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: github-actions-runner-hosts
  labels:
    node-exporter: "true"
    process-exporter: "true"
    promtail: "true"
spec:
  nets:
    - 10.0.7.0/24

These GlobalNetworkSets were referenced in Calico NetworkPolicies using labels to allow traffic from Prometheus in the cluster to services such as node_exporter:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-egress-to-node-exporter-hosts
  namespace: prometheus
spec:
  types:
  - Egress
  egress:
  - action: Allow
    protocol: TCP
    destination:
      namespaceSelector: global()
      selector: node-exporter == "true"
      ports:
      - 9100

Names of GlobalNetworkSets are reflected in the column dest_name in Whisker when inspecting traffic from Prometheus running in K3s to Proxmox and GitHub Actions self-hosted runner hosts on port 9100:

Whisker showing Prometheus scraping node_exporter service on Proxmox and GitHub Actions self-hosted runner hosts

One thing I was hoping to avoid was the need to have a NetworkPolicy in every namespace to allow traffic between workloads. I did not find a way to cover this use-case with GlobalNetworkPolicies. I had to duplicate the following NetworkPolicy in all namespaces to achieve the correct behaviour:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: <namespace>
spec:
  types:
    - Egress
    - Ingress
  egress:
    - action: Allow
      source:
        namespaceSelector: kubernetes.io/metadata.name == "<namespace>"
      destination:
        namespaceSelector: kubernetes.io/metadata.name == "<namespace>"
  ingress:
    - action: Allow
      source:
        namespaceSelector: kubernetes.io/metadata.name == "<namespace>"
      destination:
        namespaceSelector: kubernetes.io/metadata.name == "<namespace>"

Example from the Grafana namespace:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: grafana
spec:
  types:
  - Egress
  - Ingress
  egress:
  - action: Allow
    source:
      namespaceSelector: kubernetes.io/metadata.name == "grafana"
    destination:
      namespaceSelector: kubernetes.io/metadata.name == "grafana"
  ingress:
  - action: Allow
    source:
      namespaceSelector: kubernetes.io/metadata.name == "grafana"
    destination:
      namespaceSelector: kubernetes.io/metadata.name == "grafana"

There are a couple of issues addressing this already, so we’ll see if this improves in the future:

4751
6107

Monitoring Calico

Calico ships with metrics that can be scraped using Prometheus (more details on monitoring here).

I added the necessary services for Felix and Typha to expose their metrics:

apiVersion: v1
kind: Service
metadata:
  name: felix-metrics-svc
  namespace: calico-system
spec:
  clusterIP: None
  selector:
    k8s-app: calico-node
  ports:
  - port: 9091
    targetPort: 9091

apiVersion: v1
kind: Service
metadata:
  name: typha-metrics-svc
  namespace: calico-system
spec:
  clusterIP: None
  selector:
    k8s-app: calico-typha
  ports:
  - port: 9093
    targetPort: 9093

The updated scrape config in Prometheus, copied from the documentation:

...
extraScrapeConfigs: |
  ...
  - job_name: 'felix_metrics'
    scrape_interval: 5s
    scheme: http
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: felix-metrics-svc
      replacement: $1
      action: keep

  - job_name: 'typha_metrics'
    scrape_interval: 5s
    scheme: http
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: typha-metrics-svc
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_name]
      regex: calico-typha
      action: drop

  - job_name: 'kube_controllers_metrics'
    scrape_interval: 5s
    scheme: http
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: calico-kube-controllers-metrics
      replacement: $1
      action: keep

The documentation offers some premade Grafana dashboards nested in a ConfigMap here, but they’ll need a small adjustment. I extracted the JSON payloads and replaced the hardcoded value of datasource.uid with $datasource:

From this:

...
"datasource": {
  "type": "prometheus",
  "uid": "<Some-UID>"
},
...

To this:

...
"datasource": {
  "type": "prometheus",
  "uid": "$datasource"
},
...

Once done, I had Grafana dashboards showing the status of components Felix and Typha:

Grafana dashboard for Felix in Calico

Grafana dashboard for Typha in Calico

Conclusion

All network traffic within the cluster and to systems external to the cluster is now locked down with NetworkPolicies. While I initially planned to stick with standard NetworkPolicies, the Calico NetworkPolicies provide more control and flexibility.

I really like the combination of Whisker, (Global)NetworkSets and Staged(Global)NetworkPolicies. You get tools to identify active network flows, label specific IP-ranges/hosts and dry-run NetworkPolicies before enforcing them. This helps a lot when retroactively locking down network traffic in a cluster that already has a lot of services running.

In terms of observability, there are Prometheus metrics and predefined Grafana dashboards available for Felix and Typha.

Network traffic in the cluster is now secure by default for current and future workloads.

Switching kanban board

Sun, 23 Mar 2025 00:00:00 +0000

I’ve used Trello for several years as my main kanban board for ideas and personal projects. While I like the tool, taking more control of my data and having deeper links between my notes and projects is more important going forward.

I started using Obsidian for note-taking in 2020. Everything is in a single directory, as files, interlinked, in git. It just works.

Obsidian has a large ecosystem of plugins available. The Obsidian Kanban Plugin is exactly what I need. It lets me create notes, represented as cards, in a board. I can then interlink these notes (cards) to any other note, such as my internal system documentation for the homelab.

Trello provides a way to export a board to JSON. I extracted the information I wanted from the JSON export using a Python script:

Lists
- Name
- Cards
Cards
- Title
- Description
- Checklists
- Comments

The information is converted to Markdown files matching the format of the Obsidian Kanban Plugin using the same script:

Each list is created as a heading in the markdown file for the board itself (<Boardname>.md)
Each card is created as a file (kanban/<board-name>/<card-name>.md) and added as a list item beneath the correct list heading in <Boardname>.md

An example card file (kanban/<board-name>/PiKVM.md):

---
title: PiKVM
created: 2022-11-26 - 12:46
board: <Boardname>
tags: [kanban, raspberry-pi]
---
Board: [[<Boardname>]]

> This card was imported from Trello: <Trello-Card-URL>

<Content>

# Comments

<date> <time>

<Comment 2>

---

<date> <time>

<Comment 1>

---

#kanban #raspberry-pi

I kept the URL to the Trello card in the event I want to go back and check the original.

Example of card list item in the list Ideas/Backlog in <Boardname>.md:

---

kanban-plugin: basic
...

---

## Ideas/Backlog

- [ ] [[PiKVM]]
...

The end result shown below.

From this:

To this:

I’m satisfied with the result. I get the same functionality I had before, and I own the data.

Making the K3s cluster control plane HA

Tue, 29 Oct 2024 00:00:00 +0000

I’ll admit, I haven’t been running the K3s control plane in HA. The architecture of 1 control plane node and 3 worker nodes is from a time when every node was running physically on Raspberry Pis. I then gradually migrated to VMs with no changes to the architecture.

There are a few reasons this is starting to become important:

The nodes are long overdue for an update from Ubuntu 20.04 to Ubuntu 24.04
In Ubuntu 20.04, the kernel version (5.15.x) can cause issues when upgrading to Longhorn version 1.6.x
Upgrading the cluster, while still mostly automated, is still a risk without HA and requires more planning
Live migration of the control plane node VM between Proxmox hosts causes unknown behaviour at times

The new Proxmox host (Proxmox host 3) ensures I now have the physical resources required.

So here’s the plan:

IP address assignments:

IP address	Host
`10.0.3.100`	`Load Balancer frontend VIP`
`10.0.3.2`	`k8s-control-1`
`10.0.3.7`	`k8s-control-2`
`10.0.3.8`	`k8s-control-3`

Setting up the HAProxy plugin in OPNsense as the load balancer

I could spin up 2 VMs to create a HAProxy load balancer setup with failover like in the official documentation. I can also use the OPNsense HAProxy plugin.

While I prefer IaC and configuration in code, I opted for using OPNsense. The physical host of OPNsense is currently underutilized, and it would be nice to get more capabilities out of it.

I won’t go into the specifics of the HAProxy plugin itself. See the official documentation for more information. Some of the screenshots were taken after the setup was confirmed working so the titles sometimes display “Edit” as opposed to “New”.

Installing the plugin was fairly straightforward:

Once installed, the Services list is updated:

The VIP is defined as an IP Alias. I added this to the existing VLAN interface of the K3s cluster:

The VIP was added to a firewall alias K3s_Control_Plane_Loadbalancer:

A new firewall rule was created on the K3s interface to allow traffic to the VIP from the worker nodes:

What I’m doing from this point on is more or less copying the HAProxy configuration from the documentation into OPNsense. I also used information from this forum post.

Adding the existing control plane host as a server:

Creating the health monitor:

Creating the backend pool:

Creating the public service (frontend):

Statistics page:

The Config Export:

# Frontend: k3s-controlplane (K3s controlplane hosts)
frontend k3s-controlplane
    bind 10.0.3.100:6443 name 10.0.3.100:6443 
    mode tcp
    default_backend k3s-controlplane

    # logging options
    option tcplog

# Backend: k3s-controlplane (Pool serving all K3s controlplane hosts)
backend k3s-controlplane
    option log-health-checks
    # health check: k3s-controlplane
    mode tcp
    balance roundrobin

    server k8s-control-1 10.0.3.2:6443 check inter 10s

This is everything that is required until the cluster is expanded with more control plane nodes. I’ll do that in the next section.

Changing the existing single-node cluster to HA with embedded etcd

The documentation on this is pretty good. The existing control plane node (k8s-control-1) will be the node bootstrapping the cluster.

Inventory of the Ansible playbook:

[all:vars]
...
# VIP
k3s_control_plane_vip=10.0.3.100

# There will only ever be one host designated to
# bootstrap the cluster
[initial_k3s_server]
10.0.3.2 node_name=k8s-control-1

[remaining_k3s_servers]
10.0.3.7 node_name=k8s-control-2
10.0.3.8 node_name=k8s-control-3

I added a new role: k3s-initial-server, with the following tasks:

- name: Check if k3s is already installed
  ansible.builtin.stat:
    path: /usr/local/bin/k3s
  register: k3s

- name: Bootstrap k3s cluster with etcd
  become: yes
  shell: "{{ k3s_install_script_dest }}"
  environment:
    INSTALL_K3S_CHANNEL: "{{ k3s_channel }}"
    INSTALL_K3S_VERSION: "{{ k3s_version }}"
    # I extracted the existing token from /var/lib/rancher/k3s/server/token
    # and set it as a variable to be passed into the Ansible playbook
    K3S_TOKEN: "{{ k3s_token }}"
    K3S_NODE_NAME: "{{ node_name }}"
    # Set the --tls-san to the VIP
    INSTALL_K3S_EXEC: "--disable servicelb --tls-san {{ k3s_control_plane_vip }} --cluster-init"
  when: (reinstall | bool) or not k3s.stat.exists

The role is delegated to the group initial_k3s_server:

- name: Setup initial k3s server
  hosts: initial_k3s_server
  roles:
  - { role: k3s-initial-server, tags: [k3s-baseline] }

Running the Ansible playbook with -e reinstall=true changes the existing control plane installation from SQLite to etcd:

NAME            STATUS                     ROLES                       AGE     VERSION
k8s-control-1   Ready                      control-plane,etcd,master   296d    v1.24.17+k3s1

At this stage I have only reinstalled k8s-control-1, none of the worker nodes were reinstalled until after I confirmed that the HAProxy setup was working.

The configuration file /etc/rancher/k3s/config.yaml now has the new VIP in it:

k8s-control-1:~$ cat /etc/rancher/k3s/config.yaml
token: X
tls-san:
- 10.0.3.100

This was not enough to update the existing certificates. Simply downloading the new kubeconfig from k8s-control-1 and changing the IP address to the VIP produced this:

$ kubectl config view
...
clusters:
- cluster:
    ...
    server: https://10.0.3.100:6443
...

$ kubectl get pods -A
Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for 10.0.3.2, 10.43.0.1, 127.0.0.1, ::1, not 10.0.3.100

I had to delete the old certificates and generate new ones. The solution comes from this comment:

On k8s-control-1:

k8s-control-1:~$ sudo k3s kubectl -n kube-system delete secrets/k3s-serving
k8s-control-1:~$ sudo mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json
k8s-control-1:~$ sudo systemctl restart k3s
# After verifying that it works
k8s-control-1:~$ rm /tmp/dynamic-cert.json

After downloading the kubeconfig from k8s-control-1 again and changing the IP address to the VIP, I was able to access the control plane over TLS:

$ kubectl get nodes
NAME            STATUS   ROLES                       AGE    VERSION
k8s-control-1   Ready    control-plane,etcd,master   293d   v1.24.17+k3s1
...

The new control plane nodes are initialized using the old role k3s-servers with some changes:

- name: Check if k3s is already installed
  ansible.builtin.stat:
    path: /usr/local/bin/k3s
  register: k3s

- name: Install k3s on server
  become: yes
  shell: "{{ k3s_install_script_dest }}"
  environment:
    INSTALL_K3S_CHANNEL: "{{ k3s_channel }}"
    INSTALL_K3S_VERSION: "{{ k3s_version }}"
    K3S_TOKEN: "{{ k3s_token }}"
    # There will only ever be one host in this group
    K3S_URL: "https://{{ groups['initial_k3s_server'] | first }}:6443"
    K3S_NODE_NAME: "{{ node_name }}"
    # This needs to include `server` first so it does not get registered as an agent. Set the --tls-san to the VIP
    INSTALL_K3S_EXEC: "server --disable servicelb --tls-san {{ k3s_control_plane_vip }}"
  when: (reinstall | bool) or not k3s.stat.exists

The role is delegated to the group remaining_k3s_servers:

- name: Setup remaining k3s servers
  hosts: remaining_k3s_servers
  roles:
  - { role: k3s-servers, tags: [k3s-baseline] }

This installs K3s and joins the remaining control plane nodes to the cluster:

$ kubectl get nodes
NAME            STATUS   ROLES                       AGE     VERSION
k8s-control-1   Ready    control-plane,etcd,master   297d    v1.24.17+k3s1
k8s-control-2   Ready    control-plane,etcd,master   40h     v1.24.17+k3s1
k8s-control-3   Ready    control-plane,etcd,master   40h     v1.24.17+k3s1
...

The new control plane nodes were then added to the setup in HAProxy.

Creating the hosts:

Updating the backend pool:

Statistics page:

The Config Export:

# Frontend: k3s-controlplane (K3s controlplane hosts)
frontend k3s-controlplane
    bind 10.0.3.100:6443 name 10.0.3.100:6443 
    mode tcp
    default_backend k3s-controlplane

    # logging options
    option tcplog

# Backend: k3s-controlplane (Pool serving all K3s controlplane hosts)
backend k3s-controlplane
    option log-health-checks
    # health check: k3s-controlplane
    mode tcp
    balance roundrobin

    server k8s-control-1 10.0.3.2:6443 check inter 10s 
    server k8s-control-2 10.0.3.7:6443 check inter 10s 
    server k8s-control-3 10.0.3.8:6443 check inter 10s

With the HAProxy setup confirmed working, I went on to reinstall the workers and join them to the cluster using the new VIP. Role k3s-agents:

- name: Check if k3s is already installed
  ansible.builtin.stat:
    path: /usr/local/bin/k3s
  register: k3s

- name: Install k3s on agent
  become: yes
  shell: "{{ k3s_install_script_dest }}"
  environment:
    INSTALL_K3S_CHANNEL: "{{ k3s_channel }}"
    INSTALL_K3S_VERSION: "{{ k3s_version }}"
    K3S_TOKEN: "{{ k3s_token }}"
    # The VIP is used instead of k8s-control-1 IP address
    K3S_URL: "https://{{ k3s_control_plane_vip }}:6443"
    K3S_NODE_NAME: "{{ node_name }}"
  when: (reinstall | bool) or not k3s.stat.exists

/etc/systemd/system/k3s-agent.service.env after reinstalling the workers:

...
K3S_URL='https://10.0.3.100:6443'

I was expecting a reinstall with a new K3S_URL to break the existing setup in some way, but it didn’t. Everything worked and none of the workloads seemed to have been impacted. This made me suspicious enough that I had to verify the new setup.

I ended up:

Draining and shutting down k8s-cluster-1
Performing operations that would trigger traffic in the cluster, such as deleting pods
Observing that workers still functioned as normal

I considered the setup verified after this.

Enabling the HAProxy plugin Prometheus exporter

I enabled the Prometheus exporter to gather metrics from HAProxy:

The Config Export:

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }

Updating the Prometheus scrape config:

- job_name: 'k3s-controlplane-haproxy'
  scrape_interval: 15s
  static_configs:
    - targets:
        - '10.0.3.100:8404'

I imported the HAProxy 2 Grafana dashboard. Displaying metrics from the k3s-controlplane services:

The amount of traffic passing through HAProxy seemed very low compared to that I was expecting:

I could not get this to compute. There is enough traffic between the worker nodes and the control plane that no traffic for such a long period of time means something isn’t right. I verified earlier that traffic was not exclusively being sent to k8s-control-1 after the reinstall.

I checked the logs of k8s-worker-1 and discovered this:

Oct 27 08:45:36 k8s-worker-1 k3s[2406293]: time="2024-10-27T08:45:36Z" level=info msg="Removing server from load balancer k3s-agent-load-balancer: 10.0.3.2:6443"

That seemed strange, the worker nodes aren’t configured with any knowledge of the setup in HAProxy.

Turns out that K3s agents don’t actually use the Fixed Registration Address (the VIP in this case) for anything beyond initial registration, or as a failover if all control plane nodes are down: https://docs.k3s.io/architecture#fixed-registration-address-for-agent-nodes

In the high-availability server configuration, each node can also register with the Kubernetes API by using a fixed registration address, as shown in the diagram below.

After registration, the agent nodes establish a connection directly to one of the server nodes.

Which makes perfect sense as to why there is close to no traffic going through the HAProxy frontend after the initial registration. There are some issues that clarify this:

And sure enough, a restart of the k3s-agent.service produced the following:

Oct 27 18:57:32 k8s-worker-1 k3s[327800]: time="2024-10-27T18:57:32Z" level=info msg="Starting k3s agent v1.24.17+k3s1 (026bb0ec)"
Oct 27 18:57:32 k8s-worker-1 k3s[327800]: time="2024-10-27T18:57:32Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 10.0.3.100:6443"
Oct 27 18:57:32 k8s-worker-1 k3s[327800]: time="2024-10-27T18:57:32Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 10.0.3.7:6443"
Oct 27 18:57:32 k8s-worker-1 k3s[327800]: time="2024-10-27T18:57:32Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 10.0.3.8:6443"
Oct 27 18:57:32 k8s-worker-1 k3s[327800]: time="2024-10-27T18:57:32Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 10.0.3.2:6443"
Oct 27 18:57:32 k8s-worker-1 k3s[327800]: time="2024-10-27T18:57:32Z" level=info msg="Removing server from load balancer k3s-agent-load-balancer: 10.0.3.100:6443"
Oct 27 18:57:32 k8s-worker-1 k3s[327800]: time="2024-10-27T18:57:32Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [10.0.3.2:6443 10.0.3.7:6443 10.0.3.8:6443] [default: 10.0.3.100:6443]"

With that cleared up, I merged the changes to the playbook and called it a day.

Conclusion

One downside of the current setup is that I’m placing more responsibility in OPNsense by running HAProxy there. I don’t have a secondary instance so there is no failover if OPNsense goes down. In most cases I opt for using Terraform and Ansible, now I’m relying on UI configuration steps with offsite backups in the event of failure.

That being said, the certificates in the cluster are already configured with the VIP. If I ever change my mind and want to convert from OPNsense to something else, as long as whatever load balancer I end up using serves the frontend over the same IP address I should be fine.

It’s great to finally have proper HA setup for the control plane.

Using Terraform, Ansible and GitHub Actions to automate provisioning and configuration of workloads in the homelab

Mon, 14 Oct 2024 00:00:00 +0000

Around six months ago I converted all of the internal cluster workloads in k3s from manually installed Helm releases to ArgoCD applications. This has made upgrading everything within the cluster a breeze. I’ve thought about doing the same for the VMs used as cluster nodes since then. Now seemed like a good time.

I already use Terraform for provisioning VMs in Proxmox, with Ansible for installing and configuring the cluster. All of this is ran from a local machine. The task is therefore to automate these steps from another machine using pipelines.

In a perfect scenario I would have some tool with access to the Proxmox API which could spawn clusters on-demand and do all of the lifecycle management. There are a couple of options for other hypervisors, but I haven’t found anything for Proxmox at the time of this writing.

Enter GitHub Actions and self-hosted runners.

Scope

Since I’m using repository runners, spinning up new runners has to be an easy process that scales without the need for manual steps.

The criteria I have:

Provisioning and configuring runners should be done using IaC and workflows
The initial runner is the only runner which should be provisioned and configured manually
A runner connected to a repository should be able to do the following in a workflow:
- Use Terraform to provision new VMs in Proxmox
- Use Ansible to configure VMs

Architecture

High-level

graph TB
    subgraph github_actions_runner_spawner_vm[VM: GitHub Actions Runner Spawner]
        github_actions_runner_spawner[Runner: GitHub Actions Runner Spawner]
        repo_1_n_runner[Container: Repository 1..N Runner]
    end

    subgraph github[GitHub.com]
        github_actions_self_hosted_runners[GitHub Actions Self Hosted Runners Repository]
        repo_1_n[Repository 1..N]
    end

    github_actions_runner_spawner-->|Connects to repository|github_actions_self_hosted_runners
    github_actions_runner_spawner-->|Spawns|repo_1_n_runner
    repo_1_n_runner-->|Connects to repository|repo_1_n

    tf_runner_repo[GitHub Actions Runner Spawner Terraform Repository]-->|Manually provisions|github_actions_runner_spawner_vm
    ansible_runner_repo[GitHub Actions Runner Spawner Ansible Repository]-->|Manually configures|github_actions_runner_spawner_vm

In simple terms: GitHub Actions Runner Spawner is just a runner installed directly on a VM to spawn other runners as containers on the same host. The runners in containers are connected to other repositories holding either Terraform config or Ansible playbooks.

Network

graph TB
    gha_vlan[VLAN - GitHub Actions Runners - 10.0.7.0/24]
    gha_vlan-->|Port 443|proxmox_vlan[VLAN - Proxmox - 10.0.2.0/24]
    gha_vlan-->|Port 22|k3s_vlan[VLAN - k3s - 10.0.3.0/24]

The firewall rules gives access to other VLANs, more specifically the Proxmox API and SSH port of the k3s cluster nodes.

k3s provisioning

graph TB
    subgraph github_actions_runner_spawner_vm[VM: GitHub Actions Runner Spawner]
        k3s_tf_repo_runner[Container: k3s Terraform Repository Runner]
        k3s_ansible_repo_runner[Container: k3s Ansible Repository Runner]
    end

    subgraph github[GitHub.com]
        tf_repo[k3s Terraform Repository]
        ansible_repo[k3s Ansible Repository]
    end

    subgraph proxmox_vlan[Proxmox VLAN]
        pve_hosts[Proxmox hosts]
    end

    subgraph k3s_vlan[k3s VLAN]
        k3s_hosts[k3s hosts]
    end

    tf_repo-->|Delegate workflow|k3s_tf_repo_runner
    ansible_repo-->|Delegate workflow|k3s_ansible_repo_runner
    k3s_tf_repo_runner-->|Access|pve_hosts
    k3s_ansible_repo_runner-->|Configure|k3s_hosts
    pve_hosts-->|Provision|k3s_hosts

Both k3s repositories have a runner connected to them.

Implementation

The repository GitHub Actions Self Hosted Runners Repository does the following:

Builds and uploads container images for runners to GitHub Packages
Spins up new runners as containers on GitHub Actions Runner Spawner using docker compose

The workflow for spinning up new runners:

.github/workflows/spawn_runners.yaml:

name: "Spawn runners"

on:
  push:
    branches: [main]
    paths: ['.github/workflows/spawn_runners.yaml', 'docker-compose.yaml']
  workflow_dispatch:

env:
  REGISTRY: ghcr.io

jobs:
  spawn_runners:
    # Delegate the job to the self hosted runner
    # running on VM `GitHub Actions Runner Spawner`
    runs-on: [self-hosted]
    name: Spawn runners
    steps:
      - uses: actions/checkout@v4
      # Login to the container image registry
      - name: Login to the image registry
        uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      # Spawn runners as containers
      - name: Spawn runners in containers
        env:
          # Fine-Grained Personal Access Token with
          # access to the repositories
          PAT: ${{ secrets.SELF_HOSTED_RUNNERS_HOMELAB_PAT }}
        run: docker compose up -d

docker-compose.yaml:

services:
  # Runner connected to the repository with the
  # k3s terraform config
  k3s_terraform_runner:
    container_name: k3s-terraform-runner
    image: ghcr.io/fredrickb/github-actions-self-hosted-runners:2.0.0
    restart: always
    environment:
      - URL=<URL of repository holding k3s terraform config>
      - NAME=k3s-terraform-runner
      - PAT=${PAT}
  # Runner connected to the repository with the
  # k3s ansible playbook
  k3s_ansible_runner:
    container_name: k3s-ansible-runner
    image: ghcr.io/fredrickb/github-actions-self-hosted-runners:2.0.0
    restart: always
    environment:
      - URL=<URL of repository holding k3s ansible playbook>
      - NAME=k3s-ansible-runner
      - PAT=${PAT}

Running the workflow:

The runners then register to the repositories:

Using the journald logging driver with Promtail, Loki and Grafana provides insight into the logs of the runners:

Adding additional runners boils down to:

Including more repositories in the Fine-Grained Personal Access Token
Extending docker-compose.yaml with more containers
Run the workflow

Everything is defined in code and automated.

Using runners for deploying k3s cluster

Setting up k3s can now be done using workflows.

Provision k3s nodes using Terraform and workflow

Snippet from workflow:

name: Terraform workflow

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  plan:
    name: Perform Terraform plan and (only branch `main`) apply
    runs-on: [self-hosted]
    steps:
    ...

Workflow assigned to runner:

Terraform plan being run:

Workflow summary:

Install and configure k3s cluster using Ansible and workflow

Snippet from workflow:

name: Run Ansible playbook

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  run_playbook:
    name: Run playbook
    runs-on: [self-hosted, ansible]
    steps:
    ...

Workflow assigned to runner:

Ansible playbook running:

Workflow summary:

Summary

I’m at the point where I can almost bootstrap the entire cluster using IaC and automation:

Order	Task	Method
1	Provision VMs using Terraform	Automated using workflow
2	Configure VMs using Ansible	Automated using workflow
3	Install ArgoCD	Manual step
4	Configure internal cluster workloads	Automated using ArgoCD
5	Restore PVCs from Longhorn offsite backups	Manual step

Overall I’m pleased with the results. Changing the cluster configuration can be done using pull requests and workflows. The setup also scales for future workloads.

Replacing all internal DNS records in the homelab

Mon, 15 Jul 2024 00:00:00 +0000

I’ve used .local as the TLD internally in the homelab since the beginning. This is not a good idea for several reasons, and I decided it was about time to remove it and start using my own domain for internal services.

Updating the step-ca config

I wanted to expose the step-ca service outside of the k3s cluster using a domain name different from the current step-ca.local. This requires changing the dnsNames property in the configuration file (this is the same as passing a new value to the --dns flag when bootstrapping the creation of a CA).

The resulting section dnsNames in ca.json then becomes:

{
  ...
  "dnsNames": [
    "step-certificates.step-ca.svc.cluster.local",
    "step-ca.homelab.fredrickb.com",
    "localhost"
  ],
  ...
}

I had step-ca.local as a temporary entry while migrating the services over to the new domain. The config above is the final result after completing the migration process.

Updating the OPNsense Unbound DNS config

Adding the new domains in the OPNsense Unbound DNS host overrides (kudos to Home Network Guy for the tutorial):

Updating the Proxmox config

In the Ansible playbook I use for my Proxmox hosts I changed the host used for the ACME account to the following:

# roles/add-acme-certificate/vars/main.yaml
step_ca_url: https://step-ca.homelab.fredrickb.com/acme/acme/directory

And then the domain used for the hosts themselves:

; inventory.ini
10.0.2.2 domain=pve2.homelab.fredrickb.com name=pve2
10.0.2.3 domain=pve3.homelab.fredrickb.com name=pve3

Ordering a certificate for the Proxmox hosts now uses the new domain name of the step-ca service:

Accessing one of the Proxmox hosts using a new domain with a valid cert:

Updating the Ingress config

In Kubernetes its just a matter of updating the domain in the Ingress and let the existing step-issuer provision new certificates.

Below is a snippet from the Grafana ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: step-issuer
    cert-manager.io/issuer-group: certmanager.step.sm
    cert-manager.io/issuer-kind: StepClusterIssuer
  labels:
    ...
  name: grafana
  namespace: grafana
spec:
  ...
  rules:
    - host: grafana.homelab.fredrickb.com
      ...
  tls:
    - hosts:
        - grafana.homelab.fredrickb.com
      secretName: grafana-tls-cert

Accessing Grafana using the new domain with a valid cert:

Conclusion

Every service in the homelab now uses *.homelab.fredrickb.com instead of *.local.

Using step-ca for certificate provisioning in the homelab

Sat, 06 Apr 2024 00:00:00 +0000

So what is step-ca? It is an open-source Certificate Authority (CA) that can be self-hosted. Step-ca supports provisioning TLS certificates using the ACME protocol. It pairs well with cert-manager using step-issuer, meaning you get certificate provisioning for services in Kubernetes fully automated.

Since any client supporting ACME is covered, Proxmox hosts can request certificates from step-ca if you expose it outside of the Kubernetes cluster.

In this post I’ll briefly go through some of my own setup, configuration and experience of using step-ca with Kubernetes and Proxmox.

Installation

Past this point, step-ca and step-certificates are interchangeable, step-certificates is the name of the server component.

Step-certificates and step-issuer can be installed into a Kubernetes cluster using their helm charts. I won’t cover the process of installing and configuring step-certificates and step-issuer in full, the documentation on GitHub already cover this. You use the step cli and the step ca command to generate the initial configuration for step-certificates.

This is probably the most time-consuming step of the process. Making the installation repeatable and converting it to IaC that can be deployed using something like ArgoCD is going to take some time. Read the docs thoroughly.

Past this point I’ll assume you have a functioning step-certificates installation in your Kubernetes cluster.

Certificate provisioning

Download the Root CA certificate from step-certificates after the installation:

kubectl get secrets -n <namespace> <step-certificates-certs-secret> -o jsonpath='{.data.root_ca\.crt}' > root_ca.crt

Keep this handy, we’ll use it in the next sections.

Kubernetes

This requires the chart smallstep/step-certificates to be installed.

Once everything is installed and configured correctly, you can define a StepClusterIssuer. In this case I have one predefined named step-issuer. This is an excerpt of the values.yaml I use with the step-issuer Helm chart:

stepClusterIssuer:
  create: true
  caUrl: https://<step-certificates-service-name>.<namespace>.svc.cluster.local
  # Base64 encoded root_ca.crt downloaded previously
  caBundle: <Base64 encoded root_ca.crt>
  
  provisioner:
    # To get these values, see: https://github.com/smallstep/step-issuer?tab=readme-ov-file#3-configure-step-issuer
    kid: "<kid>"
    name: "<mail>"
    passwordRef:
      # Defined in the step-certificates helm installation
      namespace: <namespace>
      name: <step-certificates-provisioner-password-secret-name>
      key: <password>

There is an official sample for a StepClusterIssuer here showing more realistic values.

With a StepClusterIssuer installed, you can now update the Ingress resources that are going to be exposed over HTTPS.

Below is an excerpt of the Ingress definition in values.yaml I use with the Grafana helm chart:

I don’t recommend using .local as a TLD, even internally. This is just what I used early on and has not been changed since.

ingress:
  enabled: true
  hosts:
  - grafana.local
  annotations:
    # Sets the issuer of the TLS certificate to be the step-issuer
    cert-manager.io/issuer-group: certmanager.step.sm
    cert-manager.io/issuer-kind: StepClusterIssuer
    cert-manager.io/issuer: step-issuer
  # Enable TLS for the given domain and provide the name of
  # the secret which will contain the certifcate
  tls:
  - hosts:
    - grafana.local
    secretName: grafana-tls-cert

Fetching the certificates.cert-manager.io for Grafana now provides the certificate issued by step-certificates:

kubectl get certificates -n grafana -o wide
NAME               READY   SECRET             ISSUER        STATUS                                          AGE
grafana-tls-cert   True    grafana-tls-cert   step-issuer   Certificate is up to date and has not expired   7d5h

Add the previously downloaded root_ca.crt to the CA certificates of the devices which are going to access your services using HTTPS.

If everything is correct you’ll see that the certificate is one provisioned by your own internal CA (in my case its named “Homelab”):

Proxmox

This requires the step-certificates Service to be of type LoadBalancer and given a domain name outside of Kubernetes. Exposing the service using an Ingress will not work when using ACME. This requires that you run something like MetalLB.

Proxmox also supports using ACME to request TLS certificates for its web UI and REST API (the pveproxy service). I’m using the HTTP-01 challenge plugin to automate this. The steps detailed in the guide for Let’s Encrypt works for step-certificates as well when using the 2) Custom option listed in the ACME examples guide.

Prerequisites

The step-certificates root CA certificate is loaded into the CA bundle of the Proxmox host
Port 80 is not in use on the Proxmox host and can be reached by the step-certificates deployment

Process visualized

This is how the certificate provisioning will occur:

sequenceDiagram
    pve-host->>step-certificates: Request certificate
    step-certificates->>pve-host: Return token as part of HTTP-01 challenge
    pve-host->>pve-host: Start web-server on port 80 and serve token
    pve-host->>step-certificates: HTTP-01 challenge ready to be verified
    step-certificates->>pve-host: Verify the HTTP-01 challenge
    step-certificates->>pve-host: Issue certificate
    pve-host->>pve-host: Load TLS certificate
    pve-host->>pve-host: Stop web-server on port 80
    pve-host->>pve-host: Restart pveproxy service

Ansible

These are snippets from the Ansible playbook I use for automating the process. Everything here is contained within a role.

Import the homelab root CA certificate:

# tasks/install_step_ca_root_certificate.yaml
- name: Copy CA root certificate into correct directory
  template:
    src: templates/step_ca_root.crt.j2
    dest: "{{ step_ca_root_certificate_path }}"

- name: Reload CA certificates
  command: update-ca-certificates

# tasks/setup_acme_account.yaml
- name: Check if ACME account already exists
  shell: pvenode acme account list | grep {{ step_ca_acme_name }}
  ignore_errors: true
  register: acme_exists_cmd

- name: Add ACME account
  command: pvenode acme account register {{ step_ca_acme_name }} {{ step_ca_contact_mail }} --directory {{ step_ca_url }}
  when: acme_exists_cmd.rc == 1

- name: Add ACME config
  command: pvenode config set --acme domains={{ domain }}

- name: Order a certificate
  command: pvenode acme cert order --force

- name: Restart pveproxy service
  systemd:
    name: pveproxy
    state: restarted

Redirect incoming traffic to port 8006 from port 443 using the following iptables rule:

This is to avoid having to append port 8006 at the end of the URL. Will only work when the correct domain is used for accessing the Proxmox host.

# tasks/add_port_mapping_for_web_ui.yaml
- name: Add iptables-persistent package to ensure rules are persistent after reboot
  apt:
    name: iptables-persistent
    state: present

# This port mapping ensures that Proxmox UI is exposed
# on port 443 by port forwarding to 8006. When ACME certs
# are setup this means the URL no longer requires expicit
# port at the end.
- name: Add iptables port forwarding from port 443 to port 8006
  iptables:
    # Ensures that only traffic destined
    # for the domain of the pve node is
    # handled by this rule, otherwise all
    # traffic out on port 443 for the VMs
    # sharing the interface vmbr0 will be
    # affected and no traffic over TLS will
    # resolve to the correct CA bundle.
    destination: "{{ domain }}"
    table: nat
    chain: PREROUTING
    in_interface: vmbr0
    protocol: tcp
    match: tcp
    destination_port: '443'
    jump: REDIRECT
    to_ports: '8006'
    comment: Redirect secure web traffic to default web-ui of Proxmox

The vars used:

# vars/main.yaml
step_ca_root_certificate_path: /usr/local/share/ca-certificates/homelab_ca.crt
step_ca_url: https://<step-certificates-domain>/acme/acme/directory
step_ca_acme_name: <acme-account-name>
step_ca_contact_mail: <mail-address>
domain: <domain>

Tying it all together:

# tasks/main.yaml
- import_tasks: install_step_ca_root_certificate.yaml
- import_tasks: setup_acme_account.yaml
- import_tasks: add_port_mapping_for_web_ui.yaml

Proxmox hosts will now do periodic certificate renewals automatically:

Inspecting the certificate of the Proxmox host using the web ui now gives a valid TLS certificate with the same CA issuer as before (“Homelab”):

Summary

I’ve been running step-ca for around 1.5 years at this point. Its been great once I got past the initial bootstrapping process. The greatest benefit is that any internal service I run in the homelab which supports ACME can have certificate provisioning automated.

I recently converted the Helm installation of step-certificates to an ArgoCD app, and while it took some time, extracting the configuration of the existing installation and placing it into Sealed Secrets worked quite well since most of the configuration can be loaded from secrets (see this line in the step-certificates helm chart). Doing the same for step-issuer was fairly easy when comparing the two.

All things considered step-ca has been rock-solid for as long as I have ran it and I’m quite pleased with it.