So what is step-ca? It is an open-source Certificate Authority (CA) that can be self-hosted. Step-ca supports provisioning TLS certificates using the ACME protocol. It pairs well with cert-manager using step-issuer, meaning you get certificate provisioning for services in Kubernetes fully automated.

Since any client supporting ACME is covered, Proxmox hosts can request certificates from step-ca if you expose it outside of the Kubernetes cluster.

In this post I’ll briefly go through some of my own setup, configuration and experience of using step-ca with Kubernetes and Proxmox.

Installation

Past this point, step-ca and step-certificates are interchangeable, step-certificates is the name of the server component.

Step-certificates and step-issuer can be installed into a Kubernetes cluster using their helm charts. I won’t cover the process of installing and configuring step-certificates and step-issuer in full, the documentation on GitHub already cover this. You use the step cli and the step ca command to generate the initial configuration for step-certificates.

This is probably the most time-consuming step of the process. Making the installation repeatable and converting it to IaC that can be deployed using something like ArgoCD is going to take some time. Read the docs thoroughly.

Past this point I’ll assume you have a functioning step-certificates installation in your Kubernetes cluster.

Certificate provisioning

Download the Root CA certificate from step-certificates after the installation:

kubectl get secrets -n <namespace> <step-certificates-certs-secret> -o jsonpath='{.data.root_ca\.crt}' > root_ca.crt

Keep this handy, we’ll use it in the next sections.

Kubernetes

This requires the chart smallstep/step-certificates to be installed.

Once everything is installed and configured correctly, you can define a StepClusterIssuer. In this case I have one predefined named step-issuer. This is an excerpt of the values.yaml I use with the step-issuer Helm chart:

stepClusterIssuer:
  create: true
  caUrl: https://<step-certificates-service-name>.<namespace>.svc.cluster.local
  # Base64 encoded root_ca.crt downloaded previously
  caBundle: <Base64 encoded root_ca.crt>
  
  provisioner:
    # To get these values, see: https://github.com/smallstep/step-issuer?tab=readme-ov-file#3-configure-step-issuer
    kid: "<kid>"
    name: "<mail>"
    passwordRef:
      # Defined in the step-certificates helm installation
      namespace: <namespace>
      name: <step-certificates-provisioner-password-secret-name>
      key: <password>

There is an official sample for a StepClusterIssuer here showing more realistic values.

With a StepClusterIssuer installed, you can now update the Ingress resources that are going to be exposed over HTTPS.

Below is an excerpt of the Ingress definition in values.yaml I use with the Grafana helm chart:

I don’t recommend using .local as a TLD, even internally. This is just what I used early on and has not been changed since.

ingress:
  enabled: true
  hosts:
  - grafana.local
  annotations:
    # Sets the issuer of the TLS certificate to be the step-issuer
    cert-manager.io/issuer-group: certmanager.step.sm
    cert-manager.io/issuer-kind: StepClusterIssuer
    cert-manager.io/issuer: step-issuer
  # Enable TLS for the given domain and provide the name of
  # the secret which will contain the certifcate
  tls:
  - hosts:
    - grafana.local
    secretName: grafana-tls-cert

Fetching the certificates.cert-manager.io for Grafana now provides the certificate issued by step-certificates:

kubectl get certificates -n grafana -o wide
NAME               READY   SECRET             ISSUER        STATUS                                          AGE
grafana-tls-cert   True    grafana-tls-cert   step-issuer   Certificate is up to date and has not expired   7d5h

Add the previously downloaded root_ca.crt to the CA certificates of the devices which are going to access your services using HTTPS.

If everything is correct you’ll see that the certificate is one provisioned by your own internal CA (in my case its named “Homelab”):

Screenshot Grafana TLS certificate information

Proxmox

This requires the step-certificates Service to be of type LoadBalancer and given a domain name outside of Kubernetes. Exposing the service using an Ingress will not work when using ACME. This requires that you run something like MetalLB.

Proxmox also supports using ACME to request TLS certificates for its web UI and REST API (the pveproxy service). I’m using the HTTP-01 challenge plugin to automate this. The steps detailed in the guide for Let’s Encrypt works for step-certificates as well when using the 2) Custom option listed in the ACME examples guide.

Prerequisites

  • The step-certificates root CA certificate is loaded into the CA bundle of the Proxmox host
  • Port 80 is not in use on the Proxmox host and can be reached by the step-certificates deployment

Process visualized

This is how the certificate provisioning will occur:

sequenceDiagram
    pve-host->>step-certificates: Request certificate
    step-certificates->>pve-host: Return token as part of HTTP-01 challenge
    pve-host->>pve-host: Start web-server on port 80 and serve token
    pve-host->>step-certificates: HTTP-01 challenge ready to be verified
    step-certificates->>pve-host: Verify the HTTP-01 challenge
    step-certificates->>pve-host: Issue certificate
    pve-host->>pve-host: Load TLS certificate
    pve-host->>pve-host: Stop web-server on port 80
    pve-host->>pve-host: Restart pveproxy service

Ansible

These are snippets from the Ansible playbook I use for automating the process. Everything here is contained within a role.

Import the homelab root CA certificate:

# tasks/install_step_ca_root_certificate.yaml
- name: Copy CA root certificate into correct directory
  template:
    src: templates/step_ca_root.crt.j2
    dest: "{{ step_ca_root_certificate_path }}"

- name: Reload CA certificates
  command: update-ca-certificates

Register the ACME account:

# tasks/setup_acme_account.yaml
- name: Check if ACME account already exists
  shell: pvenode acme account list | grep {{ step_ca_acme_name }}
  ignore_errors: true
  register: acme_exists_cmd

- name: Add ACME account
  command: pvenode acme account register {{ step_ca_acme_name }} {{ step_ca_contact_mail }} --directory {{ step_ca_url }}
  when: acme_exists_cmd.rc == 1

- name: Add ACME config
  command: pvenode config set --acme domains={{ domain }}

- name: Order a certificate
  command: pvenode acme cert order --force

- name: Restart pveproxy service
  systemd:
    name: pveproxy
    state: restarted

Redirect incoming traffic to port 8006 from port 443 using the following iptables rule:

This is to avoid having to append port 8006 at the end of the URL. Will only work when the correct domain is used for accessing the Proxmox host.

# tasks/add_port_mapping_for_web_ui.yaml
- name: Add iptables-persistent package to ensure rules are persistent after reboot
  apt:
    name: iptables-persistent
    state: present

# This port mapping ensures that Proxmox UI is exposed
# on port 443 by port forwarding to 8006. When ACME certs
# are setup this means the URL no longer requires expicit
# port at the end.
- name: Add iptables port forwarding from port 443 to port 8006
  iptables:
    # Ensures that only traffic destined
    # for the domain of the pve node is
    # handled by this rule, otherwise all
    # traffic out on port 443 for the VMs
    # sharing the interface vmbr0 will be
    # affected and no traffic over TLS will
    # resolve to the correct CA bundle.
    destination: "{{ domain }}"
    table: nat
    chain: PREROUTING
    in_interface: vmbr0
    protocol: tcp
    match: tcp
    destination_port: '443'
    jump: REDIRECT
    to_ports: '8006'
    comment: Redirect secure web traffic to default web-ui of Proxmox

The vars used:

# vars/main.yaml
step_ca_root_certificate_path: /usr/local/share/ca-certificates/homelab_ca.crt
step_ca_url: https://<step-certificates-domain>/acme/acme/directory
step_ca_acme_name: <acme-account-name>
step_ca_contact_mail: <mail-address>
domain: <domain>

Tying it all together:

# tasks/main.yaml
- import_tasks: install_step_ca_root_certificate.yaml
- import_tasks: setup_acme_account.yaml
- import_tasks: add_port_mapping_for_web_ui.yaml

Proxmox hosts will now do periodic certificate renewals automatically:

Automated Proxmox certificate renewal item entry
Screenshot Proxmox certificate renewal details

Inspecting the certificate of the Proxmox host using the web ui now gives a valid TLS certificate with the same CA issuer as before (“Homelab”):

Screenshot Proxmox TLS certificate

Summary

I’ve been running step-ca for around 1.5 years at this point. Its been great once I got past the initial bootstrapping process. The greatest benefit is that any internal service I run in the homelab which supports ACME can have certificate provisioning automated.

I recently converted the Helm installation of step-certificates to an ArgoCD app, and while it took some time, extracting the configuration of the existing installation and placing it into Sealed Secrets worked quite well since most of the configuration can be loaded from secrets (see this line in the step-certificates helm chart). Doing the same for step-issuer was fairly easy when comparing the two.

All things considered step-ca has been rock-solid for as long as I have ran it and I’m quite pleased with it.

Updated: