Lately I’ve been working a small side-project which is intended to be self-hosted and needs access to devices on the local network. I am a big fan of the Raspberry Pi, and already use a couple of 3B+ models to host various other things in docker containers.

I want to expand the usage of the Pis to include deployment of side-projects, using docker and deployment pipelines. Mainly I use Gitlab CI for CI/CD in side-projects. Deploying to one of the Pis requires a dedicated Gitlab runner installed on the Pi. The runner also needs access to docker. Like most of my other self-hosted applications, I want the runner deployed as a docker container rather than performing a native install. This makes the Pis more disposable in the event of SD-card failure or if I ever need to replace one of them (treating them like Immutable Infrastructure). Lastly I want the entire process to be automated so that it can be done without ever manually using SSH to log into the Pi.

The list of requirements for the runner then becomes:

  • Run as a container in docker
  • Have access to the docker socket
  • Fully automated provisioning, including
    • Dynamic register/unregister functionality
    • On-demand destruction/recreation

Creating the docker deployment

At the time of this writing, Gitlab does not offer an official docker image for it’s runners on ARM. There is an open issue that hopefully will fix this at some point in the future. Luckily there are a set of docker images provided on Docker Hub that can be used (shoutout to klud for providing these).

Note that this runner will have access to the docker socket on your Raspberry Pi, do not use this runner for public projects.

The final docker-compose.yml:

version: "3"
services: 
  registration:
    container_name: "${RUNNER_NAME}-registration"
    image: klud/gitlab-runner:12.8.0-alpine
    environment: 
      REGISTER_NON_INTERACTIVE: "true"
      RUNNER_EXECUTOR: "docker"
      DOCKER_IMAGE: "docker:latest"
      DOCKER_PRIVILEGED: "false"
      DOCKER_PULL_POLICY: "always"
      DOCKER_VOLUMES: "/var/run/docker.sock:/var/run/docker.sock"
      CI_SERVER_URL: "${GITLAB_URL}"
      REGISTRATION_TOKEN: "${GITLAB_REGISTRATION_TOKEN}"
      RUNNER_TAG_LIST: "self-hosted,rpi,docker"
      RUN_UNTAGGED: "false"
      RUNNER_NAME: "${RUNNER_NAME}"
      DESCRIPTION: "Self hosted gitlab runner on raspberry pi"
    volumes: 
      - config-volume:/etc/gitlab-runner
    entrypoint: /bin/sh
    command: -c "gitlab-runner register"

  gitlab_runner:
    container_name: "${RUNNER_NAME}"
    image: klud/gitlab-runner:12.8.0-alpine
    volumes: 
      - /var/run/docker.sock:/var/run/docker.sock
      - config-volume:/etc/gitlab-runner
    restart: unless-stopped
    depends_on: 
      - registration

volumes:
  config-volume:

When deploying this a temporary container is created which registers the runner, creating the configuration in a dedicated volume, then exits. The container for the actual runner then starts and uses the previously created volume and proceeds to connecting to the Gitlab instance.

Set DOCKER_PRIVILEGED to "true" if you plan on using docker-in-docker, otherwise you may experience issues talking to the docker daemon: documentation (see “Troubleshooting”).

Automating deployment with Ansible

To avoid manually deploying runners we use an Ansible playbook to automate the process. Note that this requires Python to be installed on the Pi, in addition to SSH-access and line of sight from the machine running Ansible.

Playbook:

---
- hosts: gitlab_runners
  become: yes
  vars:
    install_dir: "/tmp/gitlab-runner-{{ lookup('env','RUNNER_NAME') }}"
    runner_name: "{{ lookup('env','RUNNER_NAME') }}"
  tasks:
  - name: Create app directory
    file:
      path: "{{ install_dir }}"
      state: directory
      owner: root
      mode: 700

  - name: Copy docker-compose file to remote hosts
    copy:
      src: docker-compose.yml
      dest: "{{ install_dir }}"
      mode: 700
      owner: root

  - name: Find old runner container 
    command: docker ps --filter name=^/$RUNNER_NAME$ --format \{\{.ID\}\}
    environment:
      RUNNER_NAME: "{{ runner_name }}"
    register: runner_container_id

  - name: Unregister old runner from gitlab
    command: docker exec {{ runner_name }} gitlab-runner unregister --name {{ runner_name }}
    when: runner_container_id.stdout != ""

  - name: Take down old runner container
    command: docker-compose down --volumes
    args:
      chdir: "{{ install_dir }}"
    when: runner_container_id.stdout != ""

  - name: Spin up gitlab runner container
    command: docker-compose up -d
    args:
      chdir: "{{ install_dir }}"
    environment:
      GITLAB_URL: "{{ lookup('env','GITLAB_URL') }}"
      GITLAB_REGISTRATION_TOKEN: "{{ lookup('env','GITLAB_REGISTRATION_TOKEN') }}"
      RUNNER_NAME: "{{ runner_name }}"

  - name: Remove app directory
    file:
      path: "{{ install_dir }}"
      state: absent

Inventory:

[gitlab_runners]
<raspberry-pi-private-ip>

Running this playbook requires the following environment variables to be set on the machine running Ansible:

  • RUNNER_NAME - Your chosen name for the runner.
  • GITLAB_URL - The URL of the Gitlab instance, this is located on a given project under Settings -> CI/CD -> Specific Runners section.
  • GITLAB_REGISTRATION_TOKEN - The registration token for the given Gitlab project, this is located in the same section as the URL above.

Run the following to provision a new runner:

# Presumes the SSH user is the default user "pi"
ansible-playbook -i <my_inventory.ini> --user pi <playbook.yml>

Once done, the runner should be listed as a running container:

docker ps --filter name=<runner-name>
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS               NAMES
8dcbc8111626        klud/gitlab-runner:12.8.0-alpine   "/usr/bin/dumb-init …"   6 days ago          Up 6 days                               <runner-name>

And that’s it, you now have a fully automated way of provisioning self hosted gitlab runners.

Obviously this playbook presumes you only want a single project connected to each runner. It’s possible to split up the playbook to allow for registrations to multiple projects per runner. If you want to share a runner manually this can be done manually in the project settings.