CDK Infrastructure & GitHub Actions Runner

This repository contains the Infrastructure as Code (IaC) and custom runner implementation for the CDK (Cashu Development Kit) project (cashubtc/cdk).

It defines a highly specialized "warm pool" of ephemeral NixOS containers designed to run CI/CD workloads, specifically fuzzing and integration tests for the CDK ecosystem.

Overview

The primary purpose of this infrastructure is to provide reproducible, isolated, and high-performance runners for the CDK project. Unlike standard GitHub-hosted runners, these self-hosted runners:

• Are pre-provisioned ("warm pool") for instant job pickup.
• Run in fresh, ephemeral NixOS containers that are destroyed after every job.
• Support native Nix builds and binary caching through Cachix and Attic.
• Include Docker support for containerized actions.

Hosts

The infrastructure currently defines two dedicated runner hosts:

• cdk-runner-01: Primary runner host (assigned fuzz-a label).
• cdk-runner-02: Secondary GitHub runner host (assigned fuzz-b label) that also runs the Forgejo Actions pilot runner.

These hosts are configured via Nix Flakes and deployed using nixos-rebuild.

Architecture

The system is built on a custom Rust controller (runner-controller) that manages the lifecycle of NixOS containers.

1. Rust Controller:

   • Monitors the cashubtc/cdk repository for queued jobs.
   • Maintains a pool of idle containers (e.g., 7 per host).
   • Communicates with the GitHub API to register/deregister runners.
   • Exposes a local status endpoint.

2. Ephemeral Containers:

   • Created from a template (/etc/nixos/ci-container-template.nix).
   • Each container runs a single job and is then destroyed.
   • Network isolated (dedicated subnets).

cdk-runner-02 also runs the Forgejo migration pilot. In addition to the GitHub warm-pool controller, it registers a stock forgejo-runner instance against https://git.cashu.dev with:

• runner.capacity = 2
• nix:lxc://debian:bookworm:lxc docker
• self-hosted:lxc://debian:bookworm:lxc docker

The Forgejo label still uses Forgejo Runner's Debian LXC backend, but the NixOS host prepares a local forgejo-runner-nix-bookworm LXC template first. Per-job containers are cloned from that template so the normal job path starts with Nix, flakes, CDK binary-cache config, Node, Docker, and common CI tools already installed instead of apt-installing them from scratch for every job.

Forgejo jobs can target this runner with either label:

runs-on: nix

runs-on: self-hosted

Prerequisites

• NixOS: The target machines must run NixOS.
• Nix Flakes: Enabled in the configuration.
• GitHub PAT: A Personal Access Token with repo scope is required to register runners.
• Agenix: Used to encrypt the GitHub token.

Installation & Deployment

1. Secrets Management

The GitHub Runner token is managed via agenix. To update or rotate the token:

1. Edit the secret file:

   agenix -e secrets/github-runner.age
   

2. Paste the new GitHub PAT.

The Forgejo runner connection secret for cdk-runner-02 can be managed with:

agenix -e secrets/forgejo-runner.age

The encrypted file must contain the UUID and token displayed by Forgejo when creating the runner:

FORGEJO_RUNNER_UUID=<uuid>
FORGEJO_RUNNER_TOKEN=<token>

For a temporary pilot secret before the encrypted secret exists, use:

just set-forgejo-token-runner-02 <UUID> <TOKEN>

2. Configuration

The runner configuration is defined in hosts/runner/runner-controller.nix. Key parameters include:

• githubRepo: Set to "cashubtc/cdk".
• maxConcurrentJobs: Set to 6 (adjust based on host resources).
• runnerLabels: Base labels include ["self-hosted", "ci", "nix", "x64", "Linux"].
  • Host-specific labels (e.g., fuzz-a) are injected in flake.nix.

3. Deploying to Hosts

This project uses just to simplify deployment and management tasks.

Deploying updates

Runner deployments are built by the target runner itself. After activation, the runner pushes its active NixOS system closure to the cashudevkit Attic cache at https://cache.cashudevkit.org. Store a restricted Attic publisher token once with Agenix before the first update:

just agenix-edit secrets/cachix-auth-token.age

Generate it from tsk-infra with just attic-publisher-token cashudevkit runner, then paste only the token into the editor. It is encrypted for both runner host keys and the operator keys, and is decrypted into /run/secrets at activation. After that one-time setup, apply the current configuration normally:

# Apply to runner 01
just apply-runner-01

# Apply to runner 02
just apply-runner-02

# Apply to all runners
just apply-all

If deployment succeeds but the upload is interrupted, retry only the cache push with:

just push-runner-cache root@cdk-runner-01.cashudevkit.org

The runner's Nix daemon also uses an Attic post-build hook to upload new paths from subsequent builds automatically, so the token does not need to be exported or re-entered for each deployment. Publisher tokens generated by tsk-infra expire after 30 days and must then be rotated in the Agenix secret and applied.

Bootstrapping new hosts

If setting up a fresh machine, use the bootstrap commands (requires root SSH access):

just bootstrap-runner-01
# or with a specific IP
just bootstrap-runner-01 IP=1.2.3.4

Operational Commands

Helper Scripts (On Host)

Helper scripts are available on the runner hosts for management:

Command	Description
runner-status	Displays the current pool size, active containers, and controller status.
forgejo-runner-status	Displays the Forgejo runner service, LXC containers, and recent logs.
cleanup-github-runners	Removes offline/stale runners from the GitHub UI.
cleanup-all-containers	Destructive: Stops and destroys all active containers and resets the pool.

Management via Justfile (Local)

The justfile provides convenient wrappers for common operational tasks:

Command	Description
just status-all	Checks systemd status of runners on all hosts via SSH.
just storage-all	Checks disk usage on all hosts (useful for monitoring Docker cleanup).
just push-runner-cache root@HOST	Pushes a runner's active system closure to the CDK Attic cache.
just set-token-runner-01 <TOKEN>	Updates the GitHub token on runner 01 and restarts the controller.
just agenix-edit	Edits the encrypted secrets file.
just check	Runs flake checks to validate configuration.

Monitoring

Each runner runs a loopback-only node exporter and Prometheus in agent mode. The agent scrapes host and runner-controller metrics every 30 seconds, buffers them locally, and sends them to https://metrics.cashudevkit.org/api/v1/write. Big Apple authenticates each runner independently before forwarding samples to the Prometheus instance on baile-monitoring; neither the exporter nor the controller metrics endpoint is publicly reachable.

The encrypted credentials are host-specific:

• secrets/metrics-remote-write-runner-01.age
• secrets/metrics-remote-write-runner-02.age

After deploying a runner, verify collection locally with:

curl --fail http://127.0.0.1:9100/metrics >/dev/null
curl --fail http://127.0.0.1:8080/metrics
systemctl status prometheus prometheus-node-exporter runner-controller

View logs for the main controller:

journalctl -u runner-controller -f

View logs for a specific runner container (e.g., 3a1f9-r0):

sudo journalctl -M 3a1f9-r0 -u github-runner -f

Development

The Rust controller source code is located in runner-controller/. To work on it:

1. Enter the dev environment:

   nix develop
   

2. Build and test:

   cargo build --package runner-controller
   cargo nextest run
   

Acknowledgments

This infrastructure and runner implementation is partially modeled after the Fedimint Infrastructure.

License

MIT