CDK Infrastructure & GitHub Actions Runner

This repository contains the Infrastructure as Code (IaC) and custom runner implementation for the CDK (Cashu Development Kit) project (cashubtc/cdk).

It defines a highly specialized "warm pool" of ephemeral NixOS containers designed to run CI/CD workloads, specifically fuzzing and integration tests for the CDK ecosystem.

Overview

The primary purpose of this infrastructure is to provide reproducible, isolated, and high-performance runners for the CDK project. Unlike standard GitHub-hosted runners, these self-hosted runners:

• Are pre-provisioned ("warm pool") for instant job pickup.
• Run in fresh, ephemeral NixOS containers that are destroyed after every job.
• Support native Nix builds and binary caching through Cachix and Attic.
• Include Docker support for containerized actions.

Hosts

The infrastructure defines two dedicated runner hosts:

• cdk-runner-01: Primary runner host (assigned fuzz-a label).
• cdk-runner-02: Secondary GitHub runner host (assigned fuzz-b label).

The public demo configuration is:

• geata-demo: Geata traffic/payment demo at geata.thesimplekid.dev, on the former mint VPS 80.71.235.189. cdk-mint-01 is a deployment alias for this configuration and remains its enrolled SSH/logging hostname. See Geata deployment for the interactive demo. The old mint services are disabled in the Geata configuration; baile-mint is managed by the tsk-infra repository.

These hosts are configured via Nix Flakes and deployed using nixos-rebuild.

Architecture

The system is built on a custom Rust controller (runner-controller) that manages the lifecycle of NixOS containers.

1. Rust Controller:

   • Monitors the cashubtc/cdk repository for queued jobs.
   • Maintains a pool of idle containers (e.g., 7 per host).
   • Communicates with the GitHub API to register/deregister runners.
   • Exposes a local status endpoint.

2. Ephemeral Containers:

   • Created from a template (/etc/nixos/ci-container-template.nix).
   • Each container runs a single job and is then destroyed.
   • Network isolated (dedicated subnets).

Prerequisites

• NixOS: The target machines must run NixOS.
• Nix Flakes: Enabled in the configuration.
• GitHub PAT: A Personal Access Token with repo scope is required to register runners.
• Agenix: Used to encrypt the GitHub token.

Installation & Deployment

1. Secrets Management

Runner administration and Nix fetching use separate GitHub PATs managed via agenix. The administration PAT stays root-only on the host; containers receive a registration token and a separate restricted fetching credential. See credential setup and rollout.

To update or rotate the administration PAT:

1. Edit the secret file:

   agenix -e secrets/github-runner.age
   

2. Paste the new GitHub PAT.
3. Create the separate Nix fetching secret before building or deploying runners:

   just agenix-edit secrets/github-nix.age
   

   Use a fine-grained PAT with public read access, or Contents read access to required private repositories, and no Administration or write permissions.

2. Configuration

The runner configuration is defined in hosts/runner/runner-controller.nix. Key parameters include:

• githubRepo: Set to "cashubtc/cdk".
• maxConcurrentJobs: Set to 6 (adjust based on host resources).
• runnerLabels: Base labels include ["self-hosted", "ci", "nix", "x64", "Linux"].
  • Host-specific labels (e.g., fuzz-a) are injected in flake.nix.
• Cargo builds use at most three jobs per runner container. Nix runs at most two builds concurrently on each host and assigns four cores to each build.
• Inline Nix store optimisation is disabled on runner hosts. A weekly low-priority nix-optimise timer performs store deduplication outside the critical CI build path.

3. Deploying to Hosts

This project uses just to simplify deployment and management tasks.

Generic build/apply recipes publish successful builds to the TSK cache. See build caching for setup, retries, and the Vector test workaround.

Deploying updates

Runner deployments are built on the workstation running just, then copied to and activated on the target runner. After activation, the runner pushes its active NixOS system closure to the cashudevkit Attic cache at https://cache.cashudevkit.org. Store a restricted Attic publisher token once with Agenix before the first update:

just agenix-edit secrets/cachix-auth-token.age

Generate it from tsk-infra with just attic-publisher-token cashudevkit runner, then paste only the token into the editor. It is encrypted for both runner host keys and the operator keys, and is decrypted into /run/secrets at activation. After that one-time setup, apply the current configuration normally:

# Apply to runner 01
just apply-runner-01

# Apply to runner 02
just apply-runner-02

# Apply to all runners
just apply-all

If deployment succeeds but the upload is interrupted, retry only the cache push with:

just push-runner-cache root@cdk-runner-01.cashudevkit.org

CI workflows explicitly publish the cache-worthy paths they build. The host does not run an automatic post-build upload, avoiding duplicate store reads, compression, and uploads while jobs are active. Publisher tokens generated by tsk-infra expire after 30 days and must then be rotated in the Agenix secret and applied.

Bootstrapping new hosts

If setting up a fresh machine, use the bootstrap commands (requires root SSH access):

just bootstrap-runner-01
# or with a specific IP
just bootstrap-runner-01 IP=1.2.3.4

Operational Commands

Helper Scripts (On Host)

Helper scripts are available on the runner hosts for management:

Command	Description
runner-status	Displays the current pool size, active containers, and controller status.
cleanup-github-runners	Removes offline/stale runners from the GitHub UI.
cleanup-all-containers	Destructive: Stops and destroys all active containers and resets the pool.

Management via Justfile (Local)

The justfile provides convenient wrappers for common operational tasks:

Command	Description
just status-all	Checks systemd status of runners on all hosts via SSH.
just storage-all	Checks disk usage on all hosts (useful for monitoring Docker cleanup).
just push-runner-cache root@HOST	Pushes a runner's active system closure to the CDK Attic cache.
just set-token-runner-01 <TOKEN>	Updates the GitHub token on runner 01 and restarts the controller.
just agenix-edit	Edits the encrypted secrets file.
just check	Runs flake checks to validate configuration.

Monitoring

Each runner runs a loopback-only node exporter and Prometheus in agent mode. The agent scrapes host and runner-controller metrics every 30 seconds, buffers them locally, and sends them to https://metrics.cashudevkit.org/api/v1/write. Big Apple authenticates each runner independently before forwarding samples to the Prometheus instance on baile-monitoring; neither the exporter nor the controller metrics endpoint is publicly reachable.

The encrypted credentials are host-specific:

• secrets/metrics-remote-write-runner-01.age
• secrets/metrics-remote-write-runner-02.age

After deploying a runner, verify collection locally with:

curl --fail http://127.0.0.1:9100/metrics >/dev/null
curl --fail http://127.0.0.1:8080/metrics
systemctl status prometheus prometheus-node-exporter runner-controller

View logs for the main controller:

journalctl -u runner-controller -f

View logs for a specific runner container (e.g., 3a1f9-r0):

sudo journalctl -M 3a1f9-r0 -u github-runner -f

Development

The Rust controller source code is located in runner-controller/. To work on it:

1. Enter the dev environment:

   nix develop
   

2. Build and test:

   cargo build --package runner-controller
   cargo nextest run
   

Acknowledgments

This infrastructure and runner implementation is partially modeled after the Fedimint Infrastructure.

License

MIT

Central service logs

Vector forwards host journals from both runners and cdk-mint-01 to Baile's log store using authenticated HTTPS at https://metrics.cashudevkit.org/logs/loki/api/v1/push. Each host has a separate secrets/logs-<hostname>.age credential. This is independent of the existing Prometheus remote-write credentials. A 512 MiB local disk buffer absorbs short receiver outages; the log upload endpoint accepts no queries.

Deploy the receiving configuration in tsk-infra first, including Big Apple and the gateway, then apply these hosts normally. Check systemctl status vector and journalctl -u vector -n 50 locally. Logs appear in Grafana's Loki source; incidents appear at http://10.100.100.109:3102/ on the trusted network. See ../tsk-infra/docs/log-monitoring.md for sizing, rollout, acknowledgements, credential handling and verification.

The shared modules/fleet-logs.nix and modules/fleet-logs.vrl are intentionally copied into both repositories so each flake can build independently. Keep those files in sync when changing transport or redaction. Collection covers host services, including runner-controller exit summaries and mint daemons; it does not capture complete job transcripts inside ephemeral runner containers.