thesimplekid/rith
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Expose per-job resource metrics to guide Firecracker sizing and concurrency #9

New issue
Closed
opened 2026-10-19 18:59:06 +00:00 by thesimplekid · 0 comments
thesimplekid commented 2026-10-19 18:59:06 +00:00
Owner
Copy link

Motivation

We run CDK Rust CI in disposable Firecracker guests and want to choose job sizing from measurements rather than guesswork. Each runner host currently has 14 vCPUs and 28 GiB RAM, admitting three jobs with 4 vCPUs and 8 GiB each. A possible alternative is four jobs with 3 vCPUs and 6 GiB each, retaining host headroom and the same 12-vCPU admission budget.

We need per-job measurements to compare total throughput, latency, memory headroom, and failures for representative builds, tests, Clippy, and formatting checks. Host-wide tools cannot reliably attribute these costs to individual jobs. This request is to expose a coherent resource summary, building on any existing Rith timing or accounting data.

Requested measurements

  • Guest workload memory: sampled peak usage with a documented definition (including how reclaimable page cache is treated), available-memory headroom, and OOM evidence when obtainable. Distinguish a sampled peak from an exact high-water mark.
  • Host Firecracker/VMM resource use: CPU time, peak RSS or cgroup memory, and OOM/limit events, clearly separated from guest workload memory. VMM RSS alone should not be presented as the guest workload's RAM requirement.
  • Disk and network byte counters, with the measurement boundary documented so host image staging, guest filesystem I/O, network traffic, and cache downloads are not conflated.
  • Phase durations where observable: queue wait, image preparation, VM boot/agent readiness, workload execution, and teardown. If dependency fetching can be attributed through workflow steps, expose that separately; otherwise do not infer it from aggregate network activity.
  • The configured template, vCPUs, RAM, I/O limits, and observed concurrency so comparisons retain their sizing context.

Reporting and lifecycle

Expose a per-job summary through the job API and a retained structured artifact; optionally include a concise terminal summary in Forgejo job logs. Live snapshots would help diagnose jobs still running. Include job identity and workload/ref metadata already available to the controller, without logging credentials or environment secrets.

Persist collected measurements through normal teardown, failures, and controller restarts. For guest crashes or missing final reports, preserve available host-side and last guest samples and explicitly mark partial/unavailable data rather than reporting zero usage. Bound sampling overhead and storage retention, and preserve VM isolation.

Acceptance criteria

  1. Representative Rust jobs produce attributable resource summaries that allow comparison of three 8-GiB jobs against four 6-GiB jobs on the same host.
  2. Concurrent jobs remain distinct, and an OOM, killed VM, or missing guest report produces a useful partial summary and explicit failure evidence when available.
  3. Tests cover metric attribution, counter/peak semantics, partial reports, persistence, and retention. Document the measurement limitations, especially guest memory versus host VMM memory and sampled peaks.

An initial implementation could prioritize guest memory, host CPU/memory, OOM reporting, and execution duration before adding detailed I/O and phase attribution.

## Motivation We run CDK Rust CI in disposable Firecracker guests and want to choose job sizing from measurements rather than guesswork. Each runner host currently has 14 vCPUs and 28 GiB RAM, admitting three jobs with 4 vCPUs and 8 GiB each. A possible alternative is four jobs with 3 vCPUs and 6 GiB each, retaining host headroom and the same 12-vCPU admission budget. We need per-job measurements to compare total throughput, latency, memory headroom, and failures for representative builds, tests, Clippy, and formatting checks. Host-wide tools cannot reliably attribute these costs to individual jobs. This request is to expose a coherent resource summary, building on any existing Rith timing or accounting data. ## Requested measurements - Guest workload memory: sampled peak usage with a documented definition (including how reclaimable page cache is treated), available-memory headroom, and OOM evidence when obtainable. Distinguish a sampled peak from an exact high-water mark. - Host Firecracker/VMM resource use: CPU time, peak RSS or cgroup memory, and OOM/limit events, clearly separated from guest workload memory. VMM RSS alone should not be presented as the guest workload's RAM requirement. - Disk and network byte counters, with the measurement boundary documented so host image staging, guest filesystem I/O, network traffic, and cache downloads are not conflated. - Phase durations where observable: queue wait, image preparation, VM boot/agent readiness, workload execution, and teardown. If dependency fetching can be attributed through workflow steps, expose that separately; otherwise do not infer it from aggregate network activity. - The configured template, vCPUs, RAM, I/O limits, and observed concurrency so comparisons retain their sizing context. ## Reporting and lifecycle Expose a per-job summary through the job API and a retained structured artifact; optionally include a concise terminal summary in Forgejo job logs. Live snapshots would help diagnose jobs still running. Include job identity and workload/ref metadata already available to the controller, without logging credentials or environment secrets. Persist collected measurements through normal teardown, failures, and controller restarts. For guest crashes or missing final reports, preserve available host-side and last guest samples and explicitly mark partial/unavailable data rather than reporting zero usage. Bound sampling overhead and storage retention, and preserve VM isolation. ## Acceptance criteria 1. Representative Rust jobs produce attributable resource summaries that allow comparison of three 8-GiB jobs against four 6-GiB jobs on the same host. 2. Concurrent jobs remain distinct, and an OOM, killed VM, or missing guest report produces a useful partial summary and explicit failure evidence when available. 3. Tests cover metric attribution, counter/peak semantics, partial reports, persistence, and retention. Document the measurement limitations, especially guest memory versus host VMM memory and sampled peaks. An initial implementation could prioritize guest memory, host CPU/memory, OOM reporting, and execution duration before adding detailed I/O and phase attribution.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
add_ci
shared_reg
metrics
recover_loss
disable_limits
paid-forgejo-accounts
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
thesimplekid/rith#9
No description provided.