thesimplekid/rith
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Preserve per-attempt stop reasons and diagnostic links for interrupted Firecracker jobs #13

New issue
Closed
opened 2026-10-23 13:44:58 +00:00 by thesimplekid · 0 comments
thesimplekid commented 2026-10-23 13:44:58 +00:00
Owner
Copy link

A long-running Fiach CI attempt stopped without enough information in Forgejo or the retained guest log to explain why. Debugging required manually correlating Forgejo API timestamps, controller journals, transient systemd units, and Firecracker directories. Please make that correlation and the terminal reason available directly in a durable per-attempt diagnostic record.

Observed attempt

Timeline (UTC):

  1. 2026-10-22 22:01:52: attempt starts; controller accepts task 17110 at 22:01:53.
  2. Checkout succeeds, formatting completes, and Clippy starts. Downloaded job output ends at 22:05:48 during Cargo dependency downloads, with no terminal diagnostic or exit status.
  3. 2026-10-23 01:08:33: Forgejo records failure, after 3h6m41s.
  4. 01:09:01–01:09:02: host stops the guest normally and the controller emits Cancelled { job_id: JobId("forgejo-17110"), task_id: "17110" }.

The guest serial log ends with {"protocol_version":2,"job_id":"forgejo-17110","event":"workload_ready"}. It contains no guest completion report or OOM evidence in the supplied tail. Host systemd accounting reports 11h42m25.636s CPU, 3h7m6.885s elapsed, 6.2G peak memory, 1.1T disk reads, and 22.5G writes. These are host-side VM accounting values, not measurements of the compiler or guest working set.

Forgejo's step metadata is inconsistent with delivered output: setup consumes the full duration, while checkout and subsequent steps all show failure at 0s, despite the log showing successful checkout and entry into Rust checks.

No newer Fiach run superseded this attempt when checked. Forgejo's default ENDLESS_TASK_TIMEOUT is 3h and periodic cleanup marks overlong tasks failed, which fits the timing. However, the deployed setting and server-side reason were not verified: timeout is a strong hypothesis, not a confirmed cause. Local Rith source also maps a remote terminal failure to a Cancelled poll outcome, so that outcome alone does not identify manual cancellation. Nothing here establishes an OOM or the cause of the large disk-read volume.

Requested diagnostics

  • Emit and retain the task-to-execution mapping at acceptance: repository/ref/commit, run/job/task/attempt identifiers where available, runner registration, controller host, selected template, runtime directory, systemd unit, and deployed Rith version.
  • Preserve separate fields for workload exit status/signal, guest lifecycle outcome, controller stop reason, and Forgejo terminal status. A remote failure should not be presented only as unexplained cancellation. Record the original remote response/status and timestamps; identify timeout or initiating actor only when the server provides evidence, otherwise explicitly mark unknown.
  • Write a host-side terminal diagnostic even if the guest never submits a completion report or Forgejo refuses further log updates. Preserve the last guest event, last local-output time, last successfully delivered log time, delivery errors/backlog or truncation, and pointers to retained logs. Do not require a successful remote upload to retain the explanation locally.
  • Surface a concise diagnostic and recovery location in the Forgejo attempt when possible. Preserve completed-step status and distinguish an interrupted active step from steps that never ran; avoid assigning the entire runtime to setup and marking every step failed at 0s.
  • Report prolonged log silence separately from execution inactivity. Reuse the resource measurements proposed in #9 to distinguish active computation, memory/I/O pressure, and a stalled guest; absence of output alone must not trigger a claim that the workload is hung.
  • Keep records bounded, redacted, and recoverable after teardown or controller restart. Document how to retrieve them using the Forgejo task/attempt ID without knowing a generated directory name.

Acceptance criteria

  1. A remote terminal failure, explicit cancellation, controller timeout, guest crash/OOM, and missing completion report each leave a durable record distinguishing observed facts from unknown causes.
  2. Tests cover a remote failure while the workload is still running, cancellation during log-upload failure, missing guest completion, and controller restart. Preserve already completed step outcomes and any available local diagnostics.
  3. An operator can identify the owning runner/guest and retrieve the stop reason and log-delivery state using the Forgejo task/attempt ID alone.
  4. If resource telemetry is unavailable, say so; never infer OOM from VMM peak memory or a timeout solely from elapsed time.

Related: #9 covers resource measurements and sizing; closed #3 covers log backpressure and step attribution; closed #5 covers missing remote logs and misleading step statuses. This report focuses on end-to-end failure attribution and diagnostic retention for an interrupted long-running attempt. It does not assert those earlier fixes regressed or that Rith caused the underlying slow build.

A long-running Fiach CI attempt stopped without enough information in Forgejo or the retained guest log to explain why. Debugging required manually correlating Forgejo API timestamps, controller journals, transient systemd units, and Firecracker directories. Please make that correlation and the terminal reason available directly in a durable per-attempt diagnostic record. ## Observed attempt - Run: https://git.cashu.dev/thesimplekid/fiach/actions/runs/19/jobs/0/attempt/2 - Downloadable log: https://git.cashu.dev/thesimplekid/fiach/actions/runs/19/jobs/0/attempt/2/logs - Commit: c8eb75298bfbc1498c8fd6a274f2ccab5a26efcc; workflow requests rith-nix. - Forgejo internal run ID 4485; job ID 28946; task/controller job forgejo-17110. - Controller host: baile-ci-runner-02; guest runtime forgejo-1711-4c8c9-6ada87d1-0; systemd unit rith-vm-dc545b0d.service. - Guest allocation: 6144 MiB RAM, 4 vCPUs. Deployed Rith revision was not established during this investigation; local source inspection is supporting context, not proof of the deployed code path. Timeline (UTC): 1. 2026-10-22 22:01:52: attempt starts; controller accepts task 17110 at 22:01:53. 2. Checkout succeeds, formatting completes, and Clippy starts. Downloaded job output ends at 22:05:48 during Cargo dependency downloads, with no terminal diagnostic or exit status. 3. 2026-10-23 01:08:33: Forgejo records failure, after 3h6m41s. 4. 01:09:01–01:09:02: host stops the guest normally and the controller emits `Cancelled { job_id: JobId("forgejo-17110"), task_id: "17110" }`. The guest serial log ends with `{"protocol_version":2,"job_id":"forgejo-17110","event":"workload_ready"}`. It contains no guest completion report or OOM evidence in the supplied tail. Host systemd accounting reports 11h42m25.636s CPU, 3h7m6.885s elapsed, 6.2G peak memory, 1.1T disk reads, and 22.5G writes. These are host-side VM accounting values, not measurements of the compiler or guest working set. Forgejo's step metadata is inconsistent with delivered output: setup consumes the full duration, while checkout and subsequent steps all show failure at 0s, despite the log showing successful checkout and entry into Rust checks. No newer Fiach run superseded this attempt when checked. Forgejo's default ENDLESS_TASK_TIMEOUT is 3h and periodic cleanup marks overlong tasks failed, which fits the timing. However, the deployed setting and server-side reason were not verified: timeout is a strong hypothesis, not a confirmed cause. Local Rith source also maps a remote terminal failure to a `Cancelled` poll outcome, so that outcome alone does not identify manual cancellation. Nothing here establishes an OOM or the cause of the large disk-read volume. ## Requested diagnostics - Emit and retain the task-to-execution mapping at acceptance: repository/ref/commit, run/job/task/attempt identifiers where available, runner registration, controller host, selected template, runtime directory, systemd unit, and deployed Rith version. - Preserve separate fields for workload exit status/signal, guest lifecycle outcome, controller stop reason, and Forgejo terminal status. A remote failure should not be presented only as unexplained cancellation. Record the original remote response/status and timestamps; identify timeout or initiating actor only when the server provides evidence, otherwise explicitly mark unknown. - Write a host-side terminal diagnostic even if the guest never submits a completion report or Forgejo refuses further log updates. Preserve the last guest event, last local-output time, last successfully delivered log time, delivery errors/backlog or truncation, and pointers to retained logs. Do not require a successful remote upload to retain the explanation locally. - Surface a concise diagnostic and recovery location in the Forgejo attempt when possible. Preserve completed-step status and distinguish an interrupted active step from steps that never ran; avoid assigning the entire runtime to setup and marking every step failed at 0s. - Report prolonged log silence separately from execution inactivity. Reuse the resource measurements proposed in #9 to distinguish active computation, memory/I/O pressure, and a stalled guest; absence of output alone must not trigger a claim that the workload is hung. - Keep records bounded, redacted, and recoverable after teardown or controller restart. Document how to retrieve them using the Forgejo task/attempt ID without knowing a generated directory name. ## Acceptance criteria 1. A remote terminal failure, explicit cancellation, controller timeout, guest crash/OOM, and missing completion report each leave a durable record distinguishing observed facts from unknown causes. 2. Tests cover a remote failure while the workload is still running, cancellation during log-upload failure, missing guest completion, and controller restart. Preserve already completed step outcomes and any available local diagnostics. 3. An operator can identify the owning runner/guest and retrieve the stop reason and log-delivery state using the Forgejo task/attempt ID alone. 4. If resource telemetry is unavailable, say so; never infer OOM from VMM peak memory or a timeout solely from elapsed time. Related: #9 covers resource measurements and sizing; closed #3 covers log backpressure and step attribution; closed #5 covers missing remote logs and misleading step statuses. This report focuses on end-to-end failure attribution and diagnostic retention for an interrupted long-running attempt. It does not assert those earlier fixes regressed or that Rith caused the underlying slow build.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
add_ci
shared_reg
metrics
recover_loss
disable_limits
paid-forgejo-accounts
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
thesimplekid/rith#13
No description provided.