thesimplekid/rith
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Forgejo log backpressure drops actionable failure diagnostics and loses step attribution #3

New issue
Closed
opened 2026-10-18 04:46:10 +00:00 by thesimplekid · 1 comment
thesimplekid commented 2026-10-18 04:46:10 +00:00
Owner
Copy link

A real Forgejo/Firecracker job failed its dependency advisory check, but the useful error detail was missing from the delivered logs and the user saw failed steps with no explanation.

Observed with Rith 0684459ea7, Forgejo 15.0.9, NixOS Firecracker guests, and three concurrent jobs (4 vCPUs / 8192 MiB per guest).

Reproduction evidence:

  • Run: https://git.cashu.dev/thesimplekid/satsettle/actions/runs/52
  • Failed job: https://git.cashu.dev/thesimplekid/satsettle/actions/runs/52/jobs/3/attempt/1
  • Controller job ID forgejo-16790; execution finished failure after 163 seconds.
  • Checkout succeeded. The command nix develop path:. -c just deny failed with exit status 1.
  • The remaining log contains advisories FAILED, bans ok, licenses ok, sources ok, followed by error: recipe deny failed on line 38 with exit code 1.
  • The downloadable Forgejo job log and controller result tail contain Rith truncated 1848 log records while delivery was backlogged.
  • Forgejo's downloadable log also ends with Rith: output was truncated; remaining logs are combined under the first step.
  • Neither retained artifact contains an error[...] advisory heading or a RUSTSEC ID. A yanked-crate warning survives, but that alone does not establish the reason advisories failed.
  • The job's step metadata reports checkout success, the deny command failure, and Complete job failure. The run has one failed job; the UI's two red steps can look like two unexplained failures.

Relevant code in src/ingress.rs drops log events when try_send returns Full and sets log_positions_lost; finalization then combines the remaining output under the first step. The completion log tail is only a bounded suffix, which is insufficient when Cargo Deny prints a long dependency tree after its actionable advisory header.

Please retain a complete bounded-on-disk diagnostic artifact (or equivalent durable spool) and preserve step attribution/completion diagnostics under normal verbose build output, without unbounded RAM or blocking guest execution indefinitely. At minimum surface the command, exit code, truncation count, and a clear link to recoverable logs on the failed step. Missing bytes must not be described as merely combined elsewhere.

No patched runtime has been tested. The underlying Cargo Deny failure is a project dependency issue; this report concerns loss of the information needed to diagnose it.

A real Forgejo/Firecracker job failed its dependency advisory check, but the useful error detail was missing from the delivered logs and the user saw failed steps with no explanation. Observed with Rith 0684459ea733f3da383a095e44ada23159b3717b, Forgejo 15.0.9, NixOS Firecracker guests, and three concurrent jobs (4 vCPUs / 8192 MiB per guest). Reproduction evidence: - Run: https://git.cashu.dev/thesimplekid/satsettle/actions/runs/52 - Failed job: https://git.cashu.dev/thesimplekid/satsettle/actions/runs/52/jobs/3/attempt/1 - Controller job ID forgejo-16790; execution finished failure after 163 seconds. - Checkout succeeded. The command `nix develop path:. -c just deny` failed with exit status 1. - The remaining log contains `advisories FAILED, bans ok, licenses ok, sources ok`, followed by `error: recipe deny failed on line 38 with exit code 1`. - The downloadable Forgejo job log and controller result tail contain `Rith truncated 1848 log records while delivery was backlogged`. - Forgejo's downloadable log also ends with `Rith: output was truncated; remaining logs are combined under the first step`. - Neither retained artifact contains an `error[...]` advisory heading or a RUSTSEC ID. A yanked-crate warning survives, but that alone does not establish the reason advisories failed. - The job's step metadata reports checkout success, the deny command failure, and Complete job failure. The run has one failed job; the UI's two red steps can look like two unexplained failures. Relevant code in src/ingress.rs drops log events when try_send returns Full and sets log_positions_lost; finalization then combines the remaining output under the first step. The completion log tail is only a bounded suffix, which is insufficient when Cargo Deny prints a long dependency tree after its actionable advisory header. Please retain a complete bounded-on-disk diagnostic artifact (or equivalent durable spool) and preserve step attribution/completion diagnostics under normal verbose build output, without unbounded RAM or blocking guest execution indefinitely. At minimum surface the command, exit code, truncation count, and a clear link to recoverable logs on the failed step. Missing bytes must not be described as merely combined elsewhere. No patched runtime has been tested. The underlying Cargo Deny failure is a project dependency issue; this report concerns loss of the information needed to diagnose it.
thesimplekid commented 2026-10-18 13:33:25 +00:00
Author
Owner
Copy link

Fixed on main in 8486d7e9cce2.

Replaced the lossy worker queue with a bounded disk spool and retained redacted archives, downloadable through the controller at GET /jobs/forgejo-TASK_ID/logs. Ordinary delivery backlog now preserves records and step attribution. When remote delivery loses records, failed steps expose recovery diagnostics with the command, exit code, loss counts, and archive location. Guest budget warnings now explicitly identify omitted output rather than claiming it is combined elsewhere.

Validation passed: all 251 Rust tests, strict Clippy, Rust/Nix formatting, and Nix flake evaluation. Regression coverage includes verbose backlog, rejected rows, archive recovery/redaction/retention, and spool creation failures. The reported live Forgejo/Firecracker workflow has not been rerun with this patch.

Fixed on main in [8486d7e9cce2](https://git.cashu.dev/thesimplekid/rith/commit/8486d7e9cce29251c279c2a4995069f54281277a). Replaced the lossy worker queue with a bounded disk spool and retained redacted archives, downloadable through the controller at `GET /jobs/forgejo-TASK_ID/logs`. Ordinary delivery backlog now preserves records and step attribution. When remote delivery loses records, failed steps expose recovery diagnostics with the command, exit code, loss counts, and archive location. Guest budget warnings now explicitly identify omitted output rather than claiming it is combined elsewhere. Validation passed: all 251 Rust tests, strict Clippy, Rust/Nix formatting, and Nix flake evaluation. Regression coverage includes verbose backlog, rejected rows, archive recovery/redaction/retention, and spool creation failures. The reported live Forgejo/Firecracker workflow has not been rerun with this patch.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
add_ci
shared_reg
metrics
recover_loss
disable_limits
paid-forgejo-accounts
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
thesimplekid/rith#3
No description provided.