Failed attempt has no output and Forgejo log download returns 500: missing task log file #5

New issue

Closed

opened 2026-10-18 14:16:30 +00:00 by thesimplekid · 1 comment

thesimplekid commented

2026-10-18 14:16:30 +00:00

Owner

Rith revision: 8486d7e9cc (current deployment pin and current upstream main when checked). Forgejo 15.0.9. Disposable NixOS Firecracker guests, 4 vCPUs / 8192 MiB, three concurrent jobs per controller. Persistent redacted archives configured at /data/rith/logs.

Failed attempt:
https://git.cashu.dev/thesimplekid/satsettle/actions/runs/53/jobs/2/attempt/1

Its log-download endpoint:
https://git.cashu.dev/thesimplekid/satsettle/actions/runs/53/jobs/2/attempt/1/logs

Repeated GET requests return HTTP 500 with this body:

dbfs open "actions_log/thesimplekid/satsettle/af/16815.log.zst": file does not exist

The page's structured state says:

Job cdk-compatible: failure, duration 13m27s.
Set up job: success, duration 13m27s.
actions/checkout@v4: failure, duration 0s.
Resolve the latest compatible CDK releases: failure, duration 0s.
Check compatible CDK releases: failure, duration 0s.
Complete job: failure, duration 0s.
stepsLog is empty and preExecutionError is empty.
Internal run ID is 4366; the missing log file corresponds to task 16815.

Controls checked at the same time:

Run 53 job 0 (format), job 1 (build), and job 3 (dependencies) log downloads all return HTTP 200 with nonempty output.
Run 54's cdk-compatible job for the same commit 19ef6d7824 is successful, so the evidence does not establish a deterministic project check failure.

Please investigate finalization and failure reporting when no remote log file exists. Even for an infrastructure/guest failure, the failed step should expose a diagnostic and recovery archive link, and finalized attempts should have a downloadable diagnostic log rather than an HTTP 500. The initial setup step consuming the entire runtime while all following steps fail at 0s also needs review.

The underlying job failure and responsibility for the missing remote file are not yet determined: this could be a runner upload/finalization failure or a Forgejo storage issue. Direct runner archive/journal access has been requested and is still pending; no claim is made that the local archive is missing. This report records the reproducible integration failure, not a confirmed internal root cause.

A completed failed Forgejo attempt exposes no useful logs, and its downloadable log returns HTTP 500 because the expected Forgejo log file does not exist. This occurs after deploying the log-spool fix from #3. Rith revision: 8486d7e9cce29251c279c2a4995069f54281277a (current deployment pin and current upstream main when checked). Forgejo 15.0.9. Disposable NixOS Firecracker guests, 4 vCPUs / 8192 MiB, three concurrent jobs per controller. Persistent redacted archives configured at /data/rith/logs. Failed attempt: https://git.cashu.dev/thesimplekid/satsettle/actions/runs/53/jobs/2/attempt/1 Its log-download endpoint: https://git.cashu.dev/thesimplekid/satsettle/actions/runs/53/jobs/2/attempt/1/logs Repeated GET requests return HTTP 500 with this body: ``` dbfs open "actions_log/thesimplekid/satsettle/af/16815.log.zst": file does not exist ``` The page's structured state says: - Job cdk-compatible: failure, duration 13m27s. - Set up job: success, duration 13m27s. - actions/checkout@v4: failure, duration 0s. - Resolve the latest compatible CDK releases: failure, duration 0s. - Check compatible CDK releases: failure, duration 0s. - Complete job: failure, duration 0s. - stepsLog is empty and preExecutionError is empty. - Internal run ID is 4366; the missing log file corresponds to task 16815. Controls checked at the same time: - Run 53 job 0 (format), job 1 (build), and job 3 (dependencies) log downloads all return HTTP 200 with nonempty output. - Run 54's cdk-compatible job for the same commit 19ef6d7824 is successful, so the evidence does not establish a deterministic project check failure. Please investigate finalization and failure reporting when no remote log file exists. Even for an infrastructure/guest failure, the failed step should expose a diagnostic and recovery archive link, and finalized attempts should have a downloadable diagnostic log rather than an HTTP 500. The initial setup step consuming the entire runtime while all following steps fail at 0s also needs review. The underlying job failure and responsibility for the missing remote file are not yet determined: this could be a runner upload/finalization failure or a Forgejo storage issue. Direct runner archive/journal access has been requested and is still pending; no claim is made that the local archive is missing. This report records the reproducible integration failure, not a confirmed internal root cause.

thesimplekid commented

2026-10-18 14:23:37 +00:00

Author

Owner

Runner access is now available. The evidence points to an orphaned task claim after FetchTask times out, rather than a job that executed and then lost its output.

Observed on baile-ci-runner-01:

2026-10-18T13:59:56+00:00 Accepted forgejo-16813
2026-10-18T14:00:00+00:00 Accepted forgejo-16814
2026-10-18T14:00:08+00:00 Forgejo polling or event forwarding failed:
  Forgejo HTTP error: .../runner.v1.RunnerService/FetchTask:
  Network Error: Error encountered in the status line: timed out reading response
2026-10-18T14:00:12+00:00 Accepted forgejo-16816

There is no Accepted entry for 16815 in the interval, no forgejo-16815 entry in /var/lib/unified-ci/state.json's jobs map or completed_jobs, no local archive for SHA256("forgejo-16815"), and GET /jobs/forgejo-16815 returns 404. Archives for neighbouring tasks exist. There are no kernel journal entries in the same interval indicating a host OOM. The controller continued processing other jobs.

Forgejo's attempt start time is 14:00:06 UTC; it later reports failure after 13m27s. The missing file is specifically task 16815's log. The account currently lists only runner ID 8175 / UUID 0d1b1d2c-6768-479c-a042-8adafe82ca3f; the repository has no separately registered runners.

Source observation at 8486d7e: DEFAULT_CONNECT_HTTP_TIMEOUT is hardcoded to 5 seconds in src/ingress.rs. fetch_task only ingests the response after the unary HTTP call succeeds; this failed request leaves no local task ID to heartbeat, execute, archive, or finalize.

Inference (needs Forgejo server-side confirmation): the server assigned 16815, but its response was lost/timed out; the runner moved on and the task remained orphaned until Forgejo expired it. The absence of a local log archive is therefore expected if execution never began. The disk log-spool fix cannot recover a task ID never received.

Please investigate claim/acknowledgement or reconciliation/idempotency for FetchTask with a lost response, and make transport timeouts configurable/appropriate for WAN/VPN use. Raising the timeout alone reduces likelihood but does not eliminate the lost-response problem. Server-side failures for never-started tasks should also provide a useful diagnostic and a safe log-download response instead of HTTP 500.

No live configuration was changed during this diagnosis.

Runner access is now available. The evidence points to an orphaned task claim after FetchTask times out, rather than a job that executed and then lost its output. Observed on baile-ci-runner-01: ``` 2026-10-18T13:59:56+00:00 Accepted forgejo-16813 2026-10-18T14:00:00+00:00 Accepted forgejo-16814 2026-10-18T14:00:08+00:00 Forgejo polling or event forwarding failed: Forgejo HTTP error: .../runner.v1.RunnerService/FetchTask: Network Error: Error encountered in the status line: timed out reading response 2026-10-18T14:00:12+00:00 Accepted forgejo-16816 ``` There is no Accepted entry for 16815 in the interval, no `forgejo-16815` entry in `/var/lib/unified-ci/state.json`'s jobs map or completed_jobs, no local archive for SHA256("forgejo-16815"), and GET `/jobs/forgejo-16815` returns 404. Archives for neighbouring tasks exist. There are no kernel journal entries in the same interval indicating a host OOM. The controller continued processing other jobs. Forgejo's attempt start time is 14:00:06 UTC; it later reports failure after 13m27s. The missing file is specifically task 16815's log. The account currently lists only runner ID 8175 / UUID 0d1b1d2c-6768-479c-a042-8adafe82ca3f; the repository has no separately registered runners. Source observation at 8486d7e: `DEFAULT_CONNECT_HTTP_TIMEOUT` is hardcoded to 5 seconds in src/ingress.rs. `fetch_task` only ingests the response after the unary HTTP call succeeds; this failed request leaves no local task ID to heartbeat, execute, archive, or finalize. Inference (needs Forgejo server-side confirmation): the server assigned 16815, but its response was lost/timed out; the runner moved on and the task remained orphaned until Forgejo expired it. The absence of a local log archive is therefore expected if execution never began. The disk log-spool fix cannot recover a task ID never received. Please investigate claim/acknowledgement or reconciliation/idempotency for FetchTask with a lost response, and make transport timeouts configurable/appropriate for WAN/VPN use. Raising the timeout alone reduces likelihood but does not eliminate the lost-response problem. Server-side failures for never-started tasks should also provide a useful diagnostic and a safe log-download response instead of HTTP 500. No live configuration was changed during this diagnosis.

thesimplekid referenced this issue from a commit

2026-10-18 14:34:55 +00:00

fix: recover Forgejo task claims after lost fetch responses

thesimplekid referenced this issue

2026-10-18 14:35:02 +00:00

fix: recover Forgejo task claims after lost fetch responses #7

thesimplekid closed this issue

2026-10-20 21:25:55 +00:00

thesimplekid referenced this issue

2026-10-23 13:44:59 +00:00

Preserve per-attempt stop reasons and diagnostic links for interrupted Firecracker jobs #13