github-ci-runner-no-space-left-on-device
Done. Saved to ~/DX-FIX-github-ci-runner-no-space.md.
Root cause: The GitHub-hosted runner's own disk exhausted (ENOSPC) while writing ~/actions-runner/cached/.../_diag/Worker_*.log. The disk was filled by the duckdb-sys cargo cache plus the full build. This is runner infra failure — thrown by the runner process during teardown, not by your CI. That's why all four real steps (format/build/clippy/test) were green yet conclusion=failure.
Why all 3 log paths failed:
- 403 Must have admin rights — job logs need auth/admin on the non-public endpoint
- 404 BlobNotFound — wrong id namespace, and the per-run log artifact was never written because the runner died before flushing
- gh run view --log-failed empty — the "failed" step is the runner's cleanup step with no log to fetch
The fix (no code change):
gh run rerun <run_id>
Fresh runner #2 starts with clean disk → same commit builds/tests as the green checks. If attempt 2's real steps pass, no INT-CI board task for the chore-only JSONL commit.
Key takeaway for the runbook: On infra failures, log downloads fail in three different ways — so the first move must be gh run view <run_id> (check annotations) before assuming a code regression. The ENOSPC annotation is ground truth; missing logs are red herrings.
The document also includes a verification section (rerun reaching conclusion = success, confirming no ENOSPC annotation, and the decision rule for whether a board task/Microsoft support ticket is warranted). You can review it directly at that path.
# Evidence - Problem class: github-ci-runner-no-space-left-on-device - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-08-23T21:29:24.495Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "GitHub Actions run failed with conclusion=failure on a chore-only commit (board JSONL). All real steps passed (format, build, clippy, test all green checkmarks) but run concluded failure after 37m. Job log download paths all failed: unauthenticated API logs endpoint 403 'Must have admin rights', gh api jobs/<id>/logs returned 404 BlobNotFound, gh run view --log-failed returned EMPTY output. The evidence that cracked it: `gh run view <run_id>` prints ANNOTATIONS \u2014 'Unhandled exception. System.IO.IOException: No space left on device : ~/actions-runner/cached/.../_diag/Worker_*.log' \u2014 the GitHub-hosted runner's own disk filled (duckdb-sys cargo cache + full build). This is runner INFRA failure, not repo CI: fix = `gh run rerun <run_id>` (attempt 2), no code change, no INT-CI board task if rerun passes. Detection path: gh run view annotations BEFORE assuming code regression; logs endpoints fail in 3 different ways on infra failures.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "github-ci-runner-no-space-left-on-device", "provider": "openrouter", "solved_at": "2026-08-23T21:29:24.495Z", "version": ""}Done. Saved to ~/DX-FIX-github-ci-runner-no-space.md.
Root cause: The GitHub-hosted runner's own disk exhausted (ENOSPC) while writing ~/actions-runner/cached/.../_diag/Worker_*.log. The disk was filled by the duckdb-sys cargo cache plus the full build. This is runner infra failure — thrown by the runner process during teardown, not by your CI. That's why all four real steps (format/build/clippy/test) were green yet conclusion=failure.
Why all 3 log paths failed:
- 403 Must have admin rights — job logs need auth/admin on the non-public endpoint
- 404 BlobNotFound — wrong id namespace, and the per-run log artifact was never written because the runner died before flushing
- gh run view --log-failed empty — the "failed" step is the runner's cleanup step with no log to fetch
The fix (no code change):
gh run rerun <run_id>
Fresh runner #2 starts with clean disk → same commit builds/tests as the green checks. If attempt 2's real steps pass, no INT-CI board task for the chore-only JSONL commit.
Key takeaway for the runbook: On infra failures, log downloads fail in three different ways — so the first move must be gh run view <run_id> (check annotations) before assuming a code regression. The ENOSPC annotation is ground truth; missing logs are red herrings.
The document also includes a verification section (rerun reaching conclusion = success, confirming no ENOSPC annotation, and the decision rule for whether a board task/Microsoft support ticket is warranted). You can review it directly at that path.
# Evidence - Problem class: github-ci-runner-no-space-left-on-device - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-08-23T21:29:24.495Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "GitHub Actions run failed with conclusion=failure on a chore-only commit (board JSONL). All real steps passed (format, build, clippy, test all green checkmarks) but run concluded failure after 37m. Job log download paths all failed: unauthenticated API logs endpoint 403 'Must have admin rights', gh api jobs/<id>/logs returned 404 BlobNotFound, gh run view --log-failed returned EMPTY output. The evidence that cracked it: `gh run view <run_id>` prints ANNOTATIONS \u2014 'Unhandled exception. System.IO.IOException: No space left on device : ~/actions-runner/cached/.../_diag/Worker_*.log' \u2014 the GitHub-hosted runner's own disk filled (duckdb-sys cargo cache + full build). This is runner INFRA failure, not repo CI: fix = `gh run rerun <run_id>` (attempt 2), no code change, no INT-CI board task if rerun passes. Detection path: gh run view annotations BEFORE assuming code regression; logs endpoints fail in 3 different ways on infra failures.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "github-ci-runner-no-space-left-on-device", "provider": "openrouter", "solved_at": "2026-08-23T21:29:24.495Z", "version": ""}