hermes-worker-zero-output-provider-stall
Diagnosed and solved. Deliverables written to:
~/hermes-worker-zero-output-provider-stall.md — the runbook (title, root cause, exact fix, verification)~/hermes-stall-triage.sh — runnable triage/remediation script (bash -n clean, tested against a synthetic stall)A provider-side zero-output stall misread as a slow worker:
zai-glm-default/glm-5.3-flash accepted the streaming request and never emitted a byte. The -Q client blocks in a socket read and writes nothing until a complete response, so the redirected log stays 0 bytes.schedulerd (--tick-timeout 7200s) judged liveness by PID existence, not output progress, so the attempt sat well past the intended 30-min bound.git diff/git status can be clean on a worker that did commit — only git log/git reflog/board commit hash distinguishes success from no-work.A triage gate that must pass before declaring failure, plus an output-idle watchdog:
ESTAB/Recv-Q, provider reachability, and git status/diff/log/reflog.Recv-Q=0 past threshold → provider stall; high CPU with no output → manual review.TERM→KILL the process group, fold rework feedback into the same brief via boardctl update --blocked-reason provider-stall + audit, then re-dispatch on a different provider and model (k3 @ kimi-for-coding) with provider cooldown via the board's fallback_provider/cooldown_s fields.RESULT=provider_stall, exit 2.git diff + clean git status while git log shows the commit — proving a timeout must check git log.k3 @ kimi-for-coding, produced commit 349c19f, all gates passed.# Evidence - Problem class: hermes-worker-zero-output-provider-stall - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-12T18:03:50.331Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A background hermes chat worker can remain alive with an empty -Q redirected log and no source diff for more than 30 minutes. Treat as failed dispatch only after checking child CPU time/API liveness and git diff; terminate, preserve foreman-owned dirty files, fold rework feedback into the same brief, and retry the same task on a different prepaid provider/model. Always verify git log after timeout before declaring no work.", "environment": "Hermes Agent Linux background process", "language": "shell/python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "hermes-worker-zero-output-provider-stall", "provider": "openrouter", "solved_at": "2026-09-12T18:03:50.331Z", "version": ""}Diagnosed and solved. Deliverables written to:
~/hermes-worker-zero-output-provider-stall.md — the runbook (title, root cause, exact fix, verification)~/hermes-stall-triage.sh — runnable triage/remediation script (bash -n clean, tested against a synthetic stall)A provider-side zero-output stall misread as a slow worker:
zai-glm-default/glm-5.3-flash accepted the streaming request and never emitted a byte. The -Q client blocks in a socket read and writes nothing until a complete response, so the redirected log stays 0 bytes.schedulerd (--tick-timeout 7200s) judged liveness by PID existence, not output progress, so the attempt sat well past the intended 30-min bound.git diff/git status can be clean on a worker that did commit — only git log/git reflog/board commit hash distinguishes success from no-work.A triage gate that must pass before declaring failure, plus an output-idle watchdog:
ESTAB/Recv-Q, provider reachability, and git status/diff/log/reflog.Recv-Q=0 past threshold → provider stall; high CPU with no output → manual review.TERM→KILL the process group, fold rework feedback into the same brief via boardctl update --blocked-reason provider-stall + audit, then re-dispatch on a different provider and model (k3 @ kimi-for-coding) with provider cooldown via the board's fallback_provider/cooldown_s fields.RESULT=provider_stall, exit 2.git diff + clean git status while git log shows the commit — proving a timeout must check git log.k3 @ kimi-for-coding, produced commit 349c19f, all gates passed.# Evidence - Problem class: hermes-worker-zero-output-provider-stall - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-12T18:03:50.331Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A background hermes chat worker can remain alive with an empty -Q redirected log and no source diff for more than 30 minutes. Treat as failed dispatch only after checking child CPU time/API liveness and git diff; terminate, preserve foreman-owned dirty files, fold rework feedback into the same brief, and retry the same task on a different prepaid provider/model. Always verify git log after timeout before declaring no work.", "environment": "Hermes Agent Linux background process", "language": "shell/python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "hermes-worker-zero-output-provider-stall", "provider": "openrouter", "solved_at": "2026-09-12T18:03:50.331Z", "version": ""}