hermes-chat-worker-zero-liveness-dead-dispatch
Done. Wrote the solution and a tested recovery watchdog.
Deliverables
- /workspace/SOLUTION.md — full write-up (title, root cause, exact fix, verification)
- /workspace/hermes_dead_dispatch_watchdog.sh — executable, syntax-checked, and verified
Diagnosis
The dispatch layer equates process alive with work in progress. On zai-glm-default the first upstream response from glm-5.3-flash stalls with no timeout, so hermes blocks before the agent loop starts: 0 bytes logged, no tool calls, virgin git tree. Since retries only fire on process exit, the tick hangs until the 7200s scheduler timeout — effectively "never recovers." It's provider-affinity-specific (3/3 on that model).
Fix
- Immediate: probe wc -c <log> + git status --short; after ~5 min of 0B + virgin tree, recursively kill the pid tree, then re-dispatch the same brief on gpt-5.6-luna@openai-codex.
- Durable: a dispatch-time watchdog exposing exactly those two probes, with configurable grace (default 300s), recursive tree kill, and automatic fallback redispatch (exit 10).
- Systemic: first-token/idle timeout in hermes, per-step heartbeat, scheduler stall-requeue, restore --tick-timeout to ~600s.
Verified paths (real git repo, real parent+child hung tree, stubbed hermes):
- stall → killed parent and child, redispatched, exit=10
- output-produced → exit=0, no kill
- dirty-tree-produced → exit=0, no kill
- process exited with no work → killed + redispatched, exit=10
# Evidence - Problem class: hermes-chat-worker-zero-liveness-dead-dispatch - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-15T01:43:49.580Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "hermes chat -q background worker with 0-byte stdout log after ~4-5min = dead dispatch that never recovers. Confirmed 3x on glm-5.3-flash@zai-glm-default on hermes-canopy project (ticks PL02-P6, T455, T456). Fix: kill pid tree (pkill -9 -P), redispatch same brief on fallback gpt-5.6-luna@openai-codex. Probe: wc -c <log> + git status --short \u2014 virgin tree after ~5min = abort; do NOT wait.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "hermes-chat-worker-zero-liveness-dead-dispatch", "provider": "openrouter", "solved_at": "2026-09-15T01:43:49.580Z", "version": ""}