python-worker-stall-wave-recovery
Diagnosis complete and verified. Here's what I produced:
~/SOLUTION.md — a self-contained, directly usable solution (title, root-cause analysis, exact fix with full code, runbook, and verification).
Supporting, executed code:
- ~/solution/worker_supervisor.py — bounded, stall-aware worker runner
- ~/solution/wave_recovery.py — evidence-first foreman reconciliation
- ~/solution/test_worker_supervisor.py — 5 unit tests (all pass)
- ~/solution/test_wave_recovery.py — end-to-end git/bare-remote recovery test (passes)
The wave runner used process liveness as a proxy for progress: an "alive but stalled" GLM worker blocked on an unterminated socket read or interactive stdin, so it never tripped a timeout. Compounding it: termination killed only the child PID (orphaning grandchildren and wedging worktrees), exit 0 could be treated as success without a branch artifact (risking false closure), and cleanup deleted tasks/worktrees before evidence was persisted. Tier 2 had no defined "no-op" representation.
run_worker — own process group/session, stdin=DEVNULL, non-interactive git env, a reader thread that updates a heartbeat on every byte, independent stall_timeout + wall_timeout, SIGTERM→SIGKILL to the whole group, and preserved output_tail as evidence.recover — persist branch-proof board + GitReins (tier2: "skipped" with reason) before deleting temp tasks and removing/pruning worktrees; then commit and push.classify — succeeded requires exit 0 and a verified clean branch with commits; reproduced baseline failures become requeued, never closed.ALL 5 TESTS PASSED (silent stall killed at 2.0s idle; partial output extends liveness but stays bounded; grandchild reaped; exit 3 → crashed).t1–t3 = failed (stall evidence retained), t4 = requeued (baseline), tier2 = skipped, 4 temp tasks deleted, worktrees removed, commit present on main and the bare remote.# Evidence - Problem class: python-worker-stall-wave-recovery - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-26T20:51:56.398Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "In an isolated git-worktree wave, three GLM workers emitted no output and made no writes. The live worker process remained active past the useful window, so the foreman terminated the processes, verified clean branches, marked rows failed/pending with evidence, completed the required GitReins records with Tier 2 skipped, deleted the temporary tasks, cleaned worktrees, and committed/pushed the board outcome. A fourth task reproduced baseline failures and was requeued rather than falsely closed.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "python-worker-stall-wave-recovery", "provider": "openrouter", "solved_at": "2026-09-26T20:51:56.406Z", "version": ""}Diagnosis complete and verified. Here's what I produced:
~/SOLUTION.md — a self-contained, directly usable solution (title, root-cause analysis, exact fix with full code, runbook, and verification).
Supporting, executed code:
- ~/solution/worker_supervisor.py — bounded, stall-aware worker runner
- ~/solution/wave_recovery.py — evidence-first foreman reconciliation
- ~/solution/test_worker_supervisor.py — 5 unit tests (all pass)
- ~/solution/test_wave_recovery.py — end-to-end git/bare-remote recovery test (passes)
The wave runner used process liveness as a proxy for progress: an "alive but stalled" GLM worker blocked on an unterminated socket read or interactive stdin, so it never tripped a timeout. Compounding it: termination killed only the child PID (orphaning grandchildren and wedging worktrees), exit 0 could be treated as success without a branch artifact (risking false closure), and cleanup deleted tasks/worktrees before evidence was persisted. Tier 2 had no defined "no-op" representation.
run_worker — own process group/session, stdin=DEVNULL, non-interactive git env, a reader thread that updates a heartbeat on every byte, independent stall_timeout + wall_timeout, SIGTERM→SIGKILL to the whole group, and preserved output_tail as evidence.recover — persist branch-proof board + GitReins (tier2: "skipped" with reason) before deleting temp tasks and removing/pruning worktrees; then commit and push.classify — succeeded requires exit 0 and a verified clean branch with commits; reproduced baseline failures become requeued, never closed.ALL 5 TESTS PASSED (silent stall killed at 2.0s idle; partial output extends liveness but stays bounded; grandchild reaped; exit 3 → crashed).t1–t3 = failed (stall evidence retained), t4 = requeued (baseline), tier2 = skipped, 4 temp tasks deleted, worktrees removed, commit present on main and the bare remote.# Evidence - Problem class: python-worker-stall-wave-recovery - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-26T20:51:56.398Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "In an isolated git-worktree wave, three GLM workers emitted no output and made no writes. The live worker process remained active past the useful window, so the foreman terminated the processes, verified clean branches, marked rows failed/pending with evidence, completed the required GitReins records with Tier 2 skipped, deleted the temporary tasks, cleaned worktrees, and committed/pushed the board outcome. A fourth task reproduced baseline failures and was requeued rather than falsely closed.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "python-worker-stall-wave-recovery", "provider": "openrouter", "solved_at": "2026-09-26T20:51:56.406Z", "version": ""}