scheduler-refired-wave-tick-recovery
Wrote the verified solution to ~/solution.md.
Root cause — two independent defects
- RC-1 (the refire): dispatch is fire-and-forget. waves/<tick>.json is only written complete at the end, so retry/duplicate delivery re-fires a tick id with no check-and-claim.
- RC-2 (the corruption): fold stages a directory (git add verdicts/), which auto-shrinks to the uncommitted remainder. A concurrent T249 commit mid-tick invalidates the path snapshot, so the fold can re-commit sibling files or drop content.
Immediate recovery — a drop-in recover-wave.sh implementing the exact ordered gates: confirm refire → verify merge completeness → drain async re-judges → re-derive remainder under a HEAD-move guard → T234 secret scan → fold explicit paths only → premise-check pending rows → write idempotent marker. Plus a manual command equivalent.
Durable fix — foreman_guard.py: deterministic tick id, atomic O_EXCL claim + flock, terminal-status refusal, lease expiry with a fencing token, and a locked fold window. The scheduler calls claim_tick() before any pick; a completed/in-flight tick becomes a no-op.
Verification — actually executed, not just written:
- test.sh (sibling commits before fold): all assertions passed.
- test_collision.sh (PATH shim commits board/header.md during the fold): the guard caught the HEAD move, re-derived from 3 → 2 paths, and respected auto-shrink; all checks passed.
- foreman_guard.py --selftest: passed (duplicate rejected, refire after complete rejected, fence takeover enforced, stale fence rejected).
- I re-extracted the embedded script/guard from the markdown, ran bash -n, ran both repro harnesses against the embedded recovery script, and ran the embedded guard selftest — all green.
Also included: post-recovery acceptance checks, abort/rollback criteria, and a git revert (not reset --hard) recovery for a bad fold so sibling T249 work is preserved.
# Evidence - Problem class: scheduler-refired-wave-tick-recovery - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-26T20:32:01.688Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Foreman tick re-fired an already-executed wave tick id. Detection: waves/<tick-id>.json exists with status complete + git log shows merge commits naming the wave. Recovery path: do NOT re-execute picks; verify merge completeness (git log + rev-list origin/main..HEAD), poll in-flight async re-judges (gitreins judge --status), fold ONLY the uncommitted remainder (verdict dirs after T234 secret scan, board rows/events, header), premise-check any still-pending rows against the tree before dispatching new work. Sibling hazard: a concurrent T249 session committed the same class of bookkeeping mid-tick; staged-set diff must be re-derived after the collision (git add of a directory auto-shrinks to the uncommitted remainder).", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "scheduler-refired-wave-tick-recovery", "provider": "openrouter", "solved_at": "2026-09-26T20:32:01.689Z", "version": ""}Wrote the verified solution to ~/solution.md.
Root cause — two independent defects
- RC-1 (the refire): dispatch is fire-and-forget. waves/<tick>.json is only written complete at the end, so retry/duplicate delivery re-fires a tick id with no check-and-claim.
- RC-2 (the corruption): fold stages a directory (git add verdicts/), which auto-shrinks to the uncommitted remainder. A concurrent T249 commit mid-tick invalidates the path snapshot, so the fold can re-commit sibling files or drop content.
Immediate recovery — a drop-in recover-wave.sh implementing the exact ordered gates: confirm refire → verify merge completeness → drain async re-judges → re-derive remainder under a HEAD-move guard → T234 secret scan → fold explicit paths only → premise-check pending rows → write idempotent marker. Plus a manual command equivalent.
Durable fix — foreman_guard.py: deterministic tick id, atomic O_EXCL claim + flock, terminal-status refusal, lease expiry with a fencing token, and a locked fold window. The scheduler calls claim_tick() before any pick; a completed/in-flight tick becomes a no-op.
Verification — actually executed, not just written:
- test.sh (sibling commits before fold): all assertions passed.
- test_collision.sh (PATH shim commits board/header.md during the fold): the guard caught the HEAD move, re-derived from 3 → 2 paths, and respected auto-shrink; all checks passed.
- foreman_guard.py --selftest: passed (duplicate rejected, refire after complete rejected, fence takeover enforced, stale fence rejected).
- I re-extracted the embedded script/guard from the markdown, ran bash -n, ran both repro harnesses against the embedded recovery script, and ran the embedded guard selftest — all green.
Also included: post-recovery acceptance checks, abort/rollback criteria, and a git revert (not reset --hard) recovery for a bad fold so sibling T249 work is preserved.
# Evidence - Problem class: scheduler-refired-wave-tick-recovery - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-26T20:32:01.688Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Foreman tick re-fired an already-executed wave tick id. Detection: waves/<tick-id>.json exists with status complete + git log shows merge commits naming the wave. Recovery path: do NOT re-execute picks; verify merge completeness (git log + rev-list origin/main..HEAD), poll in-flight async re-judges (gitreins judge --status), fold ONLY the uncommitted remainder (verdict dirs after T234 secret scan, board rows/events, header), premise-check any still-pending rows against the tree before dispatching new work. Sibling hazard: a concurrent T249 session committed the same class of bookkeeping mid-tick; staged-set diff must be re-derived after the collision (git add of a directory auto-shrinks to the uncommitted remainder).", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "scheduler-refired-wave-tick-recovery", "provider": "openrouter", "solved_at": "2026-09-26T20:32:01.689Z", "version": ""}