gitreins-tier2-stale-poll-loop-hang

Answer

Done. The solution is written and verified.

Deliverables

• ~/gitreins-tier2-stale-poll-loop-hang.md — full diagnosis + fix + verification (364 lines)
• ~/evaluator_stall_watchdog.sh — the ready-to-run watchdog (byte-identical to the code embedded in the doc)

Root cause

The tier2 evaluator blocks in while :; do sleep 29; wc -l <logfile>; done on a file whose byte count never changes (real work ran elsewhere/piped). That loop burns ~0 CPU, so after 60 min the judge shows 0:00:00–0:00:06 TIME while the evaluator log still emits incidental compaction warnings — making log-based liveness checks useless. A synchronous gitreins judge <id> just sits there until the 90m max_time cap fires. The 90m cap isn't the bug; the missing early detection is.

Fix

Watch the aggregate CPU time of the judge's process tree (/proc/<pid>/stat utime+stime summed over root + descendants). If CPU advances < 5s over two consecutive 5-minute windows, kill the whole process group and re-dispatch gitreins judge --async <id>. Production trip time ≈10 min, and the 90m rung stays intact for the async re-run.

Verification (reproduced, 8/8 checks pass)

Because the real gitreins venv is absent in this container, I reproduced the exact pathology and tested the watchdog with a fake judge:

• Stale run: eval log grew 5 compaction lines, polled suite.log stayed at 0, tree used 0s CPU. Watchdog detected the stall, SIGTERM'd the group, killed it, and invoked judge --async gap181 — all in 7s with test settings (300s×2 in production).
• Healthy run (busy child): CPU advanced each window, not killed — no false positive.
• Verified setsid/pgid kill removed the judge and its sleep 29 child with no survivors.
• The code block embedded in the markdown matches the tested artifact exactly.

Scripts/harness are at /tmp/wdtest/ (run_tests.sh, stale_eval.sh, busy_eval.sh, fake_gitreins.sh).