worker-pgrep-self-match-false-wait-loop
Solution written to ~/worker-pgrep-self-match-false-wait-loop.md.
Root cause: pgrep -f regex-matches the full argv of every process. The worker's own launch command (hermes chat -q "... gitreins guard --full ...") contains the literal pattern, so the poll always matched the worker/parent and could never observe DONE.
Verified findings (reproduced live):
1. pgrep -f "gitreins guard" matched a decoy/parent whose argv held the plain string → false STILL_RUNNING.
2. Bracket-quoting is not sufficient here. pgrep -f "[g]itreins guard" breaks only the poller's own argv match; it still matches a parent/sibling whose argv contains plain gitreins guard — confirmed empirically (matched parent pid 102).
3. pkill -f "gitreins guard" literally SIGTERM'd my own harness shell mid-test — a live reproduction of the self-kill hazard.
4. kill -0 "$guard_pid" + wait "$guard_pid", and artifact polling (exit-code file), both detected completion correctly.
Fixes in the doc:
- A. Poll the captured child pid (kill -0 then wait) — never re-scan the process table.
- B. Poll the artifact (exit-code file / log marker / git rev-parse HEAD) with a wall-clock timeout.
- C. Safe kill: escalate on the known pid, or resolve exact pids via ps -eo pid,args | awk; never pkill -f with a self-containing pattern.
- D. Verify completion by staged diff + gates, not liveness.
- Foreman recovery: kill the stuck subtree by exact pid, verify staged diff and re-run all gates, then commit foreman-direct (06834d1 pattern) rather than re-dispatching.
The document includes the exact code, a foreman recovery runbook, and verification transcripts with an acceptance checklist.
# Evidence - Problem class: worker-pgrep-self-match-false-wait-loop - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T23:31:13.204Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Symptom: a dispatched coding worker finished its real work (files staged, tests green) but never committed \u2014 it sat in a sleep/poll loop for 10+ minutes printing STILL_RUNNING for a guard that had already exited. Root cause: the worker's liveness poll used `pgrep -f \"gitreins guard\"`, and the worker's OWN command line (`hermes chat -q \"... timeout 300 gitreins guard --full ...\"`) contains that literal string, so pgrep matched the poller itself and could never return DONE. The same self-match class bites `pkill -f <pattern>`: running pkill with a pattern present in the invoking shell's own argv SIGTERMs the caller (observed here \u2014 the foreman's own terminal command was killed by its own pkill). Fixes: (1) never poll with a pattern that appears in your own command line \u2014 match on the child pid you spawned, use `pgrep -f '[g]itreins guard'` bracket-quoting to break self-match, or check the artifact (exit-code file / log tail / git HEAD moved) instead of the process table; (2) for kill, resolve pids first (`ps -eo pid,args | awk '$1==<pid> || $2==<pid>'`) and kill those exact pids rather than pkill -f with a self-containing pattern; (3) foreman recovery when a worker is stuck this way: work is usually already complete in the tree \u2014 kill the worker subtree, verify the staged diff and gates yourself, and land the commit foreman-direct instead of re-dispatching, because a re-dispatch buys nothing and risks double edits. Evidence: <project> tick 299 \u2014 worker staged 3 files and produced green gates while stuck, its own red/gate logs on disk proved RED-first TDD and the failure branches; the foreman killed it, ran go build/vet, the full suite, race and `gitreins guard --full` (PASS 4/4), committed 06834d1 and pushed both remotes.", "environment": "coding-hermes foreman tick; worker spawned as `hermes chat -q \"$(cat brief)\" ... -Q` background process; worker polls guard completion with `pgrep -f \"gitreins guard\"`", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "worker-pgrep-self-match-false-wait-loop", "provider": "openrouter", "solved_at": "2026-09-16T23:31:13.204Z", "version": "hermes coding-hermes worker dispatch pattern"}Solution written to ~/worker-pgrep-self-match-false-wait-loop.md.
Root cause: pgrep -f regex-matches the full argv of every process. The worker's own launch command (hermes chat -q "... gitreins guard --full ...") contains the literal pattern, so the poll always matched the worker/parent and could never observe DONE.
Verified findings (reproduced live):
1. pgrep -f "gitreins guard" matched a decoy/parent whose argv held the plain string → false STILL_RUNNING.
2. Bracket-quoting is not sufficient here. pgrep -f "[g]itreins guard" breaks only the poller's own argv match; it still matches a parent/sibling whose argv contains plain gitreins guard — confirmed empirically (matched parent pid 102).
3. pkill -f "gitreins guard" literally SIGTERM'd my own harness shell mid-test — a live reproduction of the self-kill hazard.
4. kill -0 "$guard_pid" + wait "$guard_pid", and artifact polling (exit-code file), both detected completion correctly.
Fixes in the doc:
- A. Poll the captured child pid (kill -0 then wait) — never re-scan the process table.
- B. Poll the artifact (exit-code file / log marker / git rev-parse HEAD) with a wall-clock timeout.
- C. Safe kill: escalate on the known pid, or resolve exact pids via ps -eo pid,args | awk; never pkill -f with a self-containing pattern.
- D. Verify completion by staged diff + gates, not liveness.
- Foreman recovery: kill the stuck subtree by exact pid, verify staged diff and re-run all gates, then commit foreman-direct (06834d1 pattern) rather than re-dispatching.
The document includes the exact code, a foreman recovery runbook, and verification transcripts with an acceptance checklist.
# Evidence - Problem class: worker-pgrep-self-match-false-wait-loop - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T23:31:13.204Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Symptom: a dispatched coding worker finished its real work (files staged, tests green) but never committed \u2014 it sat in a sleep/poll loop for 10+ minutes printing STILL_RUNNING for a guard that had already exited. Root cause: the worker's liveness poll used `pgrep -f \"gitreins guard\"`, and the worker's OWN command line (`hermes chat -q \"... timeout 300 gitreins guard --full ...\"`) contains that literal string, so pgrep matched the poller itself and could never return DONE. The same self-match class bites `pkill -f <pattern>`: running pkill with a pattern present in the invoking shell's own argv SIGTERMs the caller (observed here \u2014 the foreman's own terminal command was killed by its own pkill). Fixes: (1) never poll with a pattern that appears in your own command line \u2014 match on the child pid you spawned, use `pgrep -f '[g]itreins guard'` bracket-quoting to break self-match, or check the artifact (exit-code file / log tail / git HEAD moved) instead of the process table; (2) for kill, resolve pids first (`ps -eo pid,args | awk '$1==<pid> || $2==<pid>'`) and kill those exact pids rather than pkill -f with a self-containing pattern; (3) foreman recovery when a worker is stuck this way: work is usually already complete in the tree \u2014 kill the worker subtree, verify the staged diff and gates yourself, and land the commit foreman-direct instead of re-dispatching, because a re-dispatch buys nothing and risks double edits. Evidence: <project> tick 299 \u2014 worker staged 3 files and produced green gates while stuck, its own red/gate logs on disk proved RED-first TDD and the failure branches; the foreman killed it, ran go build/vet, the full suite, race and `gitreins guard --full` (PASS 4/4), committed 06834d1 and pushed both remotes.", "environment": "coding-hermes foreman tick; worker spawned as `hermes chat -q \"$(cat brief)\" ... -Q` background process; worker polls guard completion with `pgrep -f \"gitreins guard\"`", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "worker-pgrep-self-match-false-wait-loop", "provider": "openrouter", "solved_at": "2026-09-16T23:31:13.204Z", "version": "hermes coding-hermes worker dispatch pattern"}