qa-harness-preflight-failure-hides-zero-coverage
Solution written to /workspace/solution.md and verified end-to-end.
The described repo (~/<project>) is not present in this sandbox — only /workspace/problem.json. So I reconstructed the precise failure logic from the report and built a runnable harness at /tmp/qa-fix-demo that reproduces each defect and proves the fix.
Root cause is a 4-link chain:
1. *-qa scheduler row workdir points at an empty stand-in dir (.../stand-in/pm/<project>, no .git) instead of the base row's workdir (~/<project>).
2. bunker-qa.sh's git-repo preflight exits before : > "$EVIDENCE" and before the write_fail_if_empty trap is installed → no artifact, no FAIL cell.
3. parse_cells returns {cells:[],cell_count:0,head:"?"} while the node reports ok → FINDINGS findings=0 filed=0 → green.
4. qa_discover.py doesn't exclude -qa, emits the raw stand-in workdir, and never ledgers zero-coverage → dead rows re-picked first forever.
bunker-qa.sh — create/truncate $EVIDENCE and install the fail-if-empty trap before the git preflight; emit an explicit FAIL/not-a-git-repo cell on preflight failure.examples/coding-hermes/qa.ts — parse_cells(evidencePath); fail closed (status: "error") when evidence is absent or cell_count == 0, and ledger every zero-coverage run.qa_discover.py — resolve X-qa → base row X's workdir, skip rows whose resolved workdir lacks .git, and ledger the skip.| Case | Before | After |
|---|---|---|
| Dead target evidence file | absent | created |
| FAIL cell on dead target | none | 1 |
parse_cells.cell_count |
0 | 1 |
| Node status on absent evidence | ok |
error |
<project>-qa workdir |
stand-in (no .git) |
/…/<project> |
| Zero-coverage ledgered | no | yes |
| Real base repo | — | PASS |
Repro commands:
bash /tmp/qa-fix-demo/run.sh
bash /tmp/qa-fix-demo/run_discover.sh
node /tmp/qa-fix-demo/qa_parse_test.mjs
The document includes the full before/after code for all three files, the failure chain, acceptance criteria, and the exact observed harness output.
# Evidence - Problem class: qa-harness-preflight-failure-hides-zero-coverage - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-14T13:30:58.735Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "QA role pipelines (bunker-qa.sh driven by examples/coding-hermes/qa.ts) can report Passed: N | Failed: 0 while auditing ZERO real repos, and no evidence artifact is left behind. Chain: (1) the target repo path handed to the battery belongs to a *-qa scheduler row whose workdir is an empty stand-in dir (e.g. ~/.hermes/stand-in/pm/<project>, 0 entries, no .git) instead of the base project workdir (~/<project>); (2) bunker-qa.sh's preflight `[ -d \"$repo/.git\" ] || { echo \"ERROR: $repo is not a git repo\" >&2; exit 1; }` sits ABOVE the `: > \"$EVIDENCE\"` truncation and above the write_fail_if_empty traps, so it exits before the evidence file is even created (run_battery reports evidence_path /tmp/bunker-qa-evidence-*.jsonl that does not exist on disk, and no FAIL cell row is written); (3) parse_cells then returns {cells:[],cell_count:0,head:\"?\"} while the node reports status ok, so the tick's verdict is FINDINGS with findings=0 filed=0 and the pipeline closes green. Discovery keeps re-picking the same dead rows because qa_discover.py excludes -pm/-sync/dogfood suffix rows but not -qa, and it emits the row's raw workdir instead of the base project workdir; the QA ledger (~/.hermes/qa/ledger.jsonl) has zero -qa rows, so they never age out and sort first as never-qa every round. Fix direction: create the evidence file (and the fail-if-empty trap) BEFORE the git-repo preflight so a dead path yields a FAIL cell + artifact; make qa_discover.py resolve a -qa row's workdir to its BASE project row's workdir (or skip rows whose workdir has no .git); ledger every zero-coverage run so discovery can age it. Verification: repro on 2026-09-14 shows run_battery raw=\"ERROR: <dir> is not a git repo\", evidence file absent, parse_cells cells=[] cell_count=0, pipeline 'Passed: 4 | Failed: 0' with 0 real repos audited.", "environment": "", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "qa-harness-preflight-failure-hides-zero-coverage", "provider": "openrouter", "solved_at": "2026-09-14T13:30:58.736Z", "version": ""}Solution written to /workspace/solution.md and verified end-to-end.
The described repo (~/<project>) is not present in this sandbox — only /workspace/problem.json. So I reconstructed the precise failure logic from the report and built a runnable harness at /tmp/qa-fix-demo that reproduces each defect and proves the fix.
Root cause is a 4-link chain:
1. *-qa scheduler row workdir points at an empty stand-in dir (.../stand-in/pm/<project>, no .git) instead of the base row's workdir (~/<project>).
2. bunker-qa.sh's git-repo preflight exits before : > "$EVIDENCE" and before the write_fail_if_empty trap is installed → no artifact, no FAIL cell.
3. parse_cells returns {cells:[],cell_count:0,head:"?"} while the node reports ok → FINDINGS findings=0 filed=0 → green.
4. qa_discover.py doesn't exclude -qa, emits the raw stand-in workdir, and never ledgers zero-coverage → dead rows re-picked first forever.
bunker-qa.sh — create/truncate $EVIDENCE and install the fail-if-empty trap before the git preflight; emit an explicit FAIL/not-a-git-repo cell on preflight failure.examples/coding-hermes/qa.ts — parse_cells(evidencePath); fail closed (status: "error") when evidence is absent or cell_count == 0, and ledger every zero-coverage run.qa_discover.py — resolve X-qa → base row X's workdir, skip rows whose resolved workdir lacks .git, and ledger the skip.| Case | Before | After |
|---|---|---|
| Dead target evidence file | absent | created |
| FAIL cell on dead target | none | 1 |
parse_cells.cell_count |
0 | 1 |
| Node status on absent evidence | ok |
error |
<project>-qa workdir |
stand-in (no .git) |
/…/<project> |
| Zero-coverage ledgered | no | yes |
| Real base repo | — | PASS |
Repro commands:
bash /tmp/qa-fix-demo/run.sh
bash /tmp/qa-fix-demo/run_discover.sh
node /tmp/qa-fix-demo/qa_parse_test.mjs
The document includes the full before/after code for all three files, the failure chain, acceptance criteria, and the exact observed harness output.
# Evidence - Problem class: qa-harness-preflight-failure-hides-zero-coverage - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-14T13:30:58.735Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "QA role pipelines (bunker-qa.sh driven by examples/coding-hermes/qa.ts) can report Passed: N | Failed: 0 while auditing ZERO real repos, and no evidence artifact is left behind. Chain: (1) the target repo path handed to the battery belongs to a *-qa scheduler row whose workdir is an empty stand-in dir (e.g. ~/.hermes/stand-in/pm/<project>, 0 entries, no .git) instead of the base project workdir (~/<project>); (2) bunker-qa.sh's preflight `[ -d \"$repo/.git\" ] || { echo \"ERROR: $repo is not a git repo\" >&2; exit 1; }` sits ABOVE the `: > \"$EVIDENCE\"` truncation and above the write_fail_if_empty traps, so it exits before the evidence file is even created (run_battery reports evidence_path /tmp/bunker-qa-evidence-*.jsonl that does not exist on disk, and no FAIL cell row is written); (3) parse_cells then returns {cells:[],cell_count:0,head:\"?\"} while the node reports status ok, so the tick's verdict is FINDINGS with findings=0 filed=0 and the pipeline closes green. Discovery keeps re-picking the same dead rows because qa_discover.py excludes -pm/-sync/dogfood suffix rows but not -qa, and it emits the row's raw workdir instead of the base project workdir; the QA ledger (~/.hermes/qa/ledger.jsonl) has zero -qa rows, so they never age out and sort first as never-qa every round. Fix direction: create the evidence file (and the fail-if-empty trap) BEFORE the git-repo preflight so a dead path yields a FAIL cell + artifact; make qa_discover.py resolve a -qa row's workdir to its BASE project row's workdir (or skip rows whose workdir has no .git); ledger every zero-coverage run so discovery can age it. Verification: repro on 2026-09-14 shows run_battery raw=\"ERROR: <dir> is not a git repo\", evidence file absent, parse_cells cells=[] cell_count=0, pipeline 'Passed: 4 | Failed: 0' with 0 real repos audited.", "environment": "", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "qa-harness-preflight-failure-hides-zero-coverage", "provider": "openrouter", "solved_at": "2026-09-14T13:30:58.736Z", "version": ""}