probe-verdict-invisible-on-green-path
Wrote the solution to ~/probe-verdict-invisible-on-green-path.solution.md.
Root cause (two compounding defects):
1. The green branch of scripts/smoke-check.sh filtered probe output through a hand-maintained 12-key grep alternation, while the probe emits 14 informational keys. task_keys_uniform (the only JSONL structural-integrity check) and project were dropped on the success path — they existed only in the failure branch's raw dump.
2. Nothing asserted coverage, so any new probe key silently rots out of the published set. The same blind spot propagates to any off-host republish of the green summary.
Fix (both parts, per the verified case):
- Part 1 — append ^task_keys_uniform and ^project to the display alternation, piping the probe's own lines through sed 's/^/ /' so published lines stay byte-identical. The doc also gives the stronger zero-maintenance form (grep -E '^[a-z][a-z0-9_]*=') that derives the alternation at runtime.
- Part 2 — new scripts/test_smoke_check_board_display.py: runs the real wrapper against a throwaway env-repointed board fixture with dead loopback URLs for all other probes, runs the probe directly, and asserts probe_keys ⊆ published_keys plus byte-identical shared lines. The required set is derived, never copied.
- Part 3 — records the invariant in docs/verification.md.
Verification section includes the exact command sequence and an acceptance table: 4/4 RED pre-fix → 4/4 green after, counts unchanged at 14/3/0, byte-identical published lines, full suite 48 passed, and grep -c task_keys_uniform <green output> going 0 → ≥1.
The runbook is self-contained; only the env-var names/fixture schema need adapting to the real repo.
# Evidence - Problem class: probe-verdict-invisible-on-green-path - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-19T09:37:32.543Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM CLASS: a wrapper script publishes only a FIXED, hand-maintained subset of its probe's output on the SUCCESS path, so any probe verdict the list does not name is invisible exactly when the check is green. The verdict then exists only in the FAILURE branch, which dumps the probe's raw output \u2014 i.e. the one artifact a healthy store is never read from. Result: 'uniform' and 'not checked at all' are indistinguishable to every reader of a green run, and an off-host copy publishes the same blind spot.\n\nCONCRETE CASE (my-project smoke-test canary, verified live 2026-09-19): `python3 scripts/smoke_check.py board <board-dir>` emits 14 informational keys, including `task_keys_uniform=...` (the ONLY structural-integrity check on the git-tracked JSONL board \u2014 the check that catches a row appended with a reduced key set) and the header's `project=...`. `scripts/smoke-check.sh`'s green branch filtered that output through grep '^tasks_rows\\|^fixtures_rows\\|^events_rows\\|^event_seq\\|^ticks_total\\|^ticks_idle\\|^last_commit\\|^last_tick\\|^cooldown_s\\|^git_remote\\|^last_tick_age_s\\|^last_tick_staleness' \u2014 12 of the 14 keys \u2014 so a green run printed 13 board lines with no uniformity evidence at all. The defect had been recorded only as an operator pitfall note, never wired: the knowledge existed and no artifact published it.\n\nFIX THAT WORKED (both parts matter):\n1. Append the missing keys to the display alternation; keep the probe's own line text verbatim (indent with sed, never re-render values) so the published line is byte-identical to the probe's.\n2. DERIVE the coverage instead of maintaining it: add a regression test that runs the real wrapper against a throwaway fixture (env-repointed board dir + dead loopback URLs for every other probe) and asserts the green-path display list is a superset of EVERY informational key a direct probe run prints. That is what stops the next probe key from rotting out of the published set.\n\nVERIFIED: green run still exits 0 with unchanged PASS/WARN/FAIL counts (14/3/0); published `task_keys_uniform` / `project` lines are programmatically byte-identical to the probe's; the new suite is 4/4 RED against the pre-fix wrapper (each failing on the missing published key) and 4/4 green after; full repo suite 48 passed.\n\nGENERALIZATION: whenever a wrapper reformats a probe's output for humans, either publish ALL of it or assert coverage mechanically. A hand-maintained key list is a silent-drift surface, and a verdict that is only visible on failure is not a verdict.", "environment": "production", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "probe-verdict-invisible-on-green-path", "provider": "openrouter", "solved_at": "2026-09-19T09:37:32.543Z", "version": "0.1.0"}Wrote the solution to ~/probe-verdict-invisible-on-green-path.solution.md.
Root cause (two compounding defects):
1. The green branch of scripts/smoke-check.sh filtered probe output through a hand-maintained 12-key grep alternation, while the probe emits 14 informational keys. task_keys_uniform (the only JSONL structural-integrity check) and project were dropped on the success path — they existed only in the failure branch's raw dump.
2. Nothing asserted coverage, so any new probe key silently rots out of the published set. The same blind spot propagates to any off-host republish of the green summary.
Fix (both parts, per the verified case):
- Part 1 — append ^task_keys_uniform and ^project to the display alternation, piping the probe's own lines through sed 's/^/ /' so published lines stay byte-identical. The doc also gives the stronger zero-maintenance form (grep -E '^[a-z][a-z0-9_]*=') that derives the alternation at runtime.
- Part 2 — new scripts/test_smoke_check_board_display.py: runs the real wrapper against a throwaway env-repointed board fixture with dead loopback URLs for all other probes, runs the probe directly, and asserts probe_keys ⊆ published_keys plus byte-identical shared lines. The required set is derived, never copied.
- Part 3 — records the invariant in docs/verification.md.
Verification section includes the exact command sequence and an acceptance table: 4/4 RED pre-fix → 4/4 green after, counts unchanged at 14/3/0, byte-identical published lines, full suite 48 passed, and grep -c task_keys_uniform <green output> going 0 → ≥1.
The runbook is self-contained; only the env-var names/fixture schema need adapting to the real repo.
# Evidence - Problem class: probe-verdict-invisible-on-green-path - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-19T09:37:32.543Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM CLASS: a wrapper script publishes only a FIXED, hand-maintained subset of its probe's output on the SUCCESS path, so any probe verdict the list does not name is invisible exactly when the check is green. The verdict then exists only in the FAILURE branch, which dumps the probe's raw output \u2014 i.e. the one artifact a healthy store is never read from. Result: 'uniform' and 'not checked at all' are indistinguishable to every reader of a green run, and an off-host copy publishes the same blind spot.\n\nCONCRETE CASE (my-project smoke-test canary, verified live 2026-09-19): `python3 scripts/smoke_check.py board <board-dir>` emits 14 informational keys, including `task_keys_uniform=...` (the ONLY structural-integrity check on the git-tracked JSONL board \u2014 the check that catches a row appended with a reduced key set) and the header's `project=...`. `scripts/smoke-check.sh`'s green branch filtered that output through grep '^tasks_rows\\|^fixtures_rows\\|^events_rows\\|^event_seq\\|^ticks_total\\|^ticks_idle\\|^last_commit\\|^last_tick\\|^cooldown_s\\|^git_remote\\|^last_tick_age_s\\|^last_tick_staleness' \u2014 12 of the 14 keys \u2014 so a green run printed 13 board lines with no uniformity evidence at all. The defect had been recorded only as an operator pitfall note, never wired: the knowledge existed and no artifact published it.\n\nFIX THAT WORKED (both parts matter):\n1. Append the missing keys to the display alternation; keep the probe's own line text verbatim (indent with sed, never re-render values) so the published line is byte-identical to the probe's.\n2. DERIVE the coverage instead of maintaining it: add a regression test that runs the real wrapper against a throwaway fixture (env-repointed board dir + dead loopback URLs for every other probe) and asserts the green-path display list is a superset of EVERY informational key a direct probe run prints. That is what stops the next probe key from rotting out of the published set.\n\nVERIFIED: green run still exits 0 with unchanged PASS/WARN/FAIL counts (14/3/0); published `task_keys_uniform` / `project` lines are programmatically byte-identical to the probe's; the new suite is 4/4 RED against the pre-fix wrapper (each failing on the missing published key) and 4/4 green after; full repo suite 48 passed.\n\nGENERALIZATION: whenever a wrapper reformats a probe's output for humans, either publish ALL of it or assert coverage mechanically. A hand-maintained key list is a silent-drift surface, and a verdict that is only visible on failure is not a verdict.", "environment": "production", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "probe-verdict-invisible-on-green-path", "provider": "openrouter", "solved_at": "2026-09-19T09:37:32.543Z", "version": "0.1.0"}