e2e-battery-certifies-stale-binary

# Evidence
- Problem class: e2e-battery-certifies-stale-binary
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-16T05:58:54.976Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: a project's documented end-to-end verification battery prints its success banner (VERIFY-PASS / ALL CORE TESTS PASS) while exercising a DEPLOYED binary that is weeks behind the repo. The claim 'the repo was verified' is then false, and nothing in the transcript shows it. Concrete instance (bunker, 2026-09-16): e2e-full-battery.sh resolved the CLI as BUNKER=\"${BUNKER_BIN:-/usr/local/bin/bunker}\"; the host's installed binary reported commit 4af949d (built 2026-08-27) while repo HEAD was 33392a4, so `sudo bash e2e-full-battery.sh` certified a 3-week-old build. Evidence is two commands: `/usr/local/bin/bunker version` (line 2 = 'commit: <sha>') vs `git rev-parse --short HEAD`.\n\nROOT CAUSE: the harness declares only WHICH PATH it runs, never WHICH COMMIT that path reports, and it never compares that commit to the repo HEAD. A default-to-deployed-binary fallback (legitimate for deploy verification) therefore silently becomes the certification subject. Secondary cause: a dead duplicate assignment (`BUNKER=\"/usr/local/bin/bunker\"` followed later by the env-override form) makes readers believe the override does not exist, so the defect was filed and re-filed as 'hardcoded path' even after BUNKER_BIN was wired.\n\nFIX (additive, ~200 lines of shell): (1) one resolution statement only: BIN=\"${BIN_ENV_OVERRIDE:-/usr/local/bin/<tool>}\". (2) A PURE comparator comparing the reported commit to repo HEAD, tolerant of short-vs-full SHAs (prefix match, shorter side >= 7 chars), returning MATCH / MISMATCH / SKIP \u2014 SKIP when the reported commit is empty/unknown/dev or HEAD is unreadable (a SKIP must never fail a run). (3) A certification preflight BEFORE any host mutation that always prints a banner: binary path, reported commit, repo HEAD, verdict. MISMATCH is a loud NON-fatal note by default (certifying a deployed binary is legitimate); with STRICT_BIN=1 it is a hard failure that exits non-zero BEFORE the cleanup trap is armed, so nothing on the host is touched. (4) A zero-side-effect report mode (--bin-report) that prints only the banner and exits 0 on MATCH/SKIP, non-zero on MISMATCH \u2014 this is what makes the behavior independently testable on any box with stub CLIs. (5) The final RESULTS SUMMARY prints one CERTIFIED BINARY line naming path + commit + verdict, so a green transcript can no longer be read as 'HEAD was verified'. (6) CI sets STRICT_BIN=1 in the battery step, where the binary is built from the pushed commit, turning a mismatch into a job-level red.\n\nVERIFICATION USED: --bin-report exercised with foreman-built stub CLIs covering 8 cases (short SHA MATCH rc0, full SHA MATCH rc0, stale MISMATCH rc1, stale+strict MISMATCH rc1 naming both commits, HEAD+strict MATCH rc0, unknown SKIP rc0, empty SKIP rc0, deployed-default stale MISMATCH rc1); a fresh `go build` of HEAD yields MATCH rc0 under strict, proving the CI leg cannot turn red on the certification itself; the self-test grew 3 -> 9 assertions by covering the comparator's MATCH/MISMATCH/SKIP and <7-char-prefix cases with zero side effects.\n\nGENERALIZABLE LESSON: any harness that can certify an artifact must NAME the artifact AND prove it corresponds to the source revision (commit, version, or hash). Add the comparison as pure logic, expose a no-side-effect report mode for tests, and make the strict mode opt-in \u2014 strict-by-default in a shared harness just teaches operators to ignore red.", "environment": "Linux host with a deployed CLI plus a git worktree; the battery runs as root and mutates the host (users/systemd/PAM), so the certification must be verifiable without root via a stub-binary report mode", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "e2e-battery-certifies-stale-binary", "provider": "openrouter", "solved_at": "2026-09-16T05:58:54.976Z", "version": ""}

# Evidence
- Problem class: e2e-battery-certifies-stale-binary
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-16T05:58:54.976Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: a project's documented end-to-end verification battery prints its success banner (VERIFY-PASS / ALL CORE TESTS PASS) while exercising a DEPLOYED binary that is weeks behind the repo. The claim 'the repo was verified' is then false, and nothing in the transcript shows it. Concrete instance (bunker, 2026-09-16): e2e-full-battery.sh resolved the CLI as BUNKER=\"${BUNKER_BIN:-/usr/local/bin/bunker}\"; the host's installed binary reported commit 4af949d (built 2026-08-27) while repo HEAD was 33392a4, so `sudo bash e2e-full-battery.sh` certified a 3-week-old build. Evidence is two commands: `/usr/local/bin/bunker version` (line 2 = 'commit: <sha>') vs `git rev-parse --short HEAD`.\n\nROOT CAUSE: the harness declares only WHICH PATH it runs, never WHICH COMMIT that path reports, and it never compares that commit to the repo HEAD. A default-to-deployed-binary fallback (legitimate for deploy verification) therefore silently becomes the certification subject. Secondary cause: a dead duplicate assignment (`BUNKER=\"/usr/local/bin/bunker\"` followed later by the env-override form) makes readers believe the override does not exist, so the defect was filed and re-filed as 'hardcoded path' even after BUNKER_BIN was wired.\n\nFIX (additive, ~200 lines of shell): (1) one resolution statement only: BIN=\"${BIN_ENV_OVERRIDE:-/usr/local/bin/<tool>}\". (2) A PURE comparator comparing the reported commit to repo HEAD, tolerant of short-vs-full SHAs (prefix match, shorter side >= 7 chars), returning MATCH / MISMATCH / SKIP \u2014 SKIP when the reported commit is empty/unknown/dev or HEAD is unreadable (a SKIP must never fail a run). (3) A certification preflight BEFORE any host mutation that always prints a banner: binary path, reported commit, repo HEAD, verdict. MISMATCH is a loud NON-fatal note by default (certifying a deployed binary is legitimate); with STRICT_BIN=1 it is a hard failure that exits non-zero BEFORE the cleanup trap is armed, so nothing on the host is touched. (4) A zero-side-effect report mode (--bin-report) that prints only the banner and exits 0 on MATCH/SKIP, non-zero on MISMATCH \u2014 this is what makes the behavior independently testable on any box with stub CLIs. (5) The final RESULTS SUMMARY prints one CERTIFIED BINARY line naming path + commit + verdict, so a green transcript can no longer be read as 'HEAD was verified'. (6) CI sets STRICT_BIN=1 in the battery step, where the binary is built from the pushed commit, turning a mismatch into a job-level red.\n\nVERIFICATION USED: --bin-report exercised with foreman-built stub CLIs covering 8 cases (short SHA MATCH rc0, full SHA MATCH rc0, stale MISMATCH rc1, stale+strict MISMATCH rc1 naming both commits, HEAD+strict MATCH rc0, unknown SKIP rc0, empty SKIP rc0, deployed-default stale MISMATCH rc1); a fresh `go build` of HEAD yields MATCH rc0 under strict, proving the CI leg cannot turn red on the certification itself; the self-test grew 3 -> 9 assertions by covering the comparator's MATCH/MISMATCH/SKIP and <7-char-prefix cases with zero side effects.\n\nGENERALIZABLE LESSON: any harness that can certify an artifact must NAME the artifact AND prove it corresponds to the source revision (commit, version, or hash). Add the comparison as pure logic, expose a no-side-effect report mode for tests, and make the strict mode opt-in \u2014 strict-by-default in a shared harness just teaches operators to ignore red.", "environment": "Linux host with a deployed CLI plus a git worktree; the battery runs as root and mutates the host (users/systemd/PAM), so the certification must be verifiable without root via a stub-binary report mode", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "e2e-battery-certifies-stale-binary", "provider": "openrouter", "solved_at": "2026-09-16T05:58:54.976Z", "version": ""}