ci-cell-stale-green-keyless-fail-open-expectation
Diagnosis complete and verified. The solution is at ~/SOLUTION.md, with a runnable reproduction at ~/repro/envfile-parity-repro.sh.
Root cause: In scripts/bunker-matrix.sh, the guard-off (CR_GUARD_ENABLED=false) cell wrote its env file but its deploy call omitted the CR_ENV_FILE= prefix every sibling uses. The file landed in /tmp; the switch never reached the container, so the guard stayed enabled and keyless.
Why it had been green: A keyless guard used to fail open on the injection payload and return 201 — exactly what the cell asserted. A legitimate product fix made keyless guards block on the deterministic prematch (ignore_previous) instead of failing open, turning the accidental green into a correct 403 and exposing the stale expectation. The product fix was innocent.
Exact fix:
write_cell_env "$guard_off_env" "CR_GUARD_ENABLED=false"
- deploy "guard-off"
+ CR_ENV_FILE="$guard_off_env" deploy "guard-off"
plus a structural deploy_cell <name> <envfile> helper that requires the env file and asserts the value inside the running container, so a cell's name can never again disagree with its actual configuration.
Verification (run locally, both halves pass):
- GUARD_FAIL_OPEN=1 → buggy cell 201 = stale green (guard enabled but fail-open).
- GUARD_FAIL_OPEN=0 → buggy cell 403 = red (product fix exposes missing env file).
- Fixed call in both modes → 201; sibling guard-on → 403.
- CI on 7388762: bunker-e2e 7 pass / 0 fail (run 34961570431).
The document also includes the probe-body-shape guard (payload_bytes=0 / empty matched_patterns ⇒ check the JSON key first) and the three reusable rules.
# Evidence - Problem class: ci-cell-stale-green-keyless-fail-open-expectation - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-15T11:39:31.487Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Symptom: a CI config-matrix cell named 'guard-off (CR_GUARD_ENABLED=false)' failed WITHOUT any product regression, immediately after a legitimate fix landed: 'FAIL guard-off injection delivered (HTTP 403 want 201)' and the body showed the guard had run and blocked ('deterministic prematch block: ignore_previous'). Diagnosis path that found it fast: (1) read the failed step's log, not just the red conclusion; (2) notice the cell's own env file was written but the deploy call for that cell omitted the CR_ENV_FILE prefix that the sibling cells all use; (3) conclude the container never received the master switch, so the guard stayed ENABLED with no provider key and the cell only passed because a keyless guard failed open on the injection payload. So the cell was green BECAUSE of a defect, and the fix that removed the defect exposed the stale expectation. Fix: pass the env file on that cell's deploy call exactly like its siblings. Verification: reproduce both product-side halves locally with correct request shape (guard disabled -> 201 for the injection; guard enabled + keyless -> 403 with the pattern retained), then let CI re-run the matrix: 7 pass / 0 fail. Reusable rules: (a) an intentionally-red-turned-green test is the strongest signal that a test asserted a bug \u2014 grep the matrix for env-file/deploy parity before blaming the code change; (b) a cell's NAME is not evidence of its configuration: verify the variable actually reaches the process under test (env-file prefix present, container recreated, not just a file written to /tmp); (c) when a request-level probe returns an unexpected verdict with payload_bytes=0 or an empty pattern list, suspect the probe BODY SHAPE first (a wrong payload key silently yields an empty payload and therefore no deterministic match) before doubting the fix.", "environment": "GitHub Actions workflow deploying a Go server to a remote docker host, running a multi-cell config matrix (scripts/bunker-matrix.sh) where each cell writes a KEY=VALUE env file and re-deploys the container with CR_ENV_FILE=<file>; serialized workflow concurrency", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "ci-cell-stale-green-keyless-fail-open-expectation", "provider": "openrouter", "solved_at": "2026-09-15T11:39:31.488Z", "version": "<project> main 7388762"}Diagnosis complete and verified. The solution is at ~/SOLUTION.md, with a runnable reproduction at ~/repro/envfile-parity-repro.sh.
Root cause: In scripts/bunker-matrix.sh, the guard-off (CR_GUARD_ENABLED=false) cell wrote its env file but its deploy call omitted the CR_ENV_FILE= prefix every sibling uses. The file landed in /tmp; the switch never reached the container, so the guard stayed enabled and keyless.
Why it had been green: A keyless guard used to fail open on the injection payload and return 201 — exactly what the cell asserted. A legitimate product fix made keyless guards block on the deterministic prematch (ignore_previous) instead of failing open, turning the accidental green into a correct 403 and exposing the stale expectation. The product fix was innocent.
Exact fix:
write_cell_env "$guard_off_env" "CR_GUARD_ENABLED=false"
- deploy "guard-off"
+ CR_ENV_FILE="$guard_off_env" deploy "guard-off"
plus a structural deploy_cell <name> <envfile> helper that requires the env file and asserts the value inside the running container, so a cell's name can never again disagree with its actual configuration.
Verification (run locally, both halves pass):
- GUARD_FAIL_OPEN=1 → buggy cell 201 = stale green (guard enabled but fail-open).
- GUARD_FAIL_OPEN=0 → buggy cell 403 = red (product fix exposes missing env file).
- Fixed call in both modes → 201; sibling guard-on → 403.
- CI on 7388762: bunker-e2e 7 pass / 0 fail (run 34961570431).
The document also includes the probe-body-shape guard (payload_bytes=0 / empty matched_patterns ⇒ check the JSON key first) and the three reusable rules.
# Evidence - Problem class: ci-cell-stale-green-keyless-fail-open-expectation - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-15T11:39:31.487Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Symptom: a CI config-matrix cell named 'guard-off (CR_GUARD_ENABLED=false)' failed WITHOUT any product regression, immediately after a legitimate fix landed: 'FAIL guard-off injection delivered (HTTP 403 want 201)' and the body showed the guard had run and blocked ('deterministic prematch block: ignore_previous'). Diagnosis path that found it fast: (1) read the failed step's log, not just the red conclusion; (2) notice the cell's own env file was written but the deploy call for that cell omitted the CR_ENV_FILE prefix that the sibling cells all use; (3) conclude the container never received the master switch, so the guard stayed ENABLED with no provider key and the cell only passed because a keyless guard failed open on the injection payload. So the cell was green BECAUSE of a defect, and the fix that removed the defect exposed the stale expectation. Fix: pass the env file on that cell's deploy call exactly like its siblings. Verification: reproduce both product-side halves locally with correct request shape (guard disabled -> 201 for the injection; guard enabled + keyless -> 403 with the pattern retained), then let CI re-run the matrix: 7 pass / 0 fail. Reusable rules: (a) an intentionally-red-turned-green test is the strongest signal that a test asserted a bug \u2014 grep the matrix for env-file/deploy parity before blaming the code change; (b) a cell's NAME is not evidence of its configuration: verify the variable actually reaches the process under test (env-file prefix present, container recreated, not just a file written to /tmp); (c) when a request-level probe returns an unexpected verdict with payload_bytes=0 or an empty pattern list, suspect the probe BODY SHAPE first (a wrong payload key silently yields an empty payload and therefore no deterministic match) before doubting the fix.", "environment": "GitHub Actions workflow deploying a Go server to a remote docker host, running a multi-cell config matrix (scripts/bunker-matrix.sh) where each cell writes a KEY=VALUE env file and re-deploys the container with CR_ENV_FILE=<file>; serialized workflow concurrency", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "ci-cell-stale-green-keyless-fail-open-expectation", "provider": "openrouter", "solved_at": "2026-09-15T11:39:31.488Z", "version": "<project> main 7388762"}