foreman-probe-secret-leak-false-positive
Diagnosis and verified fix written to ~/foreman-probe-secret-leak-false-positive.md, backed by a runnable reproduction at /tmp/probe-repro.
The probe asked "does refusal text contain raw?" instead of "was raw echoed?":
- raw="env:" collides with the CFG-16 grammar template env:<ENV_VAR_NAME> that every refusal legitimately renders.
- The same collision recurs in the json.Marshal'd §D.1 record because </> become \u003c/\u003e.
- raw="" matches every string.
- Probe precision: it required all subtests red (unrelated ones stay green under a single-enforcement mutation), and it assumed the wrong helper arity.
stripGrammar() removes each template in both plain and jsonEscaped spellings; apply it to the refusal and the marshalled record before strings.Contains.raw == "".^--- FAIL: lines so only top-level tests are asserted red.AuthStartupRefusal() error via go doc before writing err := assertions.go vet + go test -v in /tmp/probe-repro all pass: the old check is shown to false-positive on env:/empty/escaped record, the fixed check rejects those while still catching a genuine echo (env:REAL_SECRET), the escaped template is stripped, and the top-level-only failure parser ignores unrelated tests and subtests.
--- PASS: TestBugReproducesFalsePositive
--- PASS: TestFixedCheckNoFalsePositive
--- PASS: TestFixedCheckStillCatchesRealEcho
--- PASS: TestEscapedGrammarIsStripped
--- PASS: TestTopLevelFailuresOnly
ok probe 0.002s
The document includes the exact Go code, the parser, the arity command, the verification table, and a completion checklist.
# Evidence - Problem class: foreman-probe-secret-leak-false-positive - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T13:49:30.343Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Foreman verification probe for a secret_ref leak check (DAGGER-0189, <project> tick #503): asserting that a refusal never echoes the supplied value gives a FALSE POSITIVE for scheme-prefix inputs. Passing raw=\"env:\" makes strings.Contains(err.Error(), raw) match the CFG-16 GRAMMAR TEMPLATE that every refusal legitimately renders (\"env:<ENV_VAR_NAME>\" | \"file:<absolute-path>\" | \"keyring:<entry-id>\"), and the same holds for the projected \u00a7D.1 record bytes (json.Marshal escapes < > as \\u003c). Also, raw=\"\" matches everything. Fix: strip the grammar template (both the plain and the \\u003c-escaped spellings) from the refusal text and the marshalled record before the containment check, and skip the check for an empty raw. Two further probe-precision traps in the same run: assert the TOP-LEVEL test names go RED, not every subtest (unrelated subtests legitimately stay green under a single-enforcement mutation), and check helper symbol arity (AuthStartupRefusal returns only error) before writing probe assertions.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "foreman-probe-secret-leak-false-positive", "provider": "openrouter", "solved_at": "2026-09-17T13:49:30.343Z", "version": ""}Diagnosis and verified fix written to ~/foreman-probe-secret-leak-false-positive.md, backed by a runnable reproduction at /tmp/probe-repro.
The probe asked "does refusal text contain raw?" instead of "was raw echoed?":
- raw="env:" collides with the CFG-16 grammar template env:<ENV_VAR_NAME> that every refusal legitimately renders.
- The same collision recurs in the json.Marshal'd §D.1 record because </> become \u003c/\u003e.
- raw="" matches every string.
- Probe precision: it required all subtests red (unrelated ones stay green under a single-enforcement mutation), and it assumed the wrong helper arity.
stripGrammar() removes each template in both plain and jsonEscaped spellings; apply it to the refusal and the marshalled record before strings.Contains.raw == "".^--- FAIL: lines so only top-level tests are asserted red.AuthStartupRefusal() error via go doc before writing err := assertions.go vet + go test -v in /tmp/probe-repro all pass: the old check is shown to false-positive on env:/empty/escaped record, the fixed check rejects those while still catching a genuine echo (env:REAL_SECRET), the escaped template is stripped, and the top-level-only failure parser ignores unrelated tests and subtests.
--- PASS: TestBugReproducesFalsePositive
--- PASS: TestFixedCheckNoFalsePositive
--- PASS: TestFixedCheckStillCatchesRealEcho
--- PASS: TestEscapedGrammarIsStripped
--- PASS: TestTopLevelFailuresOnly
ok probe 0.002s
The document includes the exact Go code, the parser, the arity command, the verification table, and a completion checklist.
# Evidence - Problem class: foreman-probe-secret-leak-false-positive - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T13:49:30.343Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Foreman verification probe for a secret_ref leak check (DAGGER-0189, <project> tick #503): asserting that a refusal never echoes the supplied value gives a FALSE POSITIVE for scheme-prefix inputs. Passing raw=\"env:\" makes strings.Contains(err.Error(), raw) match the CFG-16 GRAMMAR TEMPLATE that every refusal legitimately renders (\"env:<ENV_VAR_NAME>\" | \"file:<absolute-path>\" | \"keyring:<entry-id>\"), and the same holds for the projected \u00a7D.1 record bytes (json.Marshal escapes < > as \\u003c). Also, raw=\"\" matches everything. Fix: strip the grammar template (both the plain and the \\u003c-escaped spellings) from the refusal text and the marshalled record before the containment check, and skip the check for an empty raw. Two further probe-precision traps in the same run: assert the TOP-LEVEL test names go RED, not every subtest (unrelated subtests legitimately stay green under a single-enforcement mutation), and check helper symbol arity (AuthStartupRefusal returns only error) before writing probe assertions.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "foreman-probe-secret-leak-false-positive", "provider": "openrouter", "solved_at": "2026-09-17T13:49:30.343Z", "version": ""}