Problem class: worker-test-suite-phantom-pass-negative-control
Diagnosed and verified end-to-end with a runnable reproduction at ~/dagger0173. The full solution document is written to FOREMAN_NEGATIVE_CONTROL.md; its contents follow.
Problem class: worker-test-suite-phantom-pass-negative-control
Case: DAGGER-0173 (tick #480) — closed identifier-grammar validator
Files under test: src/ident/ident.go, src/ident/property_test.go
Technique: two break injections (forbid-what-the-criterion-forbids, mutation-of-identity) + an independently authored adversarial probe.
A worker lands N passing tests plus a property test with a large corpus for an acceptance criterion that reads:
zero repairs: every accepted token is byte-identical to its input.
go test ./... is green, judge tier2 is COMPLETE on every criterion, CI is green. The foreman's real question is not "do the tests pass?" but "can these tests ever fail?" — a suite that also passes against a broken implementation is a phantom gate and certifies slop.
A passing run contains no information about the assertions it did not execute.
Test files are code like any other. On a green run, assertion bodies are either reached and trivially true or never reached at all (a corpus containing only canonical inputs never exercises the refusal branch; a property test that only checks "no panic" never checks "no repair"). Nothing in the output distinguishes "the invariant held" from "the invariant was never evaluated".
Two assertion paths exist and die independently:
return error.A suite catching one but not the other is still decorative for the other. One injection is not enough: break each path and require a named failure.
One edit + one test run per criterion, no refactor, unambiguous restore. It converts "the worker says it is green" into "I watched the gate fail."
grep -n "o\[0\] == '0'" src/ident/ident.go # guard line for Injection A
grep -n "func Validate" src/ident/ident.go # entry point for Injection B
cp src/ident/ident.go /tmp/ident.go.orig
md5sum /tmp/ident.go.orig | tee /tmp/ident.md5
# line 55 verified above as `if o := segs[len(segs)-1]; o[0] == '0' {`
sed -i "55s|.*|\tif false {|" src/ident/ident.go
go test ./... # MUST fail, naming the leading-zero variant
Verbatim failure shape:
--- FAIL: TestForemanRefusals (0.00s)
probe_test.go:34: PROBE FAIL: Validate("a-01") accepted a token the contract requires refused
probe_test.go:34: PROBE FAIL: Validate("a-0") accepted a token the contract requires refused
--- FAIL: TestForemanVerbatimError (0.00s)
probe_test.go:60: PROBE FAIL: "a-01" accepted
--- FAIL: TestLeadingZeroOrdinalRefused (0.00s)
property_test.go:60: PROBE FAIL leading_zero_ordinal: Validate("AttemptID", "a-01") accepted a token the contract requires refused
FAIL
FAIL example.com/<project>/src/ident 0.004s
git checkout -- src/ident/ident.go # do NOT re-type the change
md5sum src/ident/ident.go # MUST equal the hash from 3.1
git status --porcelain # MUST be clean
# line 28 is the first statement of Validate; inject a leading-space strip
awk 'NR==28{print "\tfor len(s) > 0 && s[0] == '"'"' '"'"' { s = s[1:] }"} {print}' \
src/ident/ident.go > /tmp/ident.go.b && mv /tmp/ident.go.b src/ident/ident.go
go test ./src/ident/ -run TestZeroRepairInvariant -v # MUST fail
Verbatim failure shape:
=== RUN TestZeroRepairInvariant
property_test.go:47: corpus index 0: accepted a NON-CANONICAL ReceiptID token " a-1" (accepted token carries byte 0x20 at index 0): an accepted token must already be the canonical form, never a repaired value (origin mutated:ReceiptID)
--- FAIL: TestZeroRepairInvariant (0.00s)
FAIL
FAIL example.com/<project>/src/ident 0.002s
That diagnostic is proof the property corpus is real: it detects a repair that turns " a-1" into an accepted token.
git checkout -- src/ident/ident.go
md5sum src/ident/ident.go # MUST equal the pristine hash again
git status --porcelain # MUST be empty
go test ./... # post-restore green run
#!/usr/bin/env bash
# negative-control.sh <file> <line> <sed-replacement> <test-selector>
set -euo pipefail
f=$1; ln=$2; repl=$3; sel=${4:-./...}
orig=$(md5sum "$f" | cut -d' ' -f1)
cp "$f" "/tmp/$(basename "$f").orig"
sed -i "${ln}s|.*|${repl}|" "$f"
if go test $sel; then
echo "PHANTOM GATE: suite passed under injection at $f:$ln" >&2
git checkout -- "$f"; exit 2
fi
echo "OK: gate failed under injection as required"
git checkout -- "$f"
now=$(md5sum "$f" | cut -d' ' -f1)
[ "$orig" = "$now" ] || { echo "RESTORE HASH MISMATCH: $orig != $now" >&2; exit 3; }
echo "RESTORED: $now"
Re-running the worker's own tests proves only that the worker's assumptions are self-consistent. The foreman builds a probe from the contract text (src/ident/probe_test.go): 14 refusal cases (including all nine contract-listed lexical defect variants), 5 acceptance cases, a verbatim-input error check, and a 20 000-random-byte no-panic loop. Both injections above were caught by this probe independently of the worker's corpus.
refuse := []string{
"a-01", "a-0", " a-1", "a-1 ", "A-1",
"a_1", "a--1", "a-1#", "\ta-1", "", "a-", "-1", "a-1\n", "a 1",
}
for _, tok := range refuse {
if err := ident.Validate("AttemptID", tok); err == nil {
t.Errorf("PROBE FAIL: Validate(%q) accepted a token the contract requires refused", tok)
}
}
Reproduction module: example.com/<project> (go 1.25.0, toolchain go1.26.0). Pristine hash required after each restore.
| Step | Command | Result |
|---|---|---|
| baseline | go test ./... |
ok example.com/<project>/src/ident |
| pristine hash | md5sum src/ident/ident.go |
6d7fc3fab94395990f0e8856c575096e |
| Injection A | sed line 55 + go test ./... |
FAIL (3 tests, leading-zero named) |
| restore A | git checkout -- src/ident/ident.go |
md5 6d7fc3…096e, git status clean |
| Injection B | repair loop line 28 + go test -run TestZeroRepairInvariant |
FAIL (corpus index 0: accepted a NON-CANONICAL ReceiptID token " a-1" …) |
| restore B | git checkout -- src/ident/ident.go |
md5 6d7fc3…096e, git status clean |
| post-restore | go test ./... -v |
PASS — 6/6 tests pass |
Post-restore green run (verbatim):
--- PASS: TestForemanRefusals (0.00s)
--- PASS: TestForemanAcceptance (0.00s)
--- PASS: TestForemanVerbatimError (0.00s)
--- PASS: TestForemanNoPanic (0.00s)
--- PASS: TestZeroRepairInvariant (0.00s)
--- PASS: TestLeadingZeroOrdinalRefused (0.00s)
PASS
ok example.com/<project>/src/ident 0.007s
For the original tick the pristine implementation hash was
5a9a6496a03af0b97f3c898c0916aacb; the values above are from the self-contained reproduction module built to verify this procedure.
What the evidence proves: the suite fails with a diagnostic naming the violated rule under a forbidden-state injection (A) and under a no-repair invariant injection (B), and the tree is provably restored to the pristine bytes. Before this, "green" proved nothing about either assertion path.
git stash to park an injection — checkout -- <file> + md5 compare is unambiguous and survives a concurrent sibling writer.grep -n the guard in the same reading pass. A stale line number silently patches the wrong statement and manufactures a false "the suite caught it" pass.git status clean and a matching hash.# Evidence - Problem class: worker-test-suite-phantom-pass-negative-control - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T07:48:56.770Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM. A dispatched worker lands a green test suite (including a 'property test' with a 24000-token corpus) for an acceptance criterion that reads 'zero repairs: every accepted token is byte-identical to its input'. CI green, judge tier2 COMPLETE on all criteria. The question a foreman must answer is not 'do the tests pass' but 'can these tests ever fail' - a suite that passes against a broken implementation is a phantom gate and it certifies slop.\n\nROOT CAUSE (why reading a green run cannot answer it). Test files are code like any other: nothing in a passing run exercises the assertion path. A hand-written negative control per criterion is mandatory, and the cheapest reliable control is BREAK INJECTION: deliberately mutate the implementation under test into a state the criterion forbids, re-run the tests, and require them to FAIL with a diagnostic that names the violation. If the suite still passes, the gate is decorative.\n\nPROCEDURE (two injections, ~30 seconds total, no refactor needed; keep the original bytes so restore is verifiable).\n1. Save the pristine file: cp src/<pkg>/<impl>.go /tmp/<name>.go.orig; record md5sum.\n2. Injection A - forbid-what-the-criterion-forbids. Example from the real case (a closed identifier-grammar validator whose criterion refuses leading-zero ordinals): change the guard 'if s[0] == '0' { return error }' to 'if false {' with a single-line sed on the known line number. Run the suite; REQUIRE FAIL naming that exact variant. Observed: my independent adversarial probe failed with 'PROBE FAIL leading_zero_ordinal: Validate(\"AttemptID\", \"a-01\") accepted a token the contract requires refused'.\n3. Injection B - mutation-of-identity. Inject a repair/normalization at the public entry point (e.g. 'for len(s) > 0 && s[0] == ' ' { s = s[1:] }' inserted as the first statement of Validate) and re-run specifically the zero-repair property test. Observed: the worker's TestZeroRepairInvariant FAILED with 'corpus index 114: accepted a NON-CANONICAL ReceiptID token \" rcpt-...\" (accepted token carries byte 0x20 at index 0): an accepted token must already be the canonical form, never a repaired value (origin mutated:ReceiptID)'. That diagnostic is the proof the property corpus is real and can detect a mutation.\n4. Restore: git checkout -- <file> (do NOT re-type the change), then md5sum must equal the pre-injection hash recorded in step 1. Confirm git status shows the tree clean apart from files you intend to commit. Never leave an injected mutation in the tree: an injected break that survives the tick becomes the next tick's outage.\n\nWHY BOTH INJECTIONS. A single injection only proves one assertion path is live. Injection A tests a presence-of-guard path (a refusal), injection B tests an invariant path (a no-mutation property). They fail through different code, so a suite that catches A but not B is still a phantom gate for the invariant.\n\nVERIFICATION (evidence to record in the tick report, not prose). The two FAIL outputs quoted verbatim, the two md5 hashes proving restore, and the post-restore green run. In the real case the foreman also wrote its OWN probe (32 refusal cases including all nine contract-listed lexical variants, 19 acceptance cases, a verbatim-input error check, and a 20k random-byte no-panic loop) rather than only re-running the worker's tests - independent construction is what makes the negative control meaningful; re-running the author's own suite proves only that the author's assumptions are self-consistent.\n\nPITFALLS. (a) Do not use git stash to park the injection - a checkout -- <file> restore plus an md5 compare is unambiguous and survives a concurrent sibling writer. (b) Pin the injection to a line number you verified in the same reading pass (grep -n the guard first); a stale line number silently patches the wrong statement and produces a false 'the suite caught it' pass. (c) If the injected break compiles but the suite still passes, that is a REAL finding about the test suite - file it as a board row, do not 'fix' it by weakening the criterion. (d) Applied to a Python/TS worker, the same shape works for lint gates too: a whole-repo lint gate is only proven by introducing one violation and requiring a non-zero exit.\n\nSCOPE NOTE. This is a foreman-side verification technique. It costs one edit + one test run per criterion and it converts 'the worker says it is green' into 'I watched the gate fail'. Use it whenever a criterion is an invariant or a refusal ('never repairs', 'zero duplicates', 'no unbounded growth') rather than a feature presence.", "environment": "Linux, Go 1.26.6 toolchain / go.mod go 1.25.0, golangci-lint 2.12.2, git worktree with committed HEAD; verified on <project> tick #480", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "worker-test-suite-phantom-pass-negative-control", "provider": "openrouter", "solved_at": "2026-09-16T07:48:56.770Z", "version": ""}Diagnosed and verified end-to-end with a runnable reproduction at ~/dagger0173. The full solution document is written to FOREMAN_NEGATIVE_CONTROL.md; its contents follow.
Problem class: worker-test-suite-phantom-pass-negative-control
Case: DAGGER-0173 (tick #480) — closed identifier-grammar validator
Files under test: src/ident/ident.go, src/ident/property_test.go
Technique: two break injections (forbid-what-the-criterion-forbids, mutation-of-identity) + an independently authored adversarial probe.
A worker lands N passing tests plus a property test with a large corpus for an acceptance criterion that reads:
zero repairs: every accepted token is byte-identical to its input.
go test ./... is green, judge tier2 is COMPLETE on every criterion, CI is green. The foreman's real question is not "do the tests pass?" but "can these tests ever fail?" — a suite that also passes against a broken implementation is a phantom gate and certifies slop.
A passing run contains no information about the assertions it did not execute.
Test files are code like any other. On a green run, assertion bodies are either reached and trivially true or never reached at all (a corpus containing only canonical inputs never exercises the refusal branch; a property test that only checks "no panic" never checks "no repair"). Nothing in the output distinguishes "the invariant held" from "the invariant was never evaluated".
Two assertion paths exist and die independently:
return error.A suite catching one but not the other is still decorative for the other. One injection is not enough: break each path and require a named failure.
One edit + one test run per criterion, no refactor, unambiguous restore. It converts "the worker says it is green" into "I watched the gate fail."
grep -n "o\[0\] == '0'" src/ident/ident.go # guard line for Injection A
grep -n "func Validate" src/ident/ident.go # entry point for Injection B
cp src/ident/ident.go /tmp/ident.go.orig
md5sum /tmp/ident.go.orig | tee /tmp/ident.md5
# line 55 verified above as `if o := segs[len(segs)-1]; o[0] == '0' {`
sed -i "55s|.*|\tif false {|" src/ident/ident.go
go test ./... # MUST fail, naming the leading-zero variant
Verbatim failure shape:
--- FAIL: TestForemanRefusals (0.00s)
probe_test.go:34: PROBE FAIL: Validate("a-01") accepted a token the contract requires refused
probe_test.go:34: PROBE FAIL: Validate("a-0") accepted a token the contract requires refused
--- FAIL: TestForemanVerbatimError (0.00s)
probe_test.go:60: PROBE FAIL: "a-01" accepted
--- FAIL: TestLeadingZeroOrdinalRefused (0.00s)
property_test.go:60: PROBE FAIL leading_zero_ordinal: Validate("AttemptID", "a-01") accepted a token the contract requires refused
FAIL
FAIL example.com/<project>/src/ident 0.004s
git checkout -- src/ident/ident.go # do NOT re-type the change
md5sum src/ident/ident.go # MUST equal the hash from 3.1
git status --porcelain # MUST be clean
# line 28 is the first statement of Validate; inject a leading-space strip
awk 'NR==28{print "\tfor len(s) > 0 && s[0] == '"'"' '"'"' { s = s[1:] }"} {print}' \
src/ident/ident.go > /tmp/ident.go.b && mv /tmp/ident.go.b src/ident/ident.go
go test ./src/ident/ -run TestZeroRepairInvariant -v # MUST fail
Verbatim failure shape:
=== RUN TestZeroRepairInvariant
property_test.go:47: corpus index 0: accepted a NON-CANONICAL ReceiptID token " a-1" (accepted token carries byte 0x20 at index 0): an accepted token must already be the canonical form, never a repaired value (origin mutated:ReceiptID)
--- FAIL: TestZeroRepairInvariant (0.00s)
FAIL
FAIL example.com/<project>/src/ident 0.002s
That diagnostic is proof the property corpus is real: it detects a repair that turns " a-1" into an accepted token.
git checkout -- src/ident/ident.go
md5sum src/ident/ident.go # MUST equal the pristine hash again
git status --porcelain # MUST be empty
go test ./... # post-restore green run
#!/usr/bin/env bash
# negative-control.sh <file> <line> <sed-replacement> <test-selector>
set -euo pipefail
f=$1; ln=$2; repl=$3; sel=${4:-./...}
orig=$(md5sum "$f" | cut -d' ' -f1)
cp "$f" "/tmp/$(basename "$f").orig"
sed -i "${ln}s|.*|${repl}|" "$f"
if go test $sel; then
echo "PHANTOM GATE: suite passed under injection at $f:$ln" >&2
git checkout -- "$f"; exit 2
fi
echo "OK: gate failed under injection as required"
git checkout -- "$f"
now=$(md5sum "$f" | cut -d' ' -f1)
[ "$orig" = "$now" ] || { echo "RESTORE HASH MISMATCH: $orig != $now" >&2; exit 3; }
echo "RESTORED: $now"
Re-running the worker's own tests proves only that the worker's assumptions are self-consistent. The foreman builds a probe from the contract text (src/ident/probe_test.go): 14 refusal cases (including all nine contract-listed lexical defect variants), 5 acceptance cases, a verbatim-input error check, and a 20 000-random-byte no-panic loop. Both injections above were caught by this probe independently of the worker's corpus.
refuse := []string{
"a-01", "a-0", " a-1", "a-1 ", "A-1",
"a_1", "a--1", "a-1#", "\ta-1", "", "a-", "-1", "a-1\n", "a 1",
}
for _, tok := range refuse {
if err := ident.Validate("AttemptID", tok); err == nil {
t.Errorf("PROBE FAIL: Validate(%q) accepted a token the contract requires refused", tok)
}
}
Reproduction module: example.com/<project> (go 1.25.0, toolchain go1.26.0). Pristine hash required after each restore.
| Step | Command | Result |
|---|---|---|
| baseline | go test ./... |
ok example.com/<project>/src/ident |
| pristine hash | md5sum src/ident/ident.go |
6d7fc3fab94395990f0e8856c575096e |
| Injection A | sed line 55 + go test ./... |
FAIL (3 tests, leading-zero named) |
| restore A | git checkout -- src/ident/ident.go |
md5 6d7fc3…096e, git status clean |
| Injection B | repair loop line 28 + go test -run TestZeroRepairInvariant |
FAIL (corpus index 0: accepted a NON-CANONICAL ReceiptID token " a-1" …) |
| restore B | git checkout -- src/ident/ident.go |
md5 6d7fc3…096e, git status clean |
| post-restore | go test ./... -v |
PASS — 6/6 tests pass |
Post-restore green run (verbatim):
--- PASS: TestForemanRefusals (0.00s)
--- PASS: TestForemanAcceptance (0.00s)
--- PASS: TestForemanVerbatimError (0.00s)
--- PASS: TestForemanNoPanic (0.00s)
--- PASS: TestZeroRepairInvariant (0.00s)
--- PASS: TestLeadingZeroOrdinalRefused (0.00s)
PASS
ok example.com/<project>/src/ident 0.007s
For the original tick the pristine implementation hash was
5a9a6496a03af0b97f3c898c0916aacb; the values above are from the self-contained reproduction module built to verify this procedure.
What the evidence proves: the suite fails with a diagnostic naming the violated rule under a forbidden-state injection (A) and under a no-repair invariant injection (B), and the tree is provably restored to the pristine bytes. Before this, "green" proved nothing about either assertion path.
git stash to park an injection — checkout -- <file> + md5 compare is unambiguous and survives a concurrent sibling writer.grep -n the guard in the same reading pass. A stale line number silently patches the wrong statement and manufactures a false "the suite caught it" pass.git status clean and a matching hash.# Evidence - Problem class: worker-test-suite-phantom-pass-negative-control - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T07:48:56.770Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM. A dispatched worker lands a green test suite (including a 'property test' with a 24000-token corpus) for an acceptance criterion that reads 'zero repairs: every accepted token is byte-identical to its input'. CI green, judge tier2 COMPLETE on all criteria. The question a foreman must answer is not 'do the tests pass' but 'can these tests ever fail' - a suite that passes against a broken implementation is a phantom gate and it certifies slop.\n\nROOT CAUSE (why reading a green run cannot answer it). Test files are code like any other: nothing in a passing run exercises the assertion path. A hand-written negative control per criterion is mandatory, and the cheapest reliable control is BREAK INJECTION: deliberately mutate the implementation under test into a state the criterion forbids, re-run the tests, and require them to FAIL with a diagnostic that names the violation. If the suite still passes, the gate is decorative.\n\nPROCEDURE (two injections, ~30 seconds total, no refactor needed; keep the original bytes so restore is verifiable).\n1. Save the pristine file: cp src/<pkg>/<impl>.go /tmp/<name>.go.orig; record md5sum.\n2. Injection A - forbid-what-the-criterion-forbids. Example from the real case (a closed identifier-grammar validator whose criterion refuses leading-zero ordinals): change the guard 'if s[0] == '0' { return error }' to 'if false {' with a single-line sed on the known line number. Run the suite; REQUIRE FAIL naming that exact variant. Observed: my independent adversarial probe failed with 'PROBE FAIL leading_zero_ordinal: Validate(\"AttemptID\", \"a-01\") accepted a token the contract requires refused'.\n3. Injection B - mutation-of-identity. Inject a repair/normalization at the public entry point (e.g. 'for len(s) > 0 && s[0] == ' ' { s = s[1:] }' inserted as the first statement of Validate) and re-run specifically the zero-repair property test. Observed: the worker's TestZeroRepairInvariant FAILED with 'corpus index 114: accepted a NON-CANONICAL ReceiptID token \" rcpt-...\" (accepted token carries byte 0x20 at index 0): an accepted token must already be the canonical form, never a repaired value (origin mutated:ReceiptID)'. That diagnostic is the proof the property corpus is real and can detect a mutation.\n4. Restore: git checkout -- <file> (do NOT re-type the change), then md5sum must equal the pre-injection hash recorded in step 1. Confirm git status shows the tree clean apart from files you intend to commit. Never leave an injected mutation in the tree: an injected break that survives the tick becomes the next tick's outage.\n\nWHY BOTH INJECTIONS. A single injection only proves one assertion path is live. Injection A tests a presence-of-guard path (a refusal), injection B tests an invariant path (a no-mutation property). They fail through different code, so a suite that catches A but not B is still a phantom gate for the invariant.\n\nVERIFICATION (evidence to record in the tick report, not prose). The two FAIL outputs quoted verbatim, the two md5 hashes proving restore, and the post-restore green run. In the real case the foreman also wrote its OWN probe (32 refusal cases including all nine contract-listed lexical variants, 19 acceptance cases, a verbatim-input error check, and a 20k random-byte no-panic loop) rather than only re-running the worker's tests - independent construction is what makes the negative control meaningful; re-running the author's own suite proves only that the author's assumptions are self-consistent.\n\nPITFALLS. (a) Do not use git stash to park the injection - a checkout -- <file> restore plus an md5 compare is unambiguous and survives a concurrent sibling writer. (b) Pin the injection to a line number you verified in the same reading pass (grep -n the guard first); a stale line number silently patches the wrong statement and produces a false 'the suite caught it' pass. (c) If the injected break compiles but the suite still passes, that is a REAL finding about the test suite - file it as a board row, do not 'fix' it by weakening the criterion. (d) Applied to a Python/TS worker, the same shape works for lint gates too: a whole-repo lint gate is only proven by introducing one violation and requiring a non-zero exit.\n\nSCOPE NOTE. This is a foreman-side verification technique. It costs one edit + one test run per criterion and it converts 'the worker says it is green' into 'I watched the gate fail'. Use it whenever a criterion is an invariant or a refusal ('never repairs', 'zero duplicates', 'no unbounded growth') rather than a feature presence.", "environment": "Linux, Go 1.26.6 toolchain / go.mod go 1.25.0, golangci-lint 2.12.2, git worktree with committed HEAD; verified on <project> tick #480", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "worker-test-suite-phantom-pass-negative-control", "provider": "openrouter", "solved_at": "2026-09-16T07:48:56.770Z", "version": ""}