Problem class: an integration test that accepts 202 OR 500 while injecting an executor hardcoded to fail proves only that dispatch was attempted — not that the skill ran. Because the event row is written before the runner call, a completed-looking row can coexist with a skill that never executed. The tolerance clause (|| 500) silently absorbs exactly the failure the test was meant to catch.
Verified end-to-end. The full solution is written to ~/solution.md, with a runnable reproduction at ~/tolerance-repro/. Below is the deliverable.
go-test-tolerance-clause-false-acceptanceProblem class: an integration test that accepts 202 OR 500 while injecting an executor hardcoded to fail proves only that dispatch was attempted — not that the skill ran. Because the event row is written before the runner call, a completed-looking row can coexist with a skill that never executed. The tolerance clause (|| 500) silently absorbs exactly the failure the test was meant to catch.
This document contains a self-contained reproduction, the root cause, the exact test fix, and a verification procedure that proves the old test is blind and the new test is not.
Three independent defects combine into a false acceptance:
The tolerance clause erases the signal. if status != 202 && status != 500 accepts the environment-unavailable escape hatch. When the injected executor is hardcoded to fail, the "happy path" can never be observed, so the assertion is tautologically satisfied by the failure branch. A 500 here is not a valid outcome — it is the bug.
The event row is written before the runner call. Dispatch does roughly:
go
ev := store.Insert(Event{Status: StatusRunning}) // row exists already
if err := exec.Run(skill, payload); err != nil { // runner may never run
store.UpdateStatus(ev.ID, StatusFailed)
return 500, ev.ID
}
store.UpdateStatus(ev.ID, StatusCompleted)
Any test that merely asserts a row exists passes whether or not the executor was invoked. Worse, a mutant that inserts the row as completed and returns 202 without calling the runner satisfies both 202 and "row exists".
Run was called, (b) with which skill source/payload, (c) whether the row reached the terminal completed state, or (d) whether the fire counter moved. The test cannot distinguish dispatch from execution.The fix is a recording executor swapped onto the live manager plus exact assertions: 202 only, executor invoked once with the exact source/payload, exactly one event row, exactly one reaching completed, and the fire counter incremented. The tolerant test is retained, but demoted to a dispatch-contract check.
tolerance-repro/
├── go.mod
└── skill/
├── manager.go # production manager + mutant
├── manager_test.go # tolerant (kept) + strict (new) tests
└── mutant_strict_test.go # build-tagged: strict body vs. the mutant
skill/manager.go (core, abridged):
type Executor interface {
Run(skill Skill, payload Payload) error
}
type Manager struct {
store *Store
exec Executor
mu sync.Mutex
fireCnt int64
}
// Dispatch writes the row, then invokes the runner, then transitions the row.
func (m *Manager) Dispatch(skill Skill, payload Payload) (status int, eventID string) {
if m.exec == nil {
return 500, "" // environment-unavailable escape hatch
}
ev := m.store.Insert(Event{SkillID: skill.ID, Source: skill.Source, Status: StatusRunning})
if err := m.exec.Run(skill, payload); err != nil {
m.store.UpdateStatus(ev.ID, StatusFailed)
return 500, ev.ID
}
m.store.UpdateStatus(ev.ID, StatusCompleted)
m.incFire()
return 202, ev.ID
}
// MutantDispatch models the bug the tolerant test cannot see: the row is written
// as "completed" BEFORE (in fact, without) any runner call.
func (m *Manager) MutantDispatch(skill Skill, payload Payload) (status int, eventID string) {
ev := m.store.Insert(Event{SkillID: skill.ID, Source: skill.Source, Status: StatusCompleted})
return 202, ev.ID
}
The original, broken test:
type alwaysFailExecutor struct{}
func (alwaysFailExecutor) Run(Skill, Payload) error { return errors.New("env unavailable") }
func TestDispatch_TolerantContract(t *testing.T) {
store := NewStore()
mgr := NewManager(store, alwaysFailExecutor{}) // executor hardcoded to fail
status, _ := mgr.Dispatch(Skill{ID: "s1", Source: "skills/foo.md"}, Payload{"msg": "hi"})
if status != 202 && status != 500 { // <-- false-acceptance clause
t.Fatalf("status = %d, want 202 or 500", status)
}
if len(store.All()) == 0 {
t.Fatal("expected an event row to be written") // true even with no execution
}
}
type call struct {
Skill Skill
Payload Payload
}
type recordingExecutor struct {
mu sync.Mutex
calls []call
err error
}
func (r *recordingExecutor) Run(s Skill, p Payload) error {
r.mu.Lock()
defer r.mu.Unlock()
r.calls = append(r.calls, call{Skill: s, Payload: p})
return r.err
}
func (r *recordingExecutor) Calls() []call {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]call, len(r.calls))
copy(out, r.calls)
return out
}
func assertStrict(t *testing.T, setup func(store *Store, rec *recordingExecutor) (*Manager, func(Skill, Payload) (int, string))) {
t.Helper()
store := NewStore()
rec := &recordingExecutor{}
mgr, dispatch := setup(store, rec)
skill := Skill{ID: "s1", Source: "skills/foo.md"}
payload := Payload{"msg": "hi"}
status, _ := dispatch(skill, payload)
// 1. Accepted only. The executor is wired to succeed, so 500 is a real failure.
if status != 202 {
t.Fatalf("status = %d, want 202", status)
}
// 2. Executor actually invoked, exactly once, with the exact source/payload.
calls := rec.Calls()
if len(calls) != 1 {
t.Fatalf("executor invocations = %d, want 1", len(calls))
}
if calls[0].Skill.Source != skill.Source {
t.Fatalf("executor skill source = %q, want %q", calls[0].Skill.Source, skill.Source)
}
if !reflect.DeepEqual(calls[0].Payload, payload) {
t.Fatalf("executor payload = %#v, want %#v", calls[0].Payload, payload)
}
// 3. Exactly one event row, and exactly one reached "completed".
events := store.All()
completed := 0
for _, e := range events {
if e.Status == StatusCompleted {
completed++
}
}
if len(events) != 1 {
t.Fatalf("event rows = %d, want exactly 1", len(events))
}
if completed != 1 {
t.Fatalf("completed rows = %d, want exactly 1", completed)
}
// 4. Fire counter incremented.
if got := mgr.FireCount(); got != 1 {
t.Fatalf("fire counter = %d, want 1", got)
}
}
// The real regression test: recording executor swapped onto the live manager.
func TestDispatch_StrictAgainstRecordingExecutor(t *testing.T) {
assertStrict(t, func(store *Store, rec *recordingExecutor) (*Manager, func(Skill, Payload) (int, string)) {
mgr := NewManager(store, rec)
return mgr, mgr.Dispatch
})
}
Rename/comment it so its scope is explicit. It may assert 202 || 500 and that a row was written; it must not be treated as execution coverage.
// TestDispatch_TolerantContract verifies only the dispatch contract:
// a request is either accepted or explicitly refused because the environment
// is unavailable. It does NOT prove the skill ran.
func TestDispatch_TolerantContract(t *testing.T) { /* unchanged */ }
Build-tagged so the default suite stays green; -tags mutant points the strict body at the buggy implementation:
//go:build mutant
package skill
import "testing"
func TestStrictAgainstMutant(t *testing.T) {
assertStrict(t, func(store *Store, rec *recordingExecutor) (*Manager, func(Skill, Payload) (int, string)) {
mgr := NewManager(store, rec)
return mgr, mgr.MutantDispatch
})
}
$ cd tolerance-repro && go test ./... -v
=== RUN TestDispatch_TolerantContract
--- PASS: TestDispatch_TolerantContract (0.00s)
=== RUN TestTolerantContract_AcceptsMutant
--- PASS: TestTolerantContract_AcceptsMutant (0.00s)
=== RUN TestDispatch_StrictAgainstRecordingExecutor
--- PASS: TestDispatch_StrictAgainstRecordingExecutor (0.00s)
PASS
ok tolerance-repro/skill 0.002s
TestTolerantContract_AcceptsMutant deliberately runs the old assertions against the mutant and passes — this is the documented false acceptance: the row is completed while FireCount() == 0.
$ go test -tags mutant ./... -run 'TestStrictAgainstMutant|TestTolerantContract_AcceptsMutant' -v
=== RUN TestTolerantContract_AcceptsMutant
--- PASS: TestTolerantContract_AcceptsMutant (0.00s)
=== RUN TestStrictAgainstMutant
mutant_strict_test.go:11: executor invocations = 0, want 1
--- FAIL: TestStrictAgainstMutant (0.00s)
FAIL
FAIL tolerance-repro/skill 0.004s
FAIL
The tolerant clause passes the mutant; the strict contract fails it on the first execution-specific assertion. This is the desired discrimination.
if status != 202 && status != 500 (or similar ||-tolerance) assertion in integration tests.recordingExecutor injected into the real manager/handler under test.202 only, the recorded (skill.Source, payload), exactly one event row with terminal completed, and the fire counter delta...._TolerantContract with a comment that it only covers dispatch, never execution.Run is never called (mutant proves it).completed.# Evidence - Problem class: go-test-tolerance-clause-false-acceptance - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-25T04:53:55.203Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "An integration test asserting status is 202 OR 500 (env-unavailable escape hatch) with executors hardcoded to fail proves dispatch, not execution: the event row is written BEFORE the runner call, so a completed-looking row can coexist with a skill that never ran. Judge pierced it. Fix pattern: recording executor closure swapped onto the live manager, assert 202-only + executor invoked with skill source/payload + exactly one event row reaching completed + fire counter incremented; keep the tolerant test only as dispatch-contract check.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-test-tolerance-clause-false-acceptance", "provider": "openrouter", "solved_at": "2026-09-25T04:53:55.203Z", "version": ""}Verified end-to-end. The full solution is written to ~/solution.md, with a runnable reproduction at ~/tolerance-repro/. Below is the deliverable.
go-test-tolerance-clause-false-acceptanceProblem class: an integration test that accepts 202 OR 500 while injecting an executor hardcoded to fail proves only that dispatch was attempted — not that the skill ran. Because the event row is written before the runner call, a completed-looking row can coexist with a skill that never executed. The tolerance clause (|| 500) silently absorbs exactly the failure the test was meant to catch.
This document contains a self-contained reproduction, the root cause, the exact test fix, and a verification procedure that proves the old test is blind and the new test is not.
Three independent defects combine into a false acceptance:
The tolerance clause erases the signal. if status != 202 && status != 500 accepts the environment-unavailable escape hatch. When the injected executor is hardcoded to fail, the "happy path" can never be observed, so the assertion is tautologically satisfied by the failure branch. A 500 here is not a valid outcome — it is the bug.
The event row is written before the runner call. Dispatch does roughly:
go
ev := store.Insert(Event{Status: StatusRunning}) // row exists already
if err := exec.Run(skill, payload); err != nil { // runner may never run
store.UpdateStatus(ev.ID, StatusFailed)
return 500, ev.ID
}
store.UpdateStatus(ev.ID, StatusCompleted)
Any test that merely asserts a row exists passes whether or not the executor was invoked. Worse, a mutant that inserts the row as completed and returns 202 without calling the runner satisfies both 202 and "row exists".
Run was called, (b) with which skill source/payload, (c) whether the row reached the terminal completed state, or (d) whether the fire counter moved. The test cannot distinguish dispatch from execution.The fix is a recording executor swapped onto the live manager plus exact assertions: 202 only, executor invoked once with the exact source/payload, exactly one event row, exactly one reaching completed, and the fire counter incremented. The tolerant test is retained, but demoted to a dispatch-contract check.
tolerance-repro/
├── go.mod
└── skill/
├── manager.go # production manager + mutant
├── manager_test.go # tolerant (kept) + strict (new) tests
└── mutant_strict_test.go # build-tagged: strict body vs. the mutant
skill/manager.go (core, abridged):
type Executor interface {
Run(skill Skill, payload Payload) error
}
type Manager struct {
store *Store
exec Executor
mu sync.Mutex
fireCnt int64
}
// Dispatch writes the row, then invokes the runner, then transitions the row.
func (m *Manager) Dispatch(skill Skill, payload Payload) (status int, eventID string) {
if m.exec == nil {
return 500, "" // environment-unavailable escape hatch
}
ev := m.store.Insert(Event{SkillID: skill.ID, Source: skill.Source, Status: StatusRunning})
if err := m.exec.Run(skill, payload); err != nil {
m.store.UpdateStatus(ev.ID, StatusFailed)
return 500, ev.ID
}
m.store.UpdateStatus(ev.ID, StatusCompleted)
m.incFire()
return 202, ev.ID
}
// MutantDispatch models the bug the tolerant test cannot see: the row is written
// as "completed" BEFORE (in fact, without) any runner call.
func (m *Manager) MutantDispatch(skill Skill, payload Payload) (status int, eventID string) {
ev := m.store.Insert(Event{SkillID: skill.ID, Source: skill.Source, Status: StatusCompleted})
return 202, ev.ID
}
The original, broken test:
type alwaysFailExecutor struct{}
func (alwaysFailExecutor) Run(Skill, Payload) error { return errors.New("env unavailable") }
func TestDispatch_TolerantContract(t *testing.T) {
store := NewStore()
mgr := NewManager(store, alwaysFailExecutor{}) // executor hardcoded to fail
status, _ := mgr.Dispatch(Skill{ID: "s1", Source: "skills/foo.md"}, Payload{"msg": "hi"})
if status != 202 && status != 500 { // <-- false-acceptance clause
t.Fatalf("status = %d, want 202 or 500", status)
}
if len(store.All()) == 0 {
t.Fatal("expected an event row to be written") // true even with no execution
}
}
type call struct {
Skill Skill
Payload Payload
}
type recordingExecutor struct {
mu sync.Mutex
calls []call
err error
}
func (r *recordingExecutor) Run(s Skill, p Payload) error {
r.mu.Lock()
defer r.mu.Unlock()
r.calls = append(r.calls, call{Skill: s, Payload: p})
return r.err
}
func (r *recordingExecutor) Calls() []call {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]call, len(r.calls))
copy(out, r.calls)
return out
}
func assertStrict(t *testing.T, setup func(store *Store, rec *recordingExecutor) (*Manager, func(Skill, Payload) (int, string))) {
t.Helper()
store := NewStore()
rec := &recordingExecutor{}
mgr, dispatch := setup(store, rec)
skill := Skill{ID: "s1", Source: "skills/foo.md"}
payload := Payload{"msg": "hi"}
status, _ := dispatch(skill, payload)
// 1. Accepted only. The executor is wired to succeed, so 500 is a real failure.
if status != 202 {
t.Fatalf("status = %d, want 202", status)
}
// 2. Executor actually invoked, exactly once, with the exact source/payload.
calls := rec.Calls()
if len(calls) != 1 {
t.Fatalf("executor invocations = %d, want 1", len(calls))
}
if calls[0].Skill.Source != skill.Source {
t.Fatalf("executor skill source = %q, want %q", calls[0].Skill.Source, skill.Source)
}
if !reflect.DeepEqual(calls[0].Payload, payload) {
t.Fatalf("executor payload = %#v, want %#v", calls[0].Payload, payload)
}
// 3. Exactly one event row, and exactly one reached "completed".
events := store.All()
completed := 0
for _, e := range events {
if e.Status == StatusCompleted {
completed++
}
}
if len(events) != 1 {
t.Fatalf("event rows = %d, want exactly 1", len(events))
}
if completed != 1 {
t.Fatalf("completed rows = %d, want exactly 1", completed)
}
// 4. Fire counter incremented.
if got := mgr.FireCount(); got != 1 {
t.Fatalf("fire counter = %d, want 1", got)
}
}
// The real regression test: recording executor swapped onto the live manager.
func TestDispatch_StrictAgainstRecordingExecutor(t *testing.T) {
assertStrict(t, func(store *Store, rec *recordingExecutor) (*Manager, func(Skill, Payload) (int, string)) {
mgr := NewManager(store, rec)
return mgr, mgr.Dispatch
})
}
Rename/comment it so its scope is explicit. It may assert 202 || 500 and that a row was written; it must not be treated as execution coverage.
// TestDispatch_TolerantContract verifies only the dispatch contract:
// a request is either accepted or explicitly refused because the environment
// is unavailable. It does NOT prove the skill ran.
func TestDispatch_TolerantContract(t *testing.T) { /* unchanged */ }
Build-tagged so the default suite stays green; -tags mutant points the strict body at the buggy implementation:
//go:build mutant
package skill
import "testing"
func TestStrictAgainstMutant(t *testing.T) {
assertStrict(t, func(store *Store, rec *recordingExecutor) (*Manager, func(Skill, Payload) (int, string)) {
mgr := NewManager(store, rec)
return mgr, mgr.MutantDispatch
})
}
$ cd tolerance-repro && go test ./... -v
=== RUN TestDispatch_TolerantContract
--- PASS: TestDispatch_TolerantContract (0.00s)
=== RUN TestTolerantContract_AcceptsMutant
--- PASS: TestTolerantContract_AcceptsMutant (0.00s)
=== RUN TestDispatch_StrictAgainstRecordingExecutor
--- PASS: TestDispatch_StrictAgainstRecordingExecutor (0.00s)
PASS
ok tolerance-repro/skill 0.002s
TestTolerantContract_AcceptsMutant deliberately runs the old assertions against the mutant and passes — this is the documented false acceptance: the row is completed while FireCount() == 0.
$ go test -tags mutant ./... -run 'TestStrictAgainstMutant|TestTolerantContract_AcceptsMutant' -v
=== RUN TestTolerantContract_AcceptsMutant
--- PASS: TestTolerantContract_AcceptsMutant (0.00s)
=== RUN TestStrictAgainstMutant
mutant_strict_test.go:11: executor invocations = 0, want 1
--- FAIL: TestStrictAgainstMutant (0.00s)
FAIL
FAIL tolerance-repro/skill 0.004s
FAIL
The tolerant clause passes the mutant; the strict contract fails it on the first execution-specific assertion. This is the desired discrimination.
if status != 202 && status != 500 (or similar ||-tolerance) assertion in integration tests.recordingExecutor injected into the real manager/handler under test.202 only, the recorded (skill.Source, payload), exactly one event row with terminal completed, and the fire counter delta...._TolerantContract with a comment that it only covers dispatch, never execution.Run is never called (mutant proves it).completed.# Evidence - Problem class: go-test-tolerance-clause-false-acceptance - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-25T04:53:55.203Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "An integration test asserting status is 202 OR 500 (env-unavailable escape hatch) with executors hardcoded to fail proves dispatch, not execution: the event row is written BEFORE the runner call, so a completed-looking row can coexist with a skill that never ran. Judge pierced it. Fix pattern: recording executor closure swapped onto the live manager, assert 202-only + executor invoked with skill source/payload + exactly one event row reaching completed + fire counter incremented; keep the tolerant test only as dispatch-contract check.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-test-tolerance-clause-false-acceptance", "provider": "openrouter", "solved_at": "2026-09-25T04:53:55.203Z", "version": ""}