go-test-wallclock-request-budget-premise-flake
I wrote the complete solution to ~/INT-CI-020-solution.md. Note: the repo isn't present on this host, so the diff is anchored to the line numbers/stub shape described in the ticket; it's directly applicable but helper names (writeStub, readRecord) should match your file.
Verdict: Not a product defect. The test's premise races a fixed 50 ms application-level context budget against unbounded real work.
Root cause: spawn_failure_test.go:134 uses context.WithTimeout(ctx, 50*time.Millisecond) as the mechanism that triggers cancellation. Under contention, the deadline can expire during the useradd stub's process start, before its shell body writes $useraddLog. Spawn then dies at stage=user-create (signal: killed), the recording log is empty, and the line-144 premise check fails instead of the userdel-compensation behaviour under test.
Fix — cancel on a stub signal, not a timer:
// :134
ctx, cancel := context.WithCancel(context.Background()) // no wall-clock budget
defer cancel()
// keygen stub announces it started, then sleeps
keygenStarted := filepath.Join(tmpDir, "ssh-keygen.started")
writeStub(t, binDir, "ssh-keygen",
fmt.Sprintf("touch %s; sleep 2; exit 1", keygenStarted))
go func() { _, err := spawner.Spawn(ctx, req); errCh <- err }()
// bounded, load-tolerant waits for the real precondition
waitForPath(t, useraddLog, 30*time.Second)
waitForPath(t, keygenStarted, 30*time.Second)
cancel() // lands while keygen sleeps
// :144 premise assertion stays unchanged and is now guaranteed
if calls := readRecord(t, useraddLog); len(calls) == 0 {
t.Fatalf("test premise broken: useradd stub never ran (calls: %v)", calls)
}
Plus a bounded poll helper (waitForPath) that uses os.Stat + 5 ms sleep until a deadline.
Verification (also in the doc):
1. gofmt -l, go vet ./internal/agent/, focused test -count=1 -v.
2. Stress it: saturate all CPUs with busy loops and run the focused test -count=100, then go test ./... -count=1 — expect all green (previously would redden).
3. Keep it honest (INT-CI-005 fingerprint): temporarily rewire the compensating userdel to the cancelled request ctx, watch it fail on the userdel-ran assertion (not the premise), then restore and verify sha256sum -c is OK.
Explicit non-fixes: don't raise the 50 ms constant, don't raise the suite -timeout, and don't delete/weaken the premise assertion — the assertion is correct; only its orchestration races.
# Evidence - Problem class: go-test-wallclock-request-budget-premise-flake - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-18T05:26:38.817Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM. A full `go test ./... -count=1` run on a loaded box reddens one internal/agent test that is green everywhere else: `--- FAIL: TestSpawnRollbackRunsUserdelUnderCancelledContext/cancelled_mid-spawn_after_user_creation` at internal/agent/spawn_failure_test.go:145, message `test premise broken: useradd stub never ran (calls: [])`. The same run's log shows WHY, one line earlier: `level=ERROR msg=\"spawn failed\" agent_id=intci5-cancel-42176 stage=user-create error=\"... useradd bunker-intci5-cancel-42176 failed: signal: killed\"` - the spawn died at the user-create stage, not at the keygen stage the test is built around.\n\nROOT CAUSE (test premise, not the SUT). The test drives a spawn under a WALL-CLOCK request budget: `ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)` (spawn_failure_test.go:134), with `useradd` stubbed to succeed-and-record and `ssh-keygen` stubbed to `sleep 2; exit 1`. Its design intent is: all pre-keygen stages finish in a few ms, the cancel lands while ssh-keygen sleeps, and the rollback is then observed. Its premise assertion at :144 is `if calls := readRecord(t, useraddLog); len(calls) == 0 { t.Fatalf(\"test premise broken: useradd stub never ran\") }`. Under contention (many packages in parallel, or a busy host) the 50 ms budget expires DURING useradd's own subprocess start, so the recording stub never writes its line and the PREMISE assertion - not the behaviour under test - fails.\n\nWHY IT MASQUERADES AS A REGRESSION. The failure names a specific test and a specific assertion, so it reads as a defect in whatever commit happens to be in the tree. The discriminator that settles it: the same test passes in isolation (`go test ./internal/agent/ -run TestSpawnRollbackRunsUserdelUnderCancelledContext -count=1` -> ok 2.1s), the whole package passes on its own (ok 60.4s), and the commit gate (`gitreins guard --full`) is green - only the CONCURRENT whole-repo run fails, and only under load. No file in the failing path (spawn/keygen/useradd) was touched by the commit under suspicion.\n\nFIX / AVOIDANCE PATTERN. Never assert that a fixed wall-clock budget outlasts an unbounded amount of real work. Either (a) drive the cancel from a SIGNAL emitted by the stub (the keygen stub touching a file that the test polls with a bounded deadline) instead of a fixed 50 ms timer, or (b) await the precondition with a bounded poll (\"wait until the useradd record appears, up to N seconds\") before asserting it. This is the same family as a CI test-budget cliff: a timing constant that is sufficient on an idle box and insufficient on a loaded one. Adjacent class: raising a test suite's own -timeout is NOT the fix here, because the flake is inside the test's application-level context budget.\n\nVERIFICATION THAT KEEPS THE TEST HONEST. Do not weaken the assertion while fixing the premise: the test must still redden when the compensating `userdel` is rewired to the cancelled request context (the INT-CI-005 fingerprint). Check it by temporarily restoring the old behaviour, watching the assertion fail for the right reason, then restoring the file and confirming its hash is unchanged.\n\nFILED AS: bunker board row INT-CI-020 (P3) at tick 467, discovered while adjudicating a full-suite run during the GAP-071 (agent lifecycle RPCs) tick; the diff under suspicion touched no spawn-path file, which is what prompted the isolation runs.", "environment": "Go 1.26 on a loaded shared agent host; go test ./... -count=1 (packages run in parallel, ~20 packages, other agents competing for CPU)", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-test-wallclock-request-budget-premise-flake", "provider": "openrouter", "solved_at": "2026-09-18T05:26:38.817Z", "version": ""}I wrote the complete solution to ~/INT-CI-020-solution.md. Note: the repo isn't present on this host, so the diff is anchored to the line numbers/stub shape described in the ticket; it's directly applicable but helper names (writeStub, readRecord) should match your file.
Verdict: Not a product defect. The test's premise races a fixed 50 ms application-level context budget against unbounded real work.
Root cause: spawn_failure_test.go:134 uses context.WithTimeout(ctx, 50*time.Millisecond) as the mechanism that triggers cancellation. Under contention, the deadline can expire during the useradd stub's process start, before its shell body writes $useraddLog. Spawn then dies at stage=user-create (signal: killed), the recording log is empty, and the line-144 premise check fails instead of the userdel-compensation behaviour under test.
Fix — cancel on a stub signal, not a timer:
// :134
ctx, cancel := context.WithCancel(context.Background()) // no wall-clock budget
defer cancel()
// keygen stub announces it started, then sleeps
keygenStarted := filepath.Join(tmpDir, "ssh-keygen.started")
writeStub(t, binDir, "ssh-keygen",
fmt.Sprintf("touch %s; sleep 2; exit 1", keygenStarted))
go func() { _, err := spawner.Spawn(ctx, req); errCh <- err }()
// bounded, load-tolerant waits for the real precondition
waitForPath(t, useraddLog, 30*time.Second)
waitForPath(t, keygenStarted, 30*time.Second)
cancel() // lands while keygen sleeps
// :144 premise assertion stays unchanged and is now guaranteed
if calls := readRecord(t, useraddLog); len(calls) == 0 {
t.Fatalf("test premise broken: useradd stub never ran (calls: %v)", calls)
}
Plus a bounded poll helper (waitForPath) that uses os.Stat + 5 ms sleep until a deadline.
Verification (also in the doc):
1. gofmt -l, go vet ./internal/agent/, focused test -count=1 -v.
2. Stress it: saturate all CPUs with busy loops and run the focused test -count=100, then go test ./... -count=1 — expect all green (previously would redden).
3. Keep it honest (INT-CI-005 fingerprint): temporarily rewire the compensating userdel to the cancelled request ctx, watch it fail on the userdel-ran assertion (not the premise), then restore and verify sha256sum -c is OK.
Explicit non-fixes: don't raise the 50 ms constant, don't raise the suite -timeout, and don't delete/weaken the premise assertion — the assertion is correct; only its orchestration races.
# Evidence - Problem class: go-test-wallclock-request-budget-premise-flake - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-18T05:26:38.817Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM. A full `go test ./... -count=1` run on a loaded box reddens one internal/agent test that is green everywhere else: `--- FAIL: TestSpawnRollbackRunsUserdelUnderCancelledContext/cancelled_mid-spawn_after_user_creation` at internal/agent/spawn_failure_test.go:145, message `test premise broken: useradd stub never ran (calls: [])`. The same run's log shows WHY, one line earlier: `level=ERROR msg=\"spawn failed\" agent_id=intci5-cancel-42176 stage=user-create error=\"... useradd bunker-intci5-cancel-42176 failed: signal: killed\"` - the spawn died at the user-create stage, not at the keygen stage the test is built around.\n\nROOT CAUSE (test premise, not the SUT). The test drives a spawn under a WALL-CLOCK request budget: `ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)` (spawn_failure_test.go:134), with `useradd` stubbed to succeed-and-record and `ssh-keygen` stubbed to `sleep 2; exit 1`. Its design intent is: all pre-keygen stages finish in a few ms, the cancel lands while ssh-keygen sleeps, and the rollback is then observed. Its premise assertion at :144 is `if calls := readRecord(t, useraddLog); len(calls) == 0 { t.Fatalf(\"test premise broken: useradd stub never ran\") }`. Under contention (many packages in parallel, or a busy host) the 50 ms budget expires DURING useradd's own subprocess start, so the recording stub never writes its line and the PREMISE assertion - not the behaviour under test - fails.\n\nWHY IT MASQUERADES AS A REGRESSION. The failure names a specific test and a specific assertion, so it reads as a defect in whatever commit happens to be in the tree. The discriminator that settles it: the same test passes in isolation (`go test ./internal/agent/ -run TestSpawnRollbackRunsUserdelUnderCancelledContext -count=1` -> ok 2.1s), the whole package passes on its own (ok 60.4s), and the commit gate (`gitreins guard --full`) is green - only the CONCURRENT whole-repo run fails, and only under load. No file in the failing path (spawn/keygen/useradd) was touched by the commit under suspicion.\n\nFIX / AVOIDANCE PATTERN. Never assert that a fixed wall-clock budget outlasts an unbounded amount of real work. Either (a) drive the cancel from a SIGNAL emitted by the stub (the keygen stub touching a file that the test polls with a bounded deadline) instead of a fixed 50 ms timer, or (b) await the precondition with a bounded poll (\"wait until the useradd record appears, up to N seconds\") before asserting it. This is the same family as a CI test-budget cliff: a timing constant that is sufficient on an idle box and insufficient on a loaded one. Adjacent class: raising a test suite's own -timeout is NOT the fix here, because the flake is inside the test's application-level context budget.\n\nVERIFICATION THAT KEEPS THE TEST HONEST. Do not weaken the assertion while fixing the premise: the test must still redden when the compensating `userdel` is rewired to the cancelled request context (the INT-CI-005 fingerprint). Check it by temporarily restoring the old behaviour, watching the assertion fail for the right reason, then restoring the file and confirming its hash is unchanged.\n\nFILED AS: bunker board row INT-CI-020 (P3) at tick 467, discovered while adjudicating a full-suite run during the GAP-071 (agent lifecycle RPCs) tick; the diff under suspicion touched no spawn-path file, which is what prompted the isolation runs.", "environment": "Go 1.26 on a loaded shared agent host; go test ./... -count=1 (packages run in parallel, ~20 packages, other agents competing for CPU)", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-test-wallclock-request-budget-premise-flake", "provider": "openrouter", "solved_at": "2026-09-18T05:26:38.817Z", "version": ""}