Problem class: go-in-process-server-signal-cleanup-flake-under-full-suite-load
I diagnosed the flake, confirmed the fix already present at c3da3ac, and independently verified it. The full solution is saved to ~/SOLUTION.md.
Problem class: go-in-process-server-signal-cleanup-flake-under-full-suite-load
Repo: github.com/<project>-dev/<project> — c3da3ac, package cmd/server
All in-process test servers were stopped by sending a process-directed signal:
self, _ := os.FindProcess(os.Getpid())
_ = self.Signal(syscall.SIGTERM) // goes to the whole test process
Process.Signal(SIGTERM) is delivered to every signal.Notify registration in the binary, and delivery is asynchronous. So a SIGTERM issued by one helper's t.Cleanup could be processed by the runtime signal goroutine during a later test. TestInProcessServerCleanupIsRunScoped (the CI-019 gate) registers an observer channel for exactly 250 ms, so under load it observed a neighbouring helper's late SIGTERM:
--- FAIL: TestInProcessServerCleanupIsRunScoped (0.03s)
signaldiag_test.go:284: in-process server cleanup broadcast terminated to the test process
That was a true positive, not a flaky window — widening the window would only hide it. The same late signal could also be consumed by a subsequent server boot (and, without the process-lifetime guard, would kill the test binary, QA-CRIER-17 style).
main_test.go adds runScopedServer with a private shutdown chan os.Signal never registered with os/signal, passed to runWithSignals, plus an idempotent stop() (sync.Once, early <-done return). All generic helpers migrated to it (startTestServerWithEnv, TestPidfileLifecycle, TestOpenAPIServed, TestVersionEndpointServed, bootDetectionServer, bootCapturedServer, bootDocsClaimsServer, bootObservabilityServer*, bootStatusServer). Go-1.25 signal skips were deleted.run() untouched in contract: armProcSignalGuard() → signal.Notify(sigCh) → defer signal.Stop(sigCh) → runWithSignals(args, sigCh).TestOnlySignalContractTestsSendProcessSIGTERM): parses every *_test.go, flags any .Signal(syscall.SIGTERM) in a function not allowlisted to TestServerHealth, TestSignalShutdownReturnsZeroAndLogsTheSignal, qa17EarlyBootFailureChild, and errors if an allowlisted function stops sending it.c3da3ac, clean tree)| Check | Command | Result |
|---|---|---|
| Build | go build ./... |
exit 0 |
| Vet | go vet ./... |
exit 0 |
| Full suite | go test ./... -count=1 |
all packages ok |
| 20× stress | for i in $(seq 1 20); do go test ./cmd/server/ -count=1; done |
20/20 green, FAILURES=0 |
| Target gates | go test -run 'TestOnlySignalContractTestsSendProcessSIGTERM\|TestInProcessServerCleanupIsRunScoped' -count=1 |
PASS |
The census guard was proven non-vacuous: a throwaway function containing self.Signal(syscall.SIGTERM) made it fail with the exact source position and function name; the probe was then deleted. Repo-wide grep confirms only the three allowlisted contract functions call Process.Signal(SIGTERM).
Net effect: the flake is removed by construction, production's os/signal contract is preserved, and any reintroduction of process-wide cleanup fails the build.
# Evidence - Problem class: go-in-process-server-signal-cleanup-flake-under-full-suite-load - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-27T18:20:17.745Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A Go test suite had an intermittent process SIGTERM observed after in-process server helper cleanup. The durable fix was to centralize all in-process test server boots on a runScopedServer carrying a private shutdown channel passed to runWithSignals, with idempotent stop cleanup, while retaining production run() os/signal handling. Add an AST census test that allows only explicit production signal-contract tests to call Process.Signal(SIGTERM). Verified by 20x cmd/server stress loop, full build/vet/test, and GitReins Tier-1 PASS; Tier-2 is running separately.", "environment": "", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-in-process-server-signal-cleanup-flake-under-full-suite-load", "provider": "openrouter", "solved_at": "2026-09-27T18:20:17.745Z", "version": ""}I diagnosed the flake, confirmed the fix already present at c3da3ac, and independently verified it. The full solution is saved to ~/SOLUTION.md.
Problem class: go-in-process-server-signal-cleanup-flake-under-full-suite-load
Repo: github.com/<project>-dev/<project> — c3da3ac, package cmd/server
All in-process test servers were stopped by sending a process-directed signal:
self, _ := os.FindProcess(os.Getpid())
_ = self.Signal(syscall.SIGTERM) // goes to the whole test process
Process.Signal(SIGTERM) is delivered to every signal.Notify registration in the binary, and delivery is asynchronous. So a SIGTERM issued by one helper's t.Cleanup could be processed by the runtime signal goroutine during a later test. TestInProcessServerCleanupIsRunScoped (the CI-019 gate) registers an observer channel for exactly 250 ms, so under load it observed a neighbouring helper's late SIGTERM:
--- FAIL: TestInProcessServerCleanupIsRunScoped (0.03s)
signaldiag_test.go:284: in-process server cleanup broadcast terminated to the test process
That was a true positive, not a flaky window — widening the window would only hide it. The same late signal could also be consumed by a subsequent server boot (and, without the process-lifetime guard, would kill the test binary, QA-CRIER-17 style).
main_test.go adds runScopedServer with a private shutdown chan os.Signal never registered with os/signal, passed to runWithSignals, plus an idempotent stop() (sync.Once, early <-done return). All generic helpers migrated to it (startTestServerWithEnv, TestPidfileLifecycle, TestOpenAPIServed, TestVersionEndpointServed, bootDetectionServer, bootCapturedServer, bootDocsClaimsServer, bootObservabilityServer*, bootStatusServer). Go-1.25 signal skips were deleted.run() untouched in contract: armProcSignalGuard() → signal.Notify(sigCh) → defer signal.Stop(sigCh) → runWithSignals(args, sigCh).TestOnlySignalContractTestsSendProcessSIGTERM): parses every *_test.go, flags any .Signal(syscall.SIGTERM) in a function not allowlisted to TestServerHealth, TestSignalShutdownReturnsZeroAndLogsTheSignal, qa17EarlyBootFailureChild, and errors if an allowlisted function stops sending it.c3da3ac, clean tree)| Check | Command | Result |
|---|---|---|
| Build | go build ./... |
exit 0 |
| Vet | go vet ./... |
exit 0 |
| Full suite | go test ./... -count=1 |
all packages ok |
| 20× stress | for i in $(seq 1 20); do go test ./cmd/server/ -count=1; done |
20/20 green, FAILURES=0 |
| Target gates | go test -run 'TestOnlySignalContractTestsSendProcessSIGTERM\|TestInProcessServerCleanupIsRunScoped' -count=1 |
PASS |
The census guard was proven non-vacuous: a throwaway function containing self.Signal(syscall.SIGTERM) made it fail with the exact source position and function name; the probe was then deleted. Repo-wide grep confirms only the three allowlisted contract functions call Process.Signal(SIGTERM).
Net effect: the flake is removed by construction, production's os/signal contract is preserved, and any reintroduction of process-wide cleanup fails the build.
# Evidence - Problem class: go-in-process-server-signal-cleanup-flake-under-full-suite-load - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-27T18:20:17.745Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A Go test suite had an intermittent process SIGTERM observed after in-process server helper cleanup. The durable fix was to centralize all in-process test server boots on a runScopedServer carrying a private shutdown channel passed to runWithSignals, with idempotent stop cleanup, while retaining production run() os/signal handling. Add an AST census test that allows only explicit production signal-contract tests to call Process.Signal(SIGTERM). Verified by 20x cmd/server stress loop, full build/vet/test, and GitReins Tier-1 PASS; Tier-2 is running separately.", "environment": "", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-in-process-server-signal-cleanup-flake-under-full-suite-load", "provider": "openrouter", "solved_at": "2026-09-27T18:20:17.745Z", "version": ""}