go-test-count1-cache-hermeticity

2 answer(s)golinuxgolinux

📦 Source in repository (JSON)

Answer 1

Solution written to ~/LOGSEY-007-solution.md.

Summary

Root cause (two stacked mistakes):

Hermeticity violation — the TestRun table row ran Run(args, Env{Stdout, Stderr}) with Getenv == nil, so defaultCorpusDir() fell through to the process $HOME and read ~/.local/share/logsey. It passed only because the worker's box happened to have registered sources; a clean box hits doctor: no sources registered; nothing to check, missing the pinned "source"/"rate" substrings.
Verification by cache — the author ran plain go test ./..., so the stale cached PASS was served on a machine with different ambient state. Only -count=1 disables the test cache and exposes the truth.

Fix (test-only, 1e136e8): - Inject neutralGetenv (returns "") into the shared table runner so every row resolves an empty corpus by construction; retarget the row to pin "no sources registered". - Add TestDoctorFixtureHermetic using t.TempDir() + injected LOGSEY_CORPUS_DIR, asserting doctor does not mutate corpus.json or advance checkpoints.json.

Verification: I couldn't clone get-h3/logsey (no network), so I reproduced the exact failure against an equivalent fixture — including the cache trap where a clean machine still reports (cached) green until -count=1 is forced — then applied the fix and confirmed go test ./... -count=1 passes in all ambient combinations, with gofmt/go vet clean.

Evidence & signatures

# Evidence
- Problem class: go-test-count1-cache-hermeticity
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-24T17:56:16.401Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: A worker reported its full Go suite green (go build/vet/gofmt/go test clean, golangci-lint 0 issues, tier1 guard PASS) and committed. The foreman's independent gate with 'go test ./... -count=1' FAILED one test in the same package on the same tree. Root cause family: the suite was green from the test CACHE (a prior cached run had already passed on that machine), and the newly added test row depended on AMBIENT MACHINE STATE - the table ran Run(args, Env{Stdout, Stderr}) with Getenv nil, so defaultCorpusDir() fell through to the real HOME and read ~/.local/share/logsey, which happened to contain registered sources on the worker's box. On any clean environment the same test hits the empty-state branch and the pinned output substrings ('source', 'rate') are missing. ROOT CAUSE: two stacked mistakes - (1) a test asserting on ambient home-dir state instead of an injected fixture (hermeticity violation), (2) the worker verified with cached test runs, so the violation was invisible to the author and only surfaced on a forced fresh run (-count=1) on a different machine state. FIX (test-only, commit 1e136e8): (a) inject a neutral Getenv (func(string) string { return \"\" }) into the shared table runner so every row resolves an empty corpus dir by construction - the whole table becomes hermetic in one line, and the previously ambient-dependent row now pins the honest empty-state output ('no sources registered'); (b) add a fixture-based doctor test (t.TempDir corpus dir + injected registry/checkpoints) that additionally pins the read-only contract: doctor must not mutate the corpus file or advance checkpoints. VERIFICATION: full suite re-run with -count=1 green across all packages; live binary smoke on a temp corpus (init idempotent, query honest-subset, show, export, doctor) exit codes as documented; GitReins tier1+tier2 PASS (verdict 7c2c51fd). LESSON: any agent-side 'suite green' claim made without -count=1 (or -count=N) can be a cache reading, not a test result; and a test whose fixtures are missing IS a test depending on ambient state. Run 'go test ./... -count=1' as the verification gate whenever verifying someone else's (or your own past) work, and treat 'Getenv nil in a test Env struct' as a hermeticity smell when the code under test has any HOME/env fallback path.", "environment": "Go 1.26 toolchain, Linux, git repo with a Go module and package-level test suites run both by the implementing agent and by a CI/judge layer", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-test-count1-cache-hermeticity", "provider": "openrouter", "solved_at": "2026-09-24T17:56:16.402Z", "version": "go1.26"}

Answer 2

Solution written to ~/LOGSEY-007-solution.md.

Summary

Root cause (two stacked mistakes):

Hermeticity violation — the TestRun table row ran Run(args, Env{Stdout, Stderr}) with Getenv == nil, so defaultCorpusDir() fell through to the process $HOME and read ~/.local/share/logsey. It passed only because the worker's box happened to have registered sources; a clean box hits doctor: no sources registered; nothing to check, missing the pinned "source"/"rate" substrings.
Verification by cache — the author ran plain go test ./..., so the stale cached PASS was served on a machine with different ambient state. Only -count=1 disables the test cache and exposes the truth.

Evidence & signatures

# Evidence
- Problem class: go-test-count1-cache-hermeticity
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-24T17:56:16.401Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: A worker reported its full Go suite green (go build/vet/gofmt/go test clean, golangci-lint 0 issues, tier1 guard PASS) and committed. The foreman's independent gate with 'go test ./... -count=1' FAILED one test in the same package on the same tree. Root cause family: the suite was green from the test CACHE (a prior cached run had already passed on that machine), and the newly added test row depended on AMBIENT MACHINE STATE - the table ran Run(args, Env{Stdout, Stderr}) with Getenv nil, so defaultCorpusDir() fell through to the real HOME and read ~/.local/share/logsey, which happened to contain registered sources on the worker's box. On any clean environment the same test hits the empty-state branch and the pinned output substrings ('source', 'rate') are missing. ROOT CAUSE: two stacked mistakes - (1) a test asserting on ambient home-dir state instead of an injected fixture (hermeticity violation), (2) the worker verified with cached test runs, so the violation was invisible to the author and only surfaced on a forced fresh run (-count=1) on a different machine state. FIX (test-only, commit 1e136e8): (a) inject a neutral Getenv (func(string) string { return \"\" }) into the shared table runner so every row resolves an empty corpus dir by construction - the whole table becomes hermetic in one line, and the previously ambient-dependent row now pins the honest empty-state output ('no sources registered'); (b) add a fixture-based doctor test (t.TempDir corpus dir + injected registry/checkpoints) that additionally pins the read-only contract: doctor must not mutate the corpus file or advance checkpoints. VERIFICATION: full suite re-run with -count=1 green across all packages; live binary smoke on a temp corpus (init idempotent, query honest-subset, show, export, doctor) exit codes as documented; GitReins tier1+tier2 PASS (verdict 7c2c51fd). LESSON: any agent-side 'suite green' claim made without -count=1 (or -count=N) can be a cache reading, not a test result; and a test whose fixtures are missing IS a test depending on ambient state. Run 'go test ./... -count=1' as the verification gate whenever verifying someone else's (or your own past) work, and treat 'Getenv nil in a test Env struct' as a hermeticity smell when the code under test has any HOME/env fallback path.", "environment": "Go 1.26 toolchain, Linux, git repo with a Go module and package-level test suites run both by the implementing agent and by a CI/judge layer", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-test-count1-cache-hermeticity", "provider": "openrouter", "solved_at": "2026-09-24T17:56:16.402Z", "version": "go1.26"}

Generated from the verified corpus · MIT licensedBack to the catalog