Artifact: src/engine/demoarchitecturepipelinetest.go → TestDemoArchitecturePipeline
The repository was not mounted in this environment (only /workspace/problem.json and the toolchain were present), so I reconstructed the fix from the line-verified mechanism in the problem statement. The solution is written to /workspace/solution.md. Marker literals and output variable names are shown as placeholders to be mapped onto the test's existing private markers.
TestDemo_ArchitecturePipeline hermetic against ambient provider-lane keysRepo: Hermes-DAGger/<project>
Artifact: src/engine/demo_architecture_pipeline_test.go → TestDemo_ArchitecturePipeline
Language: Go
Symptom: the test opened a real production LLM connection when an ambient provider key was present, bypassed its own httptest stub, and let real model prose decide PASS/FAIL.
Evidence: .gitreins/history/2026-09-17/91099156/verdict.json (FAIL, 15.90s, real-model prose quoting the DAGGER_TOOL_MODEL routing guard), commit 0e411eb.
The test did the right thing at the HTTP layer: it stood up an httptest.Server and pointed the client at it with SetAPIBaseURL. But the routing decision happens before the HTTP layer, inside the bridge's provider registry, and that decision reads process-global environment variables.
bridge.NewBridge(...)
└─ NewProviderRegistry(...)
├─ loadFromEnv(...) // DAGGER_LLM_PROVIDERS / DAGGER_LLM_DEFAULT
─ autoDiscover(...)
...
agentCtx()
└─ registry.Get("<lane-name>")
─ MISS → tryRegister(name) // self-registration on lookup miss
├─ reads DAGGER_LLM_<UPPER_NAME>_KEY // envKeyForName form
└─ else laneAliases[name] e.g. "deepseek-foreman" → DEEPSEEK_FOREMAN_API_KEY
Two facts combine into the defect:
ProviderRegistry.Get(name) is not a pure lookup. On a cache miss it calls tryRegister, which pulls a provider config straight out of the ambient environment. An env var present in the developer shell / CI runner silently materializes a provider lane the test never registered.agentCtx() only falls back to the built-in "hermes" provider when Get fails. That built-in provider is exactly what SetAPIBaseURL serves. If Get succeeds via tryRegister due to DEEPSEEK_FOREMAN_API_KEY (or its DAGGER_LLM_... spelling), the bridge never degrades to hermes, never dials the mock, and egresses for real. No error, no warning.| Condition | Mock hits | Wall time | Result |
|---|---|---|---|
DEEPSEEK_FOREMAN_API_KEY set, before fix |
0 | 15.68s | real prose; passed by keyword luck |
| key unset, before fix | yes | 0.743s | mock answered |
| bogus key, after fix | chat=1 responses=1 | 0.51s | PASS |
| key unset, after fix | chat=1 responses=1 | 0.51s | PASS |
Three coordinated changes, all in the test (no production code changes):
bridge.NewBridge.total == 0.// DAGGER-0931: hermetic provider lane.
//
// bridge.NewBridge -> NewProviderRegistry -> loadFromEnv/autoDiscover, and
// ProviderRegistry.Get() self-registers on a cache miss via tryRegister(),
// which reads DAGGER_LLM_<UPPER_NAME>_KEY and then the laneAliases table
// (deepseek-foreman -> DEEPSEEK_FOREMAN_API_KEY). An ambient key would
// materialize a real provider lane and bypass the httptest mock below.
// Keep the DAGGER-130 lane NAMING intact; only clear credentials/routing
// knobs, in BOTH spellings.
t.Setenv("DEEPSEEK_FOREMAN_API_KEY", "") // lane-alias spelling
t.Setenv("DAGGER_LLM_DEEPSEEK_FOREMAN_KEY", "") // envKeyForName spelling
t.Setenv("DAGGER_LLM_PROVIDERS", "") // registry pre-population
t.Setenv("DAGGER_LLM_DEFAULT", "") // default lane selection
t.Setenv restores the prior value in t.Cleanup and makes the test incompatible with t.Parallel() (this test must not become parallel). If a future registry switches to os.LookupEnv (presence vs value), replace with an os.Unsetenv + t.Cleanup restore loop for the same four keys.
var chatHits, responsesHits, otherHits atomic.Int64
mock := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/chat/completions"):
chatHits.Add(1)
case strings.HasSuffix(r.URL.Path, "/responses"):
responsesHits.Add(1)
default:
otherHits.Add(1)
}
w.Header().Set("Content-Type", "application/json")
switch r.URL.Path {
case "/v1/chat/completions":
io.WriteString(w, `{"choices":[{"message":{"role":"assistant","content":"MOCK_HERMES_CHAT_OK"}}]}`)
case "/v1/responses":
io.WriteString(w, `{"output":[{"type":"message","content":[{"type":"output_text","text":"MOCK_HERMES_RESPONSES_OK"}]}]}`)
default:
http.Error(w, "not found", http.StatusNotFound)
}
}))
defer mock.Close()
Replace MOCK_HERMES_CHAT_OK / MOCK_HERMES_RESPONSES_OK with the test's existing private markers.
if total := chatHits.Load() + responsesHits.Load() + otherHits.Load(); total == 0 {
t.Fatalf("the mock Hermes API was never exercised "+
"(chat=%d responses=%d other=%d total=%d); "+
"a provider-lane env key bypassed the stub",
chatHits.Load(), responsesHits.Load(), otherHits.Load(), total)
}
if !strings.Contains(chatOutput, "MOCK_HERMES_CHAT_OK") {
t.Fatalf("chat node output did not come from the mock: %q", chatOutput)
}
if !strings.Contains(responsesOutput, "MOCK_HERMES_RESPONSES_OK") {
t.Fatalf("responses node output did not come from the mock: %q", responsesOutput)
}
New import: sync/atomic. The total == 0 check converts an invisible bypass into a hard, self-describing failure; the marker checks ensure consumed content is the stub's, not a real provider's.
env -u DEEPSEEK_FOREMAN_API_KEY -u DAGGER_LLM_DEEPSEEK_FOREMAN_KEY \
-u DAGGER_LLM_PROVIDERS -u DAGGER_LLM_DEFAULT \
go test ./src/engine/ -run '^TestDemo_ArchitecturePipeline$' -v -count=1
DEEPSEEK_FOREMAN_API_KEY=sk-bogus \
DAGGER_LLM_DEEPSEEK_FOREMAN_KEY=sk-bogus \
go test ./src/engine/ -run '^TestDemo_ArchitecturePipeline$' -v -count=1
Both must PASS in ~0.5s with tally chat=1 responses=1 other=0 total=2 and no network egress (optionally point HTTP_PROXY/HTTPS_PROXY at a dead port and confirm it still passes).
sha256sum src/engine/demo_architecture_pipeline_test.go | tee /tmp/orig.sha256
sed '/t.Setenv("DEEPSEEK_FOREMAN_API_KEY"/d; \
/t.Setenv("DAGGER_LLM_DEEPSEEK_FOREMAN_KEY"/d; \
/t.Setenv("DAGGER_LLM_PROVIDERS"/d; \
/t.Setenv("DAGGER_LLM_DEFAULT"/d' \
src/engine/demo_architecture_pipeline_test.go > /tmp/mutant_test.go
cat > /tmp/overlay.json <<EOF
{"Replace": {"$(pwd)/src/engine/demo_architecture_pipeline_test.go": "/tmp/mutant_test.go"}}
EOF
DEEPSEEK_FOREMAN_API_KEY=sk-real-ambient \
go test -overlay /tmp/overlay.json ./src/engine/ \
-run '^TestDemo_ArchitecturePipeline$' -v -count=1
sha256sum -c /tmp/orig.sha256 # must print OK
Expected mutant output:
demo_architecture_pipeline_test.go:NNN: the mock Hermes API was never exercised
(chat=0 responses=0 other=0 total=0); a provider-lane env key bypassed the stub
--- FAIL: TestDemo_ArchitecturePipeline (16.21s)
chat=1 responses=1 in both.total=0 and ≈16s under the real key.DAGGER_LLM_<UPPER_NAME>_KEY and the lane-alias var), plus registry-wide selection knobs.t.Setenv (auto-restoring, race-visible); never combine it with t.Parallel().go test -overlay, not by editing the committed file, so the artifact hash stays stable.# Evidence - Problem class: go-unit-test-hermetic-ambient-provider-lane - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T22:27:41.132Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Go unit test reached a REAL production LLM provider because the ambient environment carried a provider-lane key, so the test's own httptest mock was bypassed and PASS/FAIL was decided by real model prose. REPO/ARTIFACT: <project>, src/engine/demo_architecture_pipeline_test.go, TestDemo_ArchitecturePipeline; defect observed by the gitreins tier-1 tests leg (.gitreins/history/2026-09-17/91099156: '--- FAIL: TestDemo_ArchitecturePipeline (15.90s)' with real-model prose quoting the DAGGER_TOOL_MODEL routing guard), while the same commit passed standalone and under guard. MECHANISM (line-verified): bridge.NewBridge builds a ProviderRegistry via NewProviderRegistry -> loadFromEnv/autoDiscover; ProviderRegistry.Get(name) self-registers on a miss via tryRegister, which reads DAGGER_LLM_<UPPER_NAME>_KEY and then falls back to the laneAliases table (deepseek-foreman -> DEEPSEEK_FOREMAN_API_KEY); agentCtx() only degrades to the built-in 'hermes' provider (the one served by the test's SetAPIBaseURL mock) when Get FAILS. Ambient key present -> direct provider lane -> real egress, real tokens, mock bypassed. MEASURED BEFORE: with DEEPSEEK_FOREMAN_API_KEY in the environment the test took 15.68s with ZERO hits on the mock's marker text and logged real-model prose (and still PASSED, by luck of keyword overlap); with the key unset it took 0.743s with the mock actually answering. FIX (verified): inside the test, before constructing the bridge, t.Setenv('DEEPSEEK_FOREMAN_API_KEY',''), t.Setenv('DAGGER_LLM_DEEPSEEK_FOREMAN_KEY',''), t.Setenv('DAGGER_LLM_PROVIDERS',''), t.Setenv('DAGGER_LLM_DEFAULT',''), keeping the DAGGER-130 lane NAMING intact; instrument the httptest mock with an atomic per-endpoint counter and FAIL LOUDLY when the mock was never exercised (chat/responses tally is 0), and assert the mock's own marker literals appear in both node outputs so model prose can never satisfy the check. AFTER: PASS in 0.51s both with a hostile (bogus) ambient lane key and with it unset, tally chat=1 responses=1. FALSIFICATION: disabling the four t.Setenv lines (go test -overlay mutant, committed file untouched, sha256 unchanged) with the real ambient key reproduces the defect verbatim: 'the mock Hermes API was never exercised (chat=0 responses=0 other=0 total=0)' and '--- FAIL: TestDemo_ArchitecturePipeline (16.21s)'. REUSABLE RULES: (1) a test that points a client at a local stub MUST clear the ambient env knobs that can re-route the client, then PROVE the stub was exercised (a silently-bypassed stub is indistinguishable from a stub returning empty output); (2) assert on the stub's own private marker text, never on keywords a real model can also emit; (3) when a lane/provider registry self-registers from env on lookup misses, the lookup key must be cleared in both spellings (canonical envKeyForName form and the lane-alias var).", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-unit-test-hermetic-ambient-provider-lane", "provider": "openrouter", "solved_at": "2026-09-17T22:27:41.132Z", "version": ""}The repository was not mounted in this environment (only /workspace/problem.json and the toolchain were present), so I reconstructed the fix from the line-verified mechanism in the problem statement. The solution is written to /workspace/solution.md. Marker literals and output variable names are shown as placeholders to be mapped onto the test's existing private markers.
TestDemo_ArchitecturePipeline hermetic against ambient provider-lane keysRepo: Hermes-DAGger/<project>
Artifact: src/engine/demo_architecture_pipeline_test.go → TestDemo_ArchitecturePipeline
Language: Go
Symptom: the test opened a real production LLM connection when an ambient provider key was present, bypassed its own httptest stub, and let real model prose decide PASS/FAIL.
Evidence: .gitreins/history/2026-09-17/91099156/verdict.json (FAIL, 15.90s, real-model prose quoting the DAGGER_TOOL_MODEL routing guard), commit 0e411eb.
The test did the right thing at the HTTP layer: it stood up an httptest.Server and pointed the client at it with SetAPIBaseURL. But the routing decision happens before the HTTP layer, inside the bridge's provider registry, and that decision reads process-global environment variables.
bridge.NewBridge(...)
└─ NewProviderRegistry(...)
├─ loadFromEnv(...) // DAGGER_LLM_PROVIDERS / DAGGER_LLM_DEFAULT
─ autoDiscover(...)
...
agentCtx()
└─ registry.Get("<lane-name>")
─ MISS → tryRegister(name) // self-registration on lookup miss
├─ reads DAGGER_LLM_<UPPER_NAME>_KEY // envKeyForName form
└─ else laneAliases[name] e.g. "deepseek-foreman" → DEEPSEEK_FOREMAN_API_KEY
Two facts combine into the defect:
ProviderRegistry.Get(name) is not a pure lookup. On a cache miss it calls tryRegister, which pulls a provider config straight out of the ambient environment. An env var present in the developer shell / CI runner silently materializes a provider lane the test never registered.agentCtx() only falls back to the built-in "hermes" provider when Get fails. That built-in provider is exactly what SetAPIBaseURL serves. If Get succeeds via tryRegister due to DEEPSEEK_FOREMAN_API_KEY (or its DAGGER_LLM_... spelling), the bridge never degrades to hermes, never dials the mock, and egresses for real. No error, no warning.| Condition | Mock hits | Wall time | Result |
|---|---|---|---|
DEEPSEEK_FOREMAN_API_KEY set, before fix |
0 | 15.68s | real prose; passed by keyword luck |
| key unset, before fix | yes | 0.743s | mock answered |
| bogus key, after fix | chat=1 responses=1 | 0.51s | PASS |
| key unset, after fix | chat=1 responses=1 | 0.51s | PASS |
Three coordinated changes, all in the test (no production code changes):
bridge.NewBridge.total == 0.// DAGGER-0931: hermetic provider lane.
//
// bridge.NewBridge -> NewProviderRegistry -> loadFromEnv/autoDiscover, and
// ProviderRegistry.Get() self-registers on a cache miss via tryRegister(),
// which reads DAGGER_LLM_<UPPER_NAME>_KEY and then the laneAliases table
// (deepseek-foreman -> DEEPSEEK_FOREMAN_API_KEY). An ambient key would
// materialize a real provider lane and bypass the httptest mock below.
// Keep the DAGGER-130 lane NAMING intact; only clear credentials/routing
// knobs, in BOTH spellings.
t.Setenv("DEEPSEEK_FOREMAN_API_KEY", "") // lane-alias spelling
t.Setenv("DAGGER_LLM_DEEPSEEK_FOREMAN_KEY", "") // envKeyForName spelling
t.Setenv("DAGGER_LLM_PROVIDERS", "") // registry pre-population
t.Setenv("DAGGER_LLM_DEFAULT", "") // default lane selection
t.Setenv restores the prior value in t.Cleanup and makes the test incompatible with t.Parallel() (this test must not become parallel). If a future registry switches to os.LookupEnv (presence vs value), replace with an os.Unsetenv + t.Cleanup restore loop for the same four keys.
var chatHits, responsesHits, otherHits atomic.Int64
mock := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/chat/completions"):
chatHits.Add(1)
case strings.HasSuffix(r.URL.Path, "/responses"):
responsesHits.Add(1)
default:
otherHits.Add(1)
}
w.Header().Set("Content-Type", "application/json")
switch r.URL.Path {
case "/v1/chat/completions":
io.WriteString(w, `{"choices":[{"message":{"role":"assistant","content":"MOCK_HERMES_CHAT_OK"}}]}`)
case "/v1/responses":
io.WriteString(w, `{"output":[{"type":"message","content":[{"type":"output_text","text":"MOCK_HERMES_RESPONSES_OK"}]}]}`)
default:
http.Error(w, "not found", http.StatusNotFound)
}
}))
defer mock.Close()
Replace MOCK_HERMES_CHAT_OK / MOCK_HERMES_RESPONSES_OK with the test's existing private markers.
if total := chatHits.Load() + responsesHits.Load() + otherHits.Load(); total == 0 {
t.Fatalf("the mock Hermes API was never exercised "+
"(chat=%d responses=%d other=%d total=%d); "+
"a provider-lane env key bypassed the stub",
chatHits.Load(), responsesHits.Load(), otherHits.Load(), total)
}
if !strings.Contains(chatOutput, "MOCK_HERMES_CHAT_OK") {
t.Fatalf("chat node output did not come from the mock: %q", chatOutput)
}
if !strings.Contains(responsesOutput, "MOCK_HERMES_RESPONSES_OK") {
t.Fatalf("responses node output did not come from the mock: %q", responsesOutput)
}
New import: sync/atomic. The total == 0 check converts an invisible bypass into a hard, self-describing failure; the marker checks ensure consumed content is the stub's, not a real provider's.
env -u DEEPSEEK_FOREMAN_API_KEY -u DAGGER_LLM_DEEPSEEK_FOREMAN_KEY \
-u DAGGER_LLM_PROVIDERS -u DAGGER_LLM_DEFAULT \
go test ./src/engine/ -run '^TestDemo_ArchitecturePipeline$' -v -count=1
DEEPSEEK_FOREMAN_API_KEY=sk-bogus \
DAGGER_LLM_DEEPSEEK_FOREMAN_KEY=sk-bogus \
go test ./src/engine/ -run '^TestDemo_ArchitecturePipeline$' -v -count=1
Both must PASS in ~0.5s with tally chat=1 responses=1 other=0 total=2 and no network egress (optionally point HTTP_PROXY/HTTPS_PROXY at a dead port and confirm it still passes).
sha256sum src/engine/demo_architecture_pipeline_test.go | tee /tmp/orig.sha256
sed '/t.Setenv("DEEPSEEK_FOREMAN_API_KEY"/d; \
/t.Setenv("DAGGER_LLM_DEEPSEEK_FOREMAN_KEY"/d; \
/t.Setenv("DAGGER_LLM_PROVIDERS"/d; \
/t.Setenv("DAGGER_LLM_DEFAULT"/d' \
src/engine/demo_architecture_pipeline_test.go > /tmp/mutant_test.go
cat > /tmp/overlay.json <<EOF
{"Replace": {"$(pwd)/src/engine/demo_architecture_pipeline_test.go": "/tmp/mutant_test.go"}}
EOF
DEEPSEEK_FOREMAN_API_KEY=sk-real-ambient \
go test -overlay /tmp/overlay.json ./src/engine/ \
-run '^TestDemo_ArchitecturePipeline$' -v -count=1
sha256sum -c /tmp/orig.sha256 # must print OK
Expected mutant output:
demo_architecture_pipeline_test.go:NNN: the mock Hermes API was never exercised
(chat=0 responses=0 other=0 total=0); a provider-lane env key bypassed the stub
--- FAIL: TestDemo_ArchitecturePipeline (16.21s)
chat=1 responses=1 in both.total=0 and ≈16s under the real key.DAGGER_LLM_<UPPER_NAME>_KEY and the lane-alias var), plus registry-wide selection knobs.t.Setenv (auto-restoring, race-visible); never combine it with t.Parallel().go test -overlay, not by editing the committed file, so the artifact hash stays stable.# Evidence - Problem class: go-unit-test-hermetic-ambient-provider-lane - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T22:27:41.132Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Go unit test reached a REAL production LLM provider because the ambient environment carried a provider-lane key, so the test's own httptest mock was bypassed and PASS/FAIL was decided by real model prose. REPO/ARTIFACT: <project>, src/engine/demo_architecture_pipeline_test.go, TestDemo_ArchitecturePipeline; defect observed by the gitreins tier-1 tests leg (.gitreins/history/2026-09-17/91099156: '--- FAIL: TestDemo_ArchitecturePipeline (15.90s)' with real-model prose quoting the DAGGER_TOOL_MODEL routing guard), while the same commit passed standalone and under guard. MECHANISM (line-verified): bridge.NewBridge builds a ProviderRegistry via NewProviderRegistry -> loadFromEnv/autoDiscover; ProviderRegistry.Get(name) self-registers on a miss via tryRegister, which reads DAGGER_LLM_<UPPER_NAME>_KEY and then falls back to the laneAliases table (deepseek-foreman -> DEEPSEEK_FOREMAN_API_KEY); agentCtx() only degrades to the built-in 'hermes' provider (the one served by the test's SetAPIBaseURL mock) when Get FAILS. Ambient key present -> direct provider lane -> real egress, real tokens, mock bypassed. MEASURED BEFORE: with DEEPSEEK_FOREMAN_API_KEY in the environment the test took 15.68s with ZERO hits on the mock's marker text and logged real-model prose (and still PASSED, by luck of keyword overlap); with the key unset it took 0.743s with the mock actually answering. FIX (verified): inside the test, before constructing the bridge, t.Setenv('DEEPSEEK_FOREMAN_API_KEY',''), t.Setenv('DAGGER_LLM_DEEPSEEK_FOREMAN_KEY',''), t.Setenv('DAGGER_LLM_PROVIDERS',''), t.Setenv('DAGGER_LLM_DEFAULT',''), keeping the DAGGER-130 lane NAMING intact; instrument the httptest mock with an atomic per-endpoint counter and FAIL LOUDLY when the mock was never exercised (chat/responses tally is 0), and assert the mock's own marker literals appear in both node outputs so model prose can never satisfy the check. AFTER: PASS in 0.51s both with a hostile (bogus) ambient lane key and with it unset, tally chat=1 responses=1. FALSIFICATION: disabling the four t.Setenv lines (go test -overlay mutant, committed file untouched, sha256 unchanged) with the real ambient key reproduces the defect verbatim: 'the mock Hermes API was never exercised (chat=0 responses=0 other=0 total=0)' and '--- FAIL: TestDemo_ArchitecturePipeline (16.21s)'. REUSABLE RULES: (1) a test that points a client at a local stub MUST clear the ambient env knobs that can re-route the client, then PROVE the stub was exercised (a silently-bypassed stub is indistinguishable from a stub returning empty output); (2) assert on the stub's own private marker text, never on keywords a real model can also emit; (3) when a lane/provider registry self-registers from env on lookup misses, the lookup key must be cleared in both spellings (canonical envKeyForName form and the lane-alias var).", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-unit-test-hermetic-ambient-provider-lane", "provider": "openrouter", "solved_at": "2026-09-17T22:27:41.132Z", "version": ""}