pytest-test-env-dependent-fake-opener-never-called
I've completed the diagnosis and written a verified solution to ~/solution.md.
I cloned coding-hermes/task-router and reproduced both failures, then found the key thing the original problem statement understates: the test-only patch is not enough.
Root causes
Env-dependent probe test — TR-146 added a credential guard in _hermes_capabilities_metadata() that returns {'error': 'no gateway credential configured'} before touching the injected _opener. On CI (no credential env) the fake opener is never called, so seen['url'] raises KeyError.
Golden heads — the evidence is in data/tables/benchmarks.jsonl (openai/gpt-6-luna tool_use = 0.21, live-probe-2026-09-25), but the derived tables the resolver actually reads were stale: model_perf.jsonl still said 0.85 and model_tier.jsonl still said tier 5. So the head never moved in the committed tables, and merely re-anchoring the fixture makes the test fail against the live resolve.
Fix
- tests/test_proxy_hermes_session.py: add monkeypatch, setenv(rsrv.GATEWAY_CREDENTIAL_ENV_VARS[0], 'test-cred'), assert Authorization: Bearer test-cred.
- Regenerate derived tables from the evidence: python scripts/router_seed.py (updates model_perf, category_levels, model_tier). Confirmed openai/gpt-6-luna becomes perf 0.21 / tier -5, and the new heads materialize.
- tests/test_regression.py: re-anchor P0_FORE → xkiro/xiaomi/mimo-v2.6-flash and P2_AGENTIC → xkiro/z-ai/glm-5.3-flash, with evidence/date comments. Assertion unchanged.
Verification (env vars stripped, duckdb installed)
env -u GATEWAY_API_KEY -u HERMES_API_KEY -u API_SERVER_KEY pytest \
tests/test_proxy_hermes_session.py tests/test_regression.py -q
→ 57 passed
Full suite: 1230 passed, 2 skipped, 6 failed — the 6 are pre-existing sandbox-path failures in test_fleet_cooldown_policy_deploy_hash.py, unrelated to this fix.
The document includes the exact diffs, seed commands, grep-based evidence checks, the "never weaken the assertion" rationale, and the generalized lesson about validation guards bypassing injected callback seams.
# Evidence - Problem class: pytest-test-env-dependent-fake-opener-never-called - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-28T00:11:36.245Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A test that stubs an HTTP opener via an _opener= parameter fails with KeyError when the function under test gained an early-return guard (TR-146 credential gate): with no credential env var configured the function returns {'error': 'no gateway credential configured'} BEFORE firing the probe, so the fake opener is never called and the test's seen['url'] KeyError's. Pattern generalizes: any early-return validation guard added to a function that a test instruments through an injected callback seam silently bypasses the test's instrumentation instead of erroring inside it. Fix (tests/test_proxy_hermes_session.py:326): add monkeypatch, set the FIRST env var the credential resolver checks (rsrv.GATEWAY_CREDENTIAL_ENV_VARS[0], check the constant - order matters) to a dummy value like 'test-cred', so the probe path executes without any real credential; optionally assert the resulting Authorization Bearer header to lock the authentication behavior in the same test. Companion failure class in the same CI red: golden fixed-point head tests (tests/test_regression.py::test_golden_fixed_point_heads) assert a price-ordered chain head that legitimately moved after evidence-backed capability data changed (gpt-6-luna tool_use perf downgraded 0.85->0.21, tier 5->-5, from bench:live-probe-2026-09-25; the tool_use>=0 profile bars correctly exclude the lane). Golden head fixtures are deliberate drift detectors: verify the head move is evidence-backed in the data tables (grep model_perf.jsonl for the downgraded category and its source_ref), then re-anchor the fixture to the live resolve output and document the evidence + date in the fixture comment - never weaken the assertion. Verify the env-dependence fix by running the targeted tests with the credential env vars explicitly stripped: env -u GATEWAY_API_KEY -u HERMES_API_KEY -u API_SERVER_KEY pytest ...", "environment": "task-router Python repo, urllib.request opener injection seam, pytest monkeypatch, GitHub Actions CI without local env vars", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "pytest-test-env-dependent-fake-opener-never-called", "provider": "openrouter", "solved_at": "2026-09-28T00:11:36.245Z", "version": "main e1b8c5b"}I've completed the diagnosis and written a verified solution to ~/solution.md.
I cloned coding-hermes/task-router and reproduced both failures, then found the key thing the original problem statement understates: the test-only patch is not enough.
Root causes
Env-dependent probe test — TR-146 added a credential guard in _hermes_capabilities_metadata() that returns {'error': 'no gateway credential configured'} before touching the injected _opener. On CI (no credential env) the fake opener is never called, so seen['url'] raises KeyError.
Golden heads — the evidence is in data/tables/benchmarks.jsonl (openai/gpt-6-luna tool_use = 0.21, live-probe-2026-09-25), but the derived tables the resolver actually reads were stale: model_perf.jsonl still said 0.85 and model_tier.jsonl still said tier 5. So the head never moved in the committed tables, and merely re-anchoring the fixture makes the test fail against the live resolve.
Fix
- tests/test_proxy_hermes_session.py: add monkeypatch, setenv(rsrv.GATEWAY_CREDENTIAL_ENV_VARS[0], 'test-cred'), assert Authorization: Bearer test-cred.
- Regenerate derived tables from the evidence: python scripts/router_seed.py (updates model_perf, category_levels, model_tier). Confirmed openai/gpt-6-luna becomes perf 0.21 / tier -5, and the new heads materialize.
- tests/test_regression.py: re-anchor P0_FORE → xkiro/xiaomi/mimo-v2.6-flash and P2_AGENTIC → xkiro/z-ai/glm-5.3-flash, with evidence/date comments. Assertion unchanged.
Verification (env vars stripped, duckdb installed)
env -u GATEWAY_API_KEY -u HERMES_API_KEY -u API_SERVER_KEY pytest \
tests/test_proxy_hermes_session.py tests/test_regression.py -q
→ 57 passed
Full suite: 1230 passed, 2 skipped, 6 failed — the 6 are pre-existing sandbox-path failures in test_fleet_cooldown_policy_deploy_hash.py, unrelated to this fix.
The document includes the exact diffs, seed commands, grep-based evidence checks, the "never weaken the assertion" rationale, and the generalized lesson about validation guards bypassing injected callback seams.
# Evidence - Problem class: pytest-test-env-dependent-fake-opener-never-called - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-28T00:11:36.245Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A test that stubs an HTTP opener via an _opener= parameter fails with KeyError when the function under test gained an early-return guard (TR-146 credential gate): with no credential env var configured the function returns {'error': 'no gateway credential configured'} BEFORE firing the probe, so the fake opener is never called and the test's seen['url'] KeyError's. Pattern generalizes: any early-return validation guard added to a function that a test instruments through an injected callback seam silently bypasses the test's instrumentation instead of erroring inside it. Fix (tests/test_proxy_hermes_session.py:326): add monkeypatch, set the FIRST env var the credential resolver checks (rsrv.GATEWAY_CREDENTIAL_ENV_VARS[0], check the constant - order matters) to a dummy value like 'test-cred', so the probe path executes without any real credential; optionally assert the resulting Authorization Bearer header to lock the authentication behavior in the same test. Companion failure class in the same CI red: golden fixed-point head tests (tests/test_regression.py::test_golden_fixed_point_heads) assert a price-ordered chain head that legitimately moved after evidence-backed capability data changed (gpt-6-luna tool_use perf downgraded 0.85->0.21, tier 5->-5, from bench:live-probe-2026-09-25; the tool_use>=0 profile bars correctly exclude the lane). Golden head fixtures are deliberate drift detectors: verify the head move is evidence-backed in the data tables (grep model_perf.jsonl for the downgraded category and its source_ref), then re-anchor the fixture to the live resolve output and document the evidence + date in the fixture comment - never weaken the assertion. Verify the env-dependence fix by running the targeted tests with the credential env vars explicitly stripped: env -u GATEWAY_API_KEY -u HERMES_API_KEY -u API_SERVER_KEY pytest ...", "environment": "task-router Python repo, urllib.request opener injection seam, pytest monkeypatch, GitHub Actions CI without local env vars", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "pytest-test-env-dependent-fake-opener-never-called", "provider": "openrouter", "solved_at": "2026-09-28T00:11:36.245Z", "version": "main e1b8c5b"}