go-test-real-model-not-in-registry
The fix is verified and documented. Summary:
pkg/agent/ai_agent_real_model_test.go gates its live-model subtests only on OPENROUTER_API_KEY presence. With a key set (common on dev machines/CI), the subtests invoked openai/gpt-4o-mini and anthropic/claude-3.5-haiku — models absent from the pkg/harness registry. The adapter rejects them with a non-retryable VALIDATION_ERROR before any I/O, but the agent retry loop retried it anyway (3 × 4s), producing the ~24s hard FAIL on clean checkouts. Three gaps: wrong test gate, no registry lookup exposed to tests, and a retry loop that sleeps on errors that can never succeed.
/workspace/KM-GAP-047-solution.md, self-contained)pkg/harness — expose HasModel(id) + RegisteredModelIDs().t.Skipf with a reason naming the model and the registered set (so the registry drift becomes visible, not a red build).ai_agent_offline_test.go with a stub adapter so token-tracking asserts run without network/key/registry.ErrModelNotFound becomes non-retryable → the agent fails fast instead of burning 24s, covering any unguarded caller./tmp/km-sim)FAIL (16.01s), exact report error string model anthropic/claude-3.5-haiku not found in registry (VALIDATION_ERROR, non-retryable) (24.01s in production with per-attempt overhead).ok 0.003sok 0.007s (was FAIL 16–24s)81 tokens)PASS, also clean under -racegofmt -e syntax validation.One real follow-up surfaced: gpt-4o-mini and claude-3.5-haiku are simply not registered — provision them or remove them from realModelCases; the new skip text tracks it as visible debt.
# Evidence - Problem class: go-test-real-model-not-in-registry - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-08-18T18:22:33.750Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Real-model AI agent tests hard-FAIL when OPENROUTER_API_KEY is set but the model is absent from the harness registry. Root cause: pkg/agent/ai_agent_real_model_test.go gates ONLY on env-key presence; with a key set, subtests run against openai/gpt-4o-mini and anthropic/claude-3.5-haiku which are not in the pkg/harness model registry -> VALIDATION_ERROR model not found in registry -> 3 retries x 4s -> FAIL ~24s in (go test ./... fails on clean checkout with key set; env-unset skips cleanly). Fix pattern: skip when registry lacks the model (not just when key missing) with clear reason, or gate behind build tag / env flag, plus offline stub/fake adapter so token-tracking asserts run without live API.", "environment": "production", "language": "go", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "go-test-real-model-not-in-registry", "provider": "openrouter", "solved_at": "2026-08-18T18:22:33.750Z", "version": "1.25.0"}