python-local-cosine-prefilter-to-skip-llm-call
Done. Solution is at ~/solution/SOLUTION.md, backed by runnable code and tests.
Root cause: The gate verb calls the JEV LLM on every invocation even though the decision is lexically decisive for the outer bands. Near-verbatim restatements (cosine 0.9177) and disjoint evidence (0.0937) can be resolved locally, but the gate has no pre-check, so it always pays the round-trip.
Fix: A dependency-free lexical TF-cosine prefilter (cosine_prefilter.py) interposed before the existing model call:
- best_cosine >= 0.85 → HIT, zero model calls
- best_cosine <= 0.10 → MISS, zero model calls
- gray zone (e.g. related 0.2545) → falls through to model_fn unchanged
- max-over-passages pooling by default; concat pooling available; empty overlap → miss; uses the smaller vector for the dot product
Artifacts:
- cosine_prefilter.py — the fix
- test_cosine_prefilter.py — 15 tests (all green) covering the math, the green bands, and the red proofs
- demo.py — cosine measurements + 80% model-call reduction on a mixed batch
- redproof.py — mutation check proving a cosine-blind gate goes red
Verification highlights:
- Reference pair cosines reproduced: restatement 0.9177, related 0.2545, disjoint 0.0940 (documented 0.0937, within 1e-3).
- Red-proof tests push high=0.999 past restatement and low=0.01 below disjoint; the call is then forced through the model, so the thresholds are load-bearing.
- redproof.py confirms the suite fails on a cosine-blind mutation and passes on the fix: OK: suite goes red on cosine-blind mutation and green on the fix.
# Evidence - Problem class: python-local-cosine-prefilter-to-skip-llm-call - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-10-05T03:24:09.291Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "AUG-063: gate verb dominated by a JEV LLM round-trip. Fix: conservative lexical TF-cosine prefilter on pooled evidence BEFORE the model call \u2014 best cosine >= 0.85 returns the hit path with zero model calls, <= 0.10 returns miss with zero calls, gray zone falls through unchanged. Red-proven tests by pushing thresholds past the fixture cosines. Thresholds measured from real pairs (restatement 0.9177, disjoint 0.0937, related 0.2545).", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "python-local-cosine-prefilter-to-skip-llm-call", "provider": "openrouter", "solved_at": "2026-10-05T03:24:09.294Z", "version": ""}