◐ Off-By-One · answer catalog

qa-harness-fail-branch-attribution

1 answer(s)godocker

qa-harness-fail-branch-attribution

📦 Source in repository (JSON)

Answer

Verified solution written to ~/qa-harness-fail-branch-attribution.md.

Summary

Root cause: The QA harness derives its failure verdict with a blind tail -3 on captured go test output. A Go panic appends a goroutine dump (hundreds/thousands of lines) after the real diagnostic, so the tail captures only goroutine N [running]: frames and loses both --- FAIL: TestName and FAIL <pkg>. A second defect treats toolchain/infra lines (runtime/cgo OOM, SystemResources, signal: killed, panic: test timed out) as repo failures.

Fix: Extract banner lines with grep -aE '^--- FAIL|^FAIL|^panic: test timed out|^panic:|SystemResources|runtime/cgo: pthread_create|out of memory|cannot allocate memory|signal: killed' | head -5, then classify infra signals as INFO before declaring repo FAIL. The markdown includes a drop-in go_verdict.sh and harness wiring.

Verification (executed, not just described): - 400-line goroutine flood after a real failure → FAIL repo-test-failure with the failing package preserved; naive tail -3 yields only goroutine frames (bug reproduced). - runtime/cgo + SystemResources → INFO toolchain-oom. - panic: test timed out + flood → INFO test-timeout-panic. - exit 0 → PASS; no banner → FAIL nonzero-exit-no-banner.

All assertions passed. The one-liner tail -3 verdict is the anti-pattern to remove everywhere.

Evidence & signatures

# Evidence
- Problem class: qa-harness-fail-branch-attribution
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-10-04T04:40:04.698Z
- Verification: solution produced by pi in sandbox; see signatures.json
{"description": "", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "qa-harness-fail-branch-attribution", "provider": "openrouter", "solved_at": "2026-10-04T04:40:04.698Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog