gitreins-judge-tier1-timeout-artifact

Answer

Root cause. The judge's tier1 "tests" step enforces a hard 120s tool budget, but the MYPY-CLEANUP-001 suite legitimately needs ~6.5 min (390s). A healthy submission trips the budget and produces FAIL, while tier2 (LLM criteria verification) passes. The judge conflated "tool budget exceeded" (infra limit) with "tests failed" (product defect).

Fix — three parts:

1. Split the failure mode. Tag every tier1 outcome as PASS | TIMEOUT | FAIL. Only FAIL (bad exit code / failed assertions) is a regression signal; TIMEOUT is an infra condition that must be re-checked, not adjudicated.

2. Adjudication rule. FAIL only when tier1 observed a product failure, or tier1 timed out and tier2 criteria failed. TIMEOUT + tier2 PASS ⇒ ARTIFACT, then re-verify with a lifted budget before finalizing.

# adjudicate.py
from dataclasses import dataclass
from enum import Enum

TOOL_BUDGET_S   = 120    # per-step tool budget (tier1)
SUITE_RUNTIME_S = 390    # measured healthy runtime (6.5 min)
FALLBACK_BUDGET_S = 480  # re-verification budget, > suite runtime

class Tier1Outcome(Enum):
    PASS = "pass"; TIMEOUT = "timeout"; FAIL = "fail"

class Verdict(Enum):
    PASS = "PASS"; FAIL = "FAIL"; ARTIFACT = "ARTIFACT"

@dataclass
class Tier2Result: criteria_met: bool

@dataclass
class JudgeResult:
    problem_id: str
    tier1: Tier1Outcome
    tier2: Tier2Result

def classify_verdict(r: JudgeResult) -> Verdict:
    """FAIL is a regression only if tier1 observed a product failure,
    or tier1 timed out AND tier2 criteria also failed.

    tier1 TIMEOUT + tier2 PASS => tooling artifact, not a regression."""
    if r.tier1 is Tier1Outcome.FAIL:
        return Verdict.FAIL
    if r.tier1 is Tier1Outcome.TIMEOUT and not r.tier2.criteria_met:
        return Verdict.FAIL
    if r.tier1 is Tier1Outcome.TIMEOUT and r.tier2.criteria_met:
        return Verdict.ARTIFACT
    return Verdict.PASS

def re_verify(r: JudgeResult, runner) -> JudgeResult:
    """Confirmation pass: rerun tier1 with the fallback budget.
    Only after this does ARTIFACT get finalized as complete."""
    assert r.tier1 is Tier1Outcome.TIMEOUT and r.tier2.criteria_met
    rerun = runner.run_tier1(budget_s=FALLBACK_BUDGET_S)
    return JudgeResult(r.problem_id, rerun, r.tier2)  # PASS => provably healthy

3. Foreman reconciliation. On artifact classification, mark the task complete; if the verdict dir lands after the board commit, fold it into the next commit (verdict dirs are gitignored, so -f is required):

# foreman/reconcile.sh
VERDICT_DIR=".gitreins/history/$(date +%F)/$HASH"
if [ -d "$VERDICT_DIR" ] && [ -n "$(git status --porcelain -- "$VERDICT_DIR")" ]; then
  git add -f "$VERDICT_DIR"   # -f: verdict dirs are normally gitignored
  git commit -m "[artifact] fold judge verdict $HASH (tier1 timeout, tier2 PASS)"
fi

Guard rails to prevent recurrence: (a) record reason on every tier1 outcome (tier1.timeout vs tier1.fail) so aggregators never count timeouts as regressions; (b) when SUITE_RUNTIME_S + margin > TOOL_BUDGET_S, the harness should auto-raise the budget from the problem manifest instead of letting every run trip the 120s cap.