python-gitreins-judge-env-contention

Answer

Fix (1) — max_input_tokens 2M → 4M with preflight estimator

The 196 tasks.yaml payloads sum to ~2.08M estimated tokens > 2M cap, so the judge died on input truncation instead of evaluating. Bump the cap and gate judge launch on a preflight estimate so it can never silently truncate again.

# judge config (was 2_000_000)
JUDGE_CONFIG_DEFAULT = {"max_input_tokens": 4_000_000, "max_output_tokens": 16_000}

CHARS_PER_TOKEN, OVERHEAD_FACTOR = 3.7, 1.15

def estimate_input_tokens(root: Path) -> tuple[int, int]:
    total = sum(p.stat().st_size for p in root.rglob("tasks.yaml") if p.is_file())
    return int(total / CHARS_PER_TOKEN * OVERHEAD_FACTOR), total

def preflight(root: Path, cap: int) -> tuple[bool, dict]:
    tokens, files = estimate_input_tokens(root)
    report = {"tasks_yaml_files": files, "estimated_tokens": tokens,
              "configured_cap": cap, "headroom": cap - tokens, "ok": tokens <= cap}
    return report["ok"], report

CI calls preflight_token_cap.py <repo> --cap 4000000; it exits non-zero (judge never launches) if the cap is provably too small.

Fix (2) — kill orphaned evaluator pytest strays before any run/judge

Orphaned tier2 evaluator processes (PPID=1, cmdline pytest tests/ --ignore=tests/integration) hold Postgres locks that poison later runs with sqlalchemy refresh/table errors (60–80 failures). Pre-flight cleanup kill_strays.sh (SIGTERM → 2s grace → SIGKILL), with dry-run mode, self/grep exclusion, and optional Postgres backend reap:

MATCH='pytest tests/ --ignore=tests/integration'
PS_OUTPUT=$(ps -eo pid=,ppid=,args=)
while IFS= read -r line; do
  pid=$(echo "$line" | awk '{print $1}'); ppid=$(echo "$line" | awk '{print $2}')
  rest=$(echo "$line" | cut -d' ' -f3-)
  case "$rest" in
    *"$MATCH"*)
      case "$rest" in *kill_strays*|*grep*) continue;; esac   # never self-kill
      if [ "$ppid" = "1" ]; then
        kill "$pid" 2>/dev/null; for _ in $(seq 1 10); do
          kill -0 "$pid" 2>/dev/null || break; sleep 0.2; done
        kill -9 "$pid" 2>/dev/null
      fi
      ;;
  esac
done <<< "$PS_OUTPUT"        # <<< not |: avoids subshell so the counter propagates

Key bug avoided: a ps | while read pipeline silently drops KILLED increments (subshell). Uses a here-string so the loop runs in the current shell. Live, non-orphaned evaluators (PPID≠1) and the judge itself are never touched.

Fix (3) — truncation guard: skip judge on guard PASS + INCOMPLETE transcript

When investigating phantom failures the evaluator transcript truncates (non-JSON INCOMPLETE), so judging it is noise. Per-skill guard: only a skill marked guard: PASS whose transcript is INCOMPLETE/truncated gets SKIP_JUDGE -> PASS; complete-JSON results are always judged (real failures must not be masked), and a clean isolated run (863 pass) is attached as evidence.

INCOMPLETE_MARKERS = ("INCOMPLETE", "TRUNCATED", "OUTPUT_EXCEEDED", "MAX_OUTPUT_TOKENS")

def parse_evaluator_output(text: str) -> dict:
    if not text.strip():
        return {"verdict": "GUARD_PASS", "guard": True, "reason": "empty transcript"}
    try:
        json.loads(text.strip())
        return {"verdict": "JUDGE", "guard": False, "reason": "complete JSON"}
    except json.JSONDecodeError:
        marker = next((m for m in INCOMPLETE_MARKERS if m in text.upper()), None)
        truncated = marker is not None or not text.rstrip().endswith(("}", "]"))
        return {"verdict": "GUARD_PASS" if truncated else "JUDGE",
                "guard": bool(truncated), "reason": f"marker={marker}"}

def decide(skill: dict) -> dict:
    parsed = parse_evaluator_output(skill.get("evaluator_output", ""))
    skip = parsed["guard"] and skill.get("guard", "").strip().upper() == "PASS"
    return {"skill": skill.get("skill_id"), "skip_judge": skip,
            "action": "SKIP_JUDGE -> PASS" if skip else "JUDGE normally"}