go-judge-live-verified-criterion

Answer

Root cause. The GitReins tier2 judge verifies criteria from a clean checkout: /tmp is empty (or unreadable) in the evaluator sandbox. Criterion text like "verified by /tmp/verify_health.sh" references evidence that does not exist in the repository, so the criterion cannot be live-verified and the task FAILs — even though the script ran fine in your workspace.

Fix (3 steps). ① copy the ad-hoc /tmp/verify_*.sh scripts into the repo (scripts/), ② embed a [live-verified] block into each criterion's text in .gitreins/tasks.yaml containing the script path, expected exit code, and the captured stdout/stderr, ③ commit and re-run task complete so the judge re-evaluates the new head.

Before (FAIL, 78446b8a) — evidence lives only in /tmp, unreadable by the evaluator:

# .gitreins/tasks.yaml
tasks:
  df-001:
    title: "Deployment flow DF-001"
    criteria:
      - id: df-001-c1
        text: "Deploy pipeline runs and /healthz answers 200 — verified by /tmp/verify_health.sh"

After (PASS, 4dd73a8d) — evidence is self-contained in the criterion text:

tasks:
  df-001:
    title: "Deployment flow DF-001"
    criteria:
      - id: df-001-c1
        text: |
          Deploy pipeline runs and /healthz answers 200.
          [live-verified]
          script: scripts/verify_health.sh
          exit: 0
          output: |
            probe http://<ip-address>:8080/healthz
            GET /healthz -> 200 OK (12ms)
            checks: 1 passed, 0 failed
      - id: df-001-c4
        text: |
          Canary p95 latency under 500ms (output truncated; integrity pinned by sha256).
          [live-verified]
          script: scripts/verify_canary.sh
          exit: 0
          truncated: true
          sha256: e5f86795f290747de8d51aef6f309b863bc78950bd38e9bfa8b67992c62786a7
          output: |
            p95 latency: 210ms

Judge-side rule (Go, enforces the contract — key excerpt):

func CheckCriterion(c Criterion, repoRoot string, mode Mode) Result {
    ev, err := parseEvidence(c.Text) // requires a [live-verified] block
    if err != nil {
        return Result{ID: c.ID, Pass: false, Reason: err.Error()} // e.g. "evidence is not embedded"
    }
    if filepath.IsAbs(ev.Script) {
        return Result{ID: c.ID, Pass: false, Reason: "absolute path outside the repo; evaluator cannot read it"}
    }
    scriptPath := filepath.Join(repoRoot, ev.Script)
    if mode == ModeLive { // re-run committed script, diff against embedded evidence
        out, exit, _ := runScript(scriptPath, 5*time.Second)
        if exit != ev.Exit { /* FAIL */ }
        if ev.Truncated {
            if sha256(out) != ev.SHA256 { /* FAIL */ }
        } else if normalize(out) != normalize(ev.Output) { /* FAIL */ }
        return Result{Pass: true, Reason: "live re-run matches embedded evidence"}
    }
    return Result{Pass: true, Reason: "self-contained embedded evidence"} // offline evaluator
}

parseEvidence (in /tmp/judge-demo/judge/judge.go) rejects: no marker, missing script:, empty captured output, truncated without sha256:, checksum mismatch on full outputs, and secrets (api_key/token/sk-…) in the embedded output. A 12-line output: | YAML literal keeps multi-line captures diffable.