docs-test-count-drift-gitreins-criterion

Answer

Root cause (two stacked defects). Commit c1b5aeb updated only the README headline line to the new count; the tree diagram and coverage line still said 286 tests (lowercase). Meanwhile the gitreins judge criterion grep was case-sensitive (286 Tests, capital T), so the lowercase 286 tests never matched the criterion and the judge failed open — it passed a stale README. Additionally, the judge's numeric window regex (69x/70x, i.e. 6[89][0-9]|70[0-9]) could not match the real count 716, so even a correct fix would have failed the criterion until the wording was widened (the documented criterion-wording-fix path).

Step 1 — Establish ground truth: run vitest per package and sum.

#!/usr/bin/env bash
set -euo pipefail
total=0
for dir in packages/*/; do
  pkg=$(basename "$dir"); [ -f "$dir/package.json" ] || continue
  (cd "$dir" && npx vitest run --reporter=json \
      --outputFile="/tmp/tests-$pkg.json" >/dev/null 2>&1) || { echo "$pkg: FAIL"; exit 1; }
  n=$(node -e "const r=require('/tmp/tests-$pkg.json');console.log(r.numTotalTests)")
  echo "$pkg: $n"; total=$((total + n))
done
echo "TOTAL: $total"   # -> 716 (drift: 286 -> 716)

Verified methodology: numTotalTests equals summing testResults[].assertionResults.length (both = 15 for a sample package).

Step 2 — Update ALL count references in README.md (not just the headline).

 **716 tests passing**                                   # headline (was fixed by c1b5aeb — confirm)
 ...
 test/
-  unit/     286 tests                                  # tree diagram (stale, lowercase)
+  unit/     716 tests
   e2e/      112 tests
 ...
-Coverage: 94.2% (286 tests)                            # coverage line (stale)
+Coverage: 94.2% (716 tests)

Step 3 — Update SYSTEM_STATUS (same drift there).

-- Tests passing: 286 tests
+- Tests passing: 716 tests

Step 4 — Widen the judge criterion in .gitreins/tasks.yaml (criterion-wording-fix path) and make matching case-insensitive.

tasks:
  - id: readme-test-count
    title: "README test count reflects ground truth"
    criteria:
      # old: "README headline says 286 Tests"  (case-sensitive grep, could not match)
      # old: "... match suite size (69x/70x)"  (window excludes 716)
      - "README headline states 716 tests (case-insensitive match '716 tests')"
      - "Tree diagram and coverage line match suite size 716 (window 7[01][0-9] tests)"

The widened window 7[01][0-9] covers 710–719 (includes 716) and the (?i)/-i flag closes the case-sensitivity hole that let the old judge pass stale docs. If drift is expected, a range criterion is more durable than a fixed count.

Step 5 — Re-run the judge.

gitreins judge                  # full pipeline (Tier 1 guards + evaluator)
# or for criterion-only recheck: gitreins judge --skip-tier2

Expect: criterion items PASS, Overall: PASS ✓.

Step 6 — Budget the CLI suite's live-server test (~9 min). The CLI suite contains an e2e test that starts a live server and POSTs a real benchmark run (~9 min). Give it an explicit per-test timeout with headroom, and isolate it from fast suites:

// test/cli.e2e.test.ts
it("POSTs a real benchmark run end-to-end",
   { timeout: 600_000 },          // 10 min budget vs ~9 min actual runtime
   async () => { /* start server, POST run, assert 202 + persisted result */ });

# CI: run slow e2e in its own job so it can't blow the fast-suite timeout
jobs:
  cli-e2e:
    timeout-minutes: 20          # > 9 min live POST + server teardown
    run: npx vitest run test/cli.e2e.test.ts --testTimeout=600000