typescript-test-flaky-parallel-timeout-evidence-closure

Answer

The 4 stale rows (auth, assessment, launch, web page — parallel-suite 5s timeouts, filed 2026-08-03) are PRECON-001-class host-contention flakes: the code is green everywhere, only the parallel wall-clock exceeded 5s under load. The gate is immune by design (--no-file-parallelism), so these rows can never block a release — which is exactly why they rotted as forever-pending. The fix has two parts:

1. A closure script that re-runs the flagged files with fresh evidence and closes only on green.

// scripts/close-flaky-evidence.ts
// Re-runs flagged files (isolation → parallel batch → full gate) and closes
// the P3 rows only if every run is green. Host-contention flakes self-heal;
// real regressions keep the row open with the failing report attached.
import { execFileSync } from "node:child_process";
import { writeFileSync, readFileSync, mkdirSync } from "node:fs";

const EVIDENCE_DIR = "artifacts/flaky-evidence";
mkdirSync(EVIDENCE_DIR, { recursive: true });

// The 4 stale rows, mapped to their files.
const ROWS = [
  { id: "EDUOS-4821", files: ["src/api/auth/auth.test.ts"] },            // 15 tests
  { id: "EDUOS-4822", files: ["src/api/assessment/assessment.test.ts"] },// 56 tests
  { id: "EDUOS-4823", files: ["src/api/launch/launch.test.ts"] },        // 15 tests
  { id: "EDUOS-4824", files: ["src/web/page.test.ts"] },                 //  7 tests
];

function vitestRun(files: string[], extra: string[] = []) {
  const out = `${EVIDENCE_DIR}/run-${Date.now()}-${Math.random().toString(36).slice(2)}.json`;
  const t0 = performance.now();
  execFileSync("npx", ["vitest", "run", "--reporter=json", `--outputFile=${out}`, ...extra, ...files],
    { stdio: "pipe", timeout: 300_000 });
  const report = JSON.parse(readFileSync(out, "utf8"));
  const tests = report.testResults.flatMap((r: any) => r.assertionResults);
  return {
    tests: tests.length,
    passed: tests.filter((t: any) => t.status === "passed").length,
    wallMs: Math.round(performance.now() - t0),
  };
}

// npm test with --no-file-parallelism = the actual guard path.
function gateRun() {
  const t0 = performance.now();
  execFileSync("npm", ["test"], { stdio: "pipe", timeout: 600_000 });
  return { wallMs: Math.round(performance.now() - t0) };
}

const isolation: Record<string, any> = {};
for (const row of ROWS)
  for (const f of row.files)
    isolation[f] = vitestRun([f], ["--no-file-parallelism"]); // (1) each file alone

const apiFiles = ROWS.slice(0, 3).flatMap(r => r.files);
const parallelBatch = vitestRun(apiFiles);                    // (2) the mode that flaked

const gate = gateRun();                                       // (3) full sequential suite

const allGreen =
  Object.values(isolation).every(r => r.tests > 0 && r.passed === r.tests) &&
  parallelBatch.passed === parallelBatch.tests &&
  gate.wallMs >= 0; // npm test exit 0 == green

const evidence = {
  rows: ROWS.map(r => r.id),
  runs: { isolation, parallelBatch, gate },
  verdict: allGreen ? "close" : "keep-open",
  commit: execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim(),
  host: process.env.CI ? `ci:${process.env.CI_RUNNER}` : `local:${require("os").hostname()}`,
  ts: new Date().toISOString(),
};

writeFileSync(`${EVIDENCE_DIR}/evidence.json`, JSON.stringify(evidence, null, 2));
console.log(JSON.stringify(evidence, null, 2));

if (allGreen) {
  // Idempotent closure by row id + evidence hash; 404 on an already-closed row is a no-op.
  for (const row of ROWS)
    execFileSync("node", ["scripts/eduos-close.mjs", row.id, `${EVIDENCE_DIR}/evidence.json`],
      { stdio: "inherit" });
  console.log(`✓ closed ${ROWS.map(r => r.id).join(", ")} with fresh evidence`);
} else {
  console.error(`✗ kept ${ROWS.map(r => r.id).join(", ")} open — attach ${EVIDENCE_DIR}/evidence.json`);
  process.exit(1);
}

2. Hardening so this class of row stops being filed at all (triage-before-filing).

// scripts/triage-flaky.ts — run BEFORE filing a new flaky row. Generates the
// same isolation/parallel/gate evidence at filing time, so a row is only ever
// filed when a run genuinely fails; self-healing contention never becomes a row.
export async function triage(files: string[]): Promise<"flake" | "regression"> {
  const isolation = files.map(f => vitestRun([f], ["--no-file-parallelism"]));
  const parallel = vitestRun(files);                 // the mode that timed out
  const allGreen = [...isolation, parallel].every(r => r.passed === r.tests);
  return allGreen ? "flake" : "regression";          // flake → close in tracker, no row
}

Anti-patterns explicitly not used: no blanket testTimeout: 30000, no retry: 3 on the gate, no --poolOptions.threads.singleThread hack. Those would mask real regressions. The parallel timeout stays at 5s; the closure evidence is what changes.