typescript-vitest-live-benchmark-run-test-slow

Answer

Diagnosis. The suite is not hung. benchmark.test.ts spawns a real benchmark run: it POSTs --games 1 --models pair to /api/v1/benchmark and then polls GET /api/v1/benchmark/runs until the run finishes (up to 12 min). Under host load the full suite is 84 tests ≈ 528 s (≈ 6.3 s/test). Any foreground timeout shorter than that — e.g. the observed 360 s — kills vitest mid-run, which looks exactly like a hang (the log stops after the RUN banner). Never diagnose "hung" before checking the API: a run in status: "running" proves the process is working.

Fix — operational (must-do): run vitest in the background with a log file and wait it out; do not kill at 5 min.

# run-with-log.sh — the correct way to run the slow suite
cd "$REPO"

# 1. Before diagnosing anything, verify a live run exists:
curl -s http://<ip-address>:18080/api/v1/benchmark/runs
#    -> {"runs":[{"id":"...","status":"running", ...}]}  => NOT hung, just slow

# 2. Start the suite in the background with a log file (not foreground):
npx vitest run --reporter=verbose > vitest.log 2>&1 &
VITEST_PID=$!
echo "vitest pid=$VITEST_PID, log=vitest.log"

# 3. Wait for completion — do NOT kill at 5 min. Budget ~12 min:
while kill -0 "$VITEST_PID" 2>/dev/null; do sleep 10; done
wait "$VITEST_PID"; CODE=$?

# 4. Report:
tail -n 20 vitest.log          # expect "Tests  84 passed (84)"
echo "exit=$CODE"
exit "$CODE"

Fix — code (prevents future false alarms):

1. Gate the live benchmark behind an env flag so the default suite stays fast and the live run is explicit:

// benchmark.test.ts
const RUN_LIVE = process.env.RUN_LIVE_BENCHMARK === "1";

beforeAll(async () => {
  if (!RUN_LIVE) return;
  // POST /api/v1/benchmark {games: 1, models: ["pair"]}
});

it("GET /api/v1/benchmark/runs shows the live run before diagnosing", { timeout: 30_000 }, async () => {
  if (!RUN_LIVE) return;
  const { runs } = await (await fetch(`${API}/api/v1/benchmark/runs`)).json();
  const live = runs.find((r) => r.id === runId);
  expect(live?.status).toBe("running");          // proves "slow", not "hung"
});

it("waits for the live run to complete (do NOT kill at 5 min)", { timeout: 720_000 }, async () => {
  if (!RUN_LIVE) return;
  // poll GET /api/v1/benchmark/runs every 2 s until status === "completed"
});

1. Raise vitest + CI timeouts to cover the 528 s wall-clock:

// vitest.config.ts
export default defineConfig({
  test: {
    testTimeout: 720_000,   // 12 min — covers the benchmark poll budget
    hookTimeout: 720_000,
  },
});

# .github/workflows/test.yml — job timeout must exceed the slowest suite
jobs:
  test:
    timeout-minutes: 20   # 528 s runtime + margin; a 6-min job timeout kills it
    steps:
      - run: npx vitest run --reporter=verbose > vitest.log 2>&1
      - run: |
          tail -n 20 vitest.log
          grep -q "Tests  *84 passed" vitest.log || exit 1

1. Optional split: move the live benchmark into its own config/test path (vitest.benchmark.config.ts) so the 83 fast tests gate PRs in seconds and the benchmark runs only in the nightly/regression job.