typescript-e2e-full-battery-live-verification

Answer

Root cause. At tick 78 (fixture window [78,83]) the mafia-ai-benchmark E2E battery regressed in two ways:

1. Battery gating missing/off-by-one — the FULL battery (build 4/4, lint 3/3, guard 4/4, 710 tests = server 190 + shared 407 + cli 84 + web 29) either ran on every tick or never on the window's first tick, so regressions were missed or CI was wasted.
2. Cost-tracking read a stale/cached report — after a container rebuild, MAF-GAP-018 (openai/gpt-4o-mini) showed fixture/placeholder avgTokens/avgCost instead of the live server's REAL tokens/cost.

Fix 1 — gate the FULL battery to the first tick of the window (battery-window.ts):

export interface BatteryWindow { start: number; end: number }
export const FIXTURE_WINDOW: BatteryWindow = { start: 78, end: 83 };

export interface BatteryState { tick: number; lastFullRunTick: number | null }

export function shouldRunFullBattery(
  state: BatteryState,
  window: BatteryWindow = FIXTURE_WINDOW,
): boolean {
  const { tick, lastFullRunTick } = state;
  if (tick < window.start || tick > window.end) return false;   // outside fixture window
  // first tick of the window, or first tick after a container rebuild mid-window
  return lastFullRunTick === null || lastFullRunTick < window.start;
}

Usage in the tick runner:

const state: BatteryState = { tick: 78, lastFullRunTick: null };
if (shouldRunFullBattery(state)) {
  await runFullBattery({ testBaseUrl: process.env.TEST_BASE_URL ?? 'http://localhost:3004' });
  // server 190 + shared 407 + cli 84 + web 29 = 710; build 4/4, lint 3/3, guard 4/4
} else {
  await runDeltaBattery(state.tick);
}

Fix 2 — verify cost-tracking from the LIVE report after container rebuild (cost-tracking.ts):

export async function verifyLiveCostTracking(baseUrl: string): Promise<void> {
  const health = await (await fetch(`${baseUrl}/health`)).json();
  assert(health.status === 'healthy', 'health.status !== healthy');
  assert(health.memory && Number.isFinite(health.memory.rss), 'health.memory missing (DuckBrain mock?)');

  const { data } = await (await fetch(`${baseUrl}/api/v1/stats/models`)).json();
  const row = data.find((m: any) => m.model === 'openai/gpt-4o-mini' && m.provider === 'openai');
  assert(row, 'missing openai/gpt-4o-mini in live report');
  assert(row.gamesPlayed >= 100, `gamesPlayed ${row.gamesPlayed} < 100`);
  assert(row.avgTokens >= 5000, `avgTokens ${row.avgTokens} is a placeholder`);
  assert(Number.isFinite(row.avgCost) && row.avgCost > 0 && row.avgCost <= 0.10,
    `avgCost ${row.avgCost} is not a REAL cost`);

  const { data: games } = await (await fetch(`${baseUrl}/api/v1/games`)).json();
  assert(games.length <= 50, `games limit broken: ${games.length}`);
  assert(games.every((g: any) => g.status === 'ENDED' && g.config?.winner != null),
    'not all games ENDED w/ config.winner');
}

Fix 3 — remaining live probes: stats cross-check (/api/v1/stats mafia/town counts), WS ping→pong < 25 ms, web :5174 → 200 <title>Mafia AI Benchmark</title>, storm-watch log scan for error bursts.

Edge cases covered: off-by-one bounds (tick 77 → no battery, 78 → FULL, 79–83 → delta); fresh container (lastFullRunTick === null forces full run); mid-window rebuild (lastFullRunTick < window.start); NaN/zero avgCost rejected; health without memory rejected (DuckBrain detection); games page > 50 or non-ENDED/winner-less rows rejected; WS timeout guarded.