typescript-stats-aggregation-key-normalization

Answer

The bug had three independent causes, fixed in stats-aggregation.ts (full file at /tmp/mafia-stats/stats-aggregation.ts):

1. Key normalization — rows carried a provider prefix on the model column, so keying on raw provider/model produced openai/openai/gpt-4o-mini that never merged with openai/gpt-4o-mini:

// strip a "<provider>/" prefix before GROUP BY/merge
export function normalizeModelKey(
  model: string,
  provider?: string,
  knownProviders: readonly string[] = DEFAULT_PROVIDER_PREFIXES,
): string {
  const m = (model ?? "").trim();
  if (!m) return m;
  const p = (provider ?? "").trim();

  // 1) exact provider prefix, repeated (collapses double-writes too)
  if (p) {
    const prefix = `${p.toLowerCase()}/`;
    if (m.toLowerCase().startsWith(prefix)) {
      let out = m;
      while (out.length > p.length && out.toLowerCase().startsWith(prefix))
        out = out.slice(p.length + 1);
      return out;
    }
  }
  // 2) any observed provider prefix (provider column missing / spelled differently)
  for (const k of knownProviders) {
    const prefix = `${k.toLowerCase()}/`;
    if (m.toLowerCase().startsWith(prefix)) {
      let out = m;
      while (out.length > k.length && out.toLowerCase().startsWith(prefix))
        out = out.slice(k.length + 1);
      return out;
    }
  }
  return m;
}

The key becomes provider.toLowerCase() + "/" + normalizeModelKey(model, provider, known) — so openai/gpt-4o-mini and gpt-4o-mini (both provider=openai) collapse onto openai/gpt-4o-mini, and missing-provider rows infer the provider from the model prefix:

export function inferProvider(row: PlayerRow, knownProviders: readonly string[]): string {
  const p = (row.provider ?? "").trim();
  if (p) return p;
  const lower = (row.model ?? "").trim().toLowerCase();
  for (const k of knownProviders)
    if (lower.startsWith(`${k.toLowerCase()}/`)) return k;
  return "unknown";
}

2. Wins only from real per-player rows — the game-level-winner fallback is deleted. buildReport never receives the games table (the SQL drops the winner join), and wins are counted strictly from players.won === 1:

// SQL feeding the report — the games table / winner column are NOT selected:
//   SELECT game_id, player_id, provider, model, role, won,
//          tokens_in, tokens_out, cost_usd, latency_ms
//   FROM players ORDER BY id;

const won = row.won === 1 ? 1 : 0;   // the ONLY source of wins
r.wins += won;
r.winRate = r.games > 0 ? r.wins / r.games : 0;

3. First/most-complete source on key collision — numeric metrics are always summed (nothing dropped); display metadata keeps the first source unless an incoming row is strictly more complete:

const sourceScoreBy = new WeakMap<ReportRow, number>();

function mergeRow(r: ReportRow, row: PlayerRow, knownProviders: readonly string[]): void {
  // ... r.games++, r.wins += won, metrics += ... (never dropped)
  const incoming = completenessScore(row);   // count of populated fields
  if (incoming > (sourceScoreBy.get(r) ?? 0)) {
    r.model = normalizeModelKey(row.model, row.provider, knownProviders);
    r.provider = inferProvider(row, knownProviders);
    sourceScoreBy.set(r, incoming);          // strictly more complete → replaces
  }
}

knownProviders is data-driven (collectProviders: static safe list ∪ providers observed in the dataset), so a legitimately-slashed id like meta-llama/llama-3.1-70b is never corrupted unless meta-llama is actually a provider.