ts-idle-audit-probe-drift
The repo was not present in this environment — only the problem spec (/workspace/problem.json) — so I reconstructed the system from the description and applied the two fixes.
Fix 1 — Heading idle audit tick: assert against the canonical seed, not prior board entries.
The probe's expected count was copied forward from the previous tick's board entry (prev.probes.find(p => p.name === "clients").expected), so a stale 3 from an ancient tick (when the seed had 3 clients) propagated forever. The seed (packages/crm/data/clients.json + src/seed.ts) now has 5.
Before (drift source — expectation borrowed from the board):
const prev = board[board.length - 1];
const expected =
prev?.probes.find((p) => p.name === "clients")?.expected ?? 3; // stale copy-forward
After (expectation derived from the seed at probe time):
// packages/crm/src/seed.ts — single source of truth
export const SEED_CLIENTS: ClientSeed[] = loadClientSeed(); // reads data/clients.json
// audits/heading/heading-audit.ts — fixed probe
const expected = SEED_CLIENTS.length; // 5, from clients.json + seed.ts
const { clients } = await apiGet("/api/clients"); // live probe
assert(clients.length === expected,
`GET /api/clients returned ${clients.length}, canonical crm seed has ${expected}`);
Board entries become output records only; they are never consulted for expectations.
Fix 2 — Chat-flow probe: count payload.type sub-events, not the top-level envelope type.
Every WS event is a phase_change envelope, so bucketing by e.type collapses everything into one bucket ({ phase_change: 6 }) and the sub-event discriminators (option_update — singular — and surface_complete) are lost.
Before (wrong bucket key):
for (const evt of events) {
counts[evt.type] = (counts[evt.type] ?? 0) + 1; // always "phase_change"
}
After (bucket by sub-event):
for (const evt of events) {
if (evt.type === "phase_change" && evt.payload?.type) {
const sub = evt.payload.type; // "option_update" | "surface_complete" | ...
counts[sub] = (counts[sub] ?? 0) + 1;
}
}
// then assert exact counts: option_update===N, surface_complete===M, ...
I built a runnable miniature (`/tmp/audit-fix/`) with the canonical seed (`packages/crm/data/clients.json` — 5 clients, loaded by `src/seed.ts`), both the buggy and fixed probe implementations, and executed it with Node 22 (`node --experimental-strip-types test.ts`): **16/16 checks passed.**
Key results:
- **Drift reproduced:** a legacy board tick had `expected=3`; the next buggy tick copied `3` forward and false-failed against the live API (`expected=3 actual=5 -> false`). The fixed tick derived `expected=5` from `SEED_CLIENTS` and passed (`5/5 -> true`).
- **Stability:** across a second tick, buggy stayed frozen at `3` while fixed kept deriving `5` from the seed — no board-state dependency.
- **Chat flow:** a stream of 6 envelopes (3 `option_update`, 2 `surface_complete`, 1 `phase_enter`) produced `{phase_change:6}` under the buggy bucketing vs `{option_update:3, surface_complete:2, phase_enter:1}` under the fix — exact-match assertion helper agreed.
- **Edge cases:** empty stream → all-zero counts without throwing; envelope missing `payload.type` → skipped safely; exact sub-event names (`option_update` singular, not `option_updates`) matched byte-for-byte.
The fix means future seed changes (e.g., a 6th client added to `clients.json`) are picked up automatically by the probe, and chat-flow metrics are per-sub-event instead of one flat `phase_change` number.{"model": "deepseek-v4-flash", "problem_class": "ts-idle-audit-probe-drift", "result": "passed", "tests": 16}