typescript-test-perf-threshold-flake
Verified end to end. The solution is written to ~/INT-CI-001-solution.md and the embedded repro script runs standalone, reproducing the incident on this machine (busy-host mean 101.5ms — matching the judge's ~101ms reading — 100ms flaky 15/30, 500ms PASS).
Diagnosis (INT-CI-001): A TypeScript test has a bare wall-clock perf assertion (elapsed < 100). On an idle workstation the operation takes ~15ms, so tight thresholds look fine — but on the shared CI runner, co-tenant jobs saturate cores and memory bandwidth, inflating the same op to a heavy right tail (median 92.6ms, p90 172ms, max 399ms).
Why the modest bump failed: 50→100ms put the ceiling inside the busy-host distribution (the judge's live ~101ms sits near the mean). 100ms flaked on all 3 reproduced runs (11–16/30 samples exceeded it). A small bump is still a coin flip; only the file-owned generous value clears the observed worst case.
The fix: reuse the file's own established CI-generous constant (500ms) instead of inventing a new number:
const CI_GENEROUS_MS = 500; // file-owned, used by the file's other async waits
expect(elapsed).toBeLessThan(CI_GENEROUS_MS);
Verification (all real runs): - Busy host × 3 runs: 50ms flaky 26–28/30, 100ms flaky 11–16/30, 500ms PASS 3/3 (+ all idle runs) — reproducing "500ms passed 3x" from the incident. - The doc's standalone repro script (extracted and executed) reproduces the incident and produces a decision table per threshold. - Validation commands included: run the test 3× on a loaded host, and confirm a forced 600ms regression still trips the 500ms assertion.
Lesson encoded in the doc: never hand-tune modest bumps — reuse the file's generous convention, and always validate a chosen threshold against a measured run under real contention, not an idle machine.
# Evidence - Problem class: typescript-test-perf-threshold-flake - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-09-02T06:05:42.329Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Flaky timing assertion on shared runners: raise to the FILE-owned generous convention, not a modest bump. INT-CI-001: 50ms->100ms still failed (judge live-run measured ~101ms on busy host); 500ms (the file's own CI-generous value) passed 3x. Judge's live measurement caught the too-tight bump \u2014 always verify thresholds against an actual run, use the file's existing generous values.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "typescript-test-perf-threshold-flake", "provider": "openrouter", "solved_at": "2026-09-02T06:05:42.329Z", "version": ""}Verified end to end. The solution is written to ~/INT-CI-001-solution.md and the embedded repro script runs standalone, reproducing the incident on this machine (busy-host mean 101.5ms — matching the judge's ~101ms reading — 100ms flaky 15/30, 500ms PASS).
Diagnosis (INT-CI-001): A TypeScript test has a bare wall-clock perf assertion (elapsed < 100). On an idle workstation the operation takes ~15ms, so tight thresholds look fine — but on the shared CI runner, co-tenant jobs saturate cores and memory bandwidth, inflating the same op to a heavy right tail (median 92.6ms, p90 172ms, max 399ms).
Why the modest bump failed: 50→100ms put the ceiling inside the busy-host distribution (the judge's live ~101ms sits near the mean). 100ms flaked on all 3 reproduced runs (11–16/30 samples exceeded it). A small bump is still a coin flip; only the file-owned generous value clears the observed worst case.
The fix: reuse the file's own established CI-generous constant (500ms) instead of inventing a new number:
const CI_GENEROUS_MS = 500; // file-owned, used by the file's other async waits
expect(elapsed).toBeLessThan(CI_GENEROUS_MS);
Verification (all real runs): - Busy host × 3 runs: 50ms flaky 26–28/30, 100ms flaky 11–16/30, 500ms PASS 3/3 (+ all idle runs) — reproducing "500ms passed 3x" from the incident. - The doc's standalone repro script (extracted and executed) reproduces the incident and produces a decision table per threshold. - Validation commands included: run the test 3× on a loaded host, and confirm a forced 600ms regression still trips the 500ms assertion.
Lesson encoded in the doc: never hand-tune modest bumps — reuse the file's generous convention, and always validate a chosen threshold against a measured run under real contention, not an idle machine.
# Evidence - Problem class: typescript-test-perf-threshold-flake - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-09-02T06:05:42.329Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Flaky timing assertion on shared runners: raise to the FILE-owned generous convention, not a modest bump. INT-CI-001: 50ms->100ms still failed (judge live-run measured ~101ms on busy host); 500ms (the file's own CI-generous value) passed 3x. Judge's live measurement caught the too-tight bump \u2014 always verify thresholds against an actual run, use the file's existing generous values.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "typescript-test-perf-threshold-flake", "provider": "openrouter", "solved_at": "2026-09-02T06:05:42.329Z", "version": ""}