vitest-integration-timeout-host-load-saturation
Solution written to ~/solution.md.
Diagnosis: Not a code bug. The shared host was saturated — load 77.65 on 16 CPUs, 33 GiB swap in use — by three git pack-objects (namespace packing) and multiple C++ compilers. Vitest's testTimeout is wall-clock, not CPU-time, so synchronous Docker/HTTP cases blocked in uninterruptible I/O hit 60 s without making progress. The moving target (different unrelated files failing on consecutive runs) and the fully green rerun on the unchanged commit 2bbbeac confirm resource starvation.
Fix (operational):
1. Confirm contention: uptime, nproc, free -h, vmstat 1 5, top+D-state ps.
2. Identify offenders: pgrep -a -f 'pack-objects', pgrep -a -x cc1plus/c++/clang.
3. Wait for them to exit with a polling loop (don't kill other tenants), then re-check load/swap.
4. Run the suite serially: pnpm test:integration -- --no-file-parallelism --maxWorkers=1 --minWorkers=1 --bail=1.
5. Require a full green rerun of 2bbbeac with a clean tree.
Verification: 6/6 files, 44/44 tests passed with no source changes.
The markdown also includes optional hardening (cgroup throttling via systemd-run, host-health gating, serialize-by-default) that prevents recurrence without masking the true cause.
# Evidence - Problem class: vitest-integration-timeout-host-load-saturation - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-11T05:28:21.284Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Integration tests timed out in different unrelated synchronous Docker/HTTP cases on consecutive runs. The host was saturated (load average 77.65 on 16 CPUs, 33 GiB swap in use) by three git pack-objects processes and multiple C++ compilers. No product-code change was needed: wait for the identified heavy processes to exit, run the integration suite serially, and require a full green rerun. The same unchanged commit then passed all 6 files and 44 tests.", "environment": "Linux, 16 CPUs, shared foreman host with concurrent namespace git packing and compiler jobs", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "vitest-integration-timeout-host-load-saturation", "provider": "openrouter", "solved_at": "2026-09-11T05:28:21.285Z", "version": "Node 22; Vitest 4.1.10"}Solution written to ~/solution.md.
Diagnosis: Not a code bug. The shared host was saturated — load 77.65 on 16 CPUs, 33 GiB swap in use — by three git pack-objects (namespace packing) and multiple C++ compilers. Vitest's testTimeout is wall-clock, not CPU-time, so synchronous Docker/HTTP cases blocked in uninterruptible I/O hit 60 s without making progress. The moving target (different unrelated files failing on consecutive runs) and the fully green rerun on the unchanged commit 2bbbeac confirm resource starvation.
Fix (operational):
1. Confirm contention: uptime, nproc, free -h, vmstat 1 5, top+D-state ps.
2. Identify offenders: pgrep -a -f 'pack-objects', pgrep -a -x cc1plus/c++/clang.
3. Wait for them to exit with a polling loop (don't kill other tenants), then re-check load/swap.
4. Run the suite serially: pnpm test:integration -- --no-file-parallelism --maxWorkers=1 --minWorkers=1 --bail=1.
5. Require a full green rerun of 2bbbeac with a clean tree.
Verification: 6/6 files, 44/44 tests passed with no source changes.
The markdown also includes optional hardening (cgroup throttling via systemd-run, host-health gating, serialize-by-default) that prevents recurrence without masking the true cause.
# Evidence - Problem class: vitest-integration-timeout-host-load-saturation - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-11T05:28:21.284Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Integration tests timed out in different unrelated synchronous Docker/HTTP cases on consecutive runs. The host was saturated (load average 77.65 on 16 CPUs, 33 GiB swap in use) by three git pack-objects processes and multiple C++ compilers. No product-code change was needed: wait for the identified heavy processes to exit, run the integration suite serially, and require a full green rerun. The same unchanged commit then passed all 6 files and 44 tests.", "environment": "Linux, 16 CPUs, shared foreman host with concurrent namespace git packing and compiler jobs", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "vitest-integration-timeout-host-load-saturation", "provider": "openrouter", "solved_at": "2026-09-11T05:28:21.285Z", "version": "Node 22; Vitest 4.1.10"}