vitest-mutation-test-proving-fallback-test-realness
Wrote the solution to ~/SOLUTION.md.
The repo itself isn't present in this sandbox (~ is empty, only /workspace/problem.json), so I reproduced the load-bearing mechanism locally and verified it before writing the deliverable.
Diagnosis
- The ladder tests can pass for the wrong reason because they assert a helper's return instead of executing the ordered walk. The only proof is a red-proof mutation.
- Naively mutating a hardlinked sandbox file with open(path,'w') truncates the shared inode and dirties the tracked tree — the exact trap.
- A full monorepo copy is unnecessary: tests/vitest.config.js resolves open-sse/ and @/ relative to itself, so a sandbox mirroring only open-sse, src, tests routes the aliases to the mutant.
cp -al hardlink-farm only open-sse, src, tests into /tmp/<tick>-sbx.node_modules so the toolchain resolves (not copied).os.unlink(path) first, then open(path,'w') — this breaks the hardlink so the tracked inode is untouched.$SBX/tests so relative aliases load the sandbox mutant.git status --short, grep -Fc <marker> on the tracked path, and stat -c %h (link count back to 1).Local verification output confirmed: tracked inode stayed at 643 (links 2→1), sandbox mutant got a new inode 673, sandbox test printed MUTANT_BROKE_AFTER_ONE_HOP, real test printed REAL_LOOP, and git status was empty.
The document includes the exact harness script, a realistic vitest 4 ladder test that mocks only the transport seam, the checkFallbackError 429/400 control, expected red-proof results (4/5 ladder fail, 4 original green), and a traps checklist (cross-device hardlinks, cache writes, sed -i, precise mutation point).
# Evidence - Problem class: vitest-mutation-test-proving-fallback-test-realness - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-19T00:32:32.853Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A repo advertises an 'Auto fallback: Subscription -> Cheap -> Free' routing ladder but has no multi-hop test. You add tests that drive an ordered combo (provider/model list) through two failures and assert the provider selected at each hop, mocking only the transport seam. The risk: the test may pass for the wrong reason (it asserts a static helper return, not the real walk). You must falsify the new tests yourself without dirtying the tracked tree.", "environment": "Node ESM + vitest 4; tests live in a nested package (tests/) whose vitest.config.js resolves aliases relative to itself (open-sse/ -> ../open-sse, @/ -> ../src). Git tree must stay clean.", "language": "javascript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "vitest-mutation-test-proving-fallback-test-realness", "provider": "openrouter", "solved_at": "2026-09-19T00:32:32.853Z", "version": ""}Wrote the solution to ~/SOLUTION.md.
The repo itself isn't present in this sandbox (~ is empty, only /workspace/problem.json), so I reproduced the load-bearing mechanism locally and verified it before writing the deliverable.
Diagnosis
- The ladder tests can pass for the wrong reason because they assert a helper's return instead of executing the ordered walk. The only proof is a red-proof mutation.
- Naively mutating a hardlinked sandbox file with open(path,'w') truncates the shared inode and dirties the tracked tree — the exact trap.
- A full monorepo copy is unnecessary: tests/vitest.config.js resolves open-sse/ and @/ relative to itself, so a sandbox mirroring only open-sse, src, tests routes the aliases to the mutant.
cp -al hardlink-farm only open-sse, src, tests into /tmp/<tick>-sbx.node_modules so the toolchain resolves (not copied).os.unlink(path) first, then open(path,'w') — this breaks the hardlink so the tracked inode is untouched.$SBX/tests so relative aliases load the sandbox mutant.git status --short, grep -Fc <marker> on the tracked path, and stat -c %h (link count back to 1).Local verification output confirmed: tracked inode stayed at 643 (links 2→1), sandbox mutant got a new inode 673, sandbox test printed MUTANT_BROKE_AFTER_ONE_HOP, real test printed REAL_LOOP, and git status was empty.
The document includes the exact harness script, a realistic vitest 4 ladder test that mocks only the transport seam, the checkFallbackError 429/400 control, expected red-proof results (4/5 ladder fail, 4 original green), and a traps checklist (cross-device hardlinks, cache writes, sed -i, precise mutation point).
# Evidence - Problem class: vitest-mutation-test-proving-fallback-test-realness - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-19T00:32:32.853Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "A repo advertises an 'Auto fallback: Subscription -> Cheap -> Free' routing ladder but has no multi-hop test. You add tests that drive an ordered combo (provider/model list) through two failures and assert the provider selected at each hop, mocking only the transport seam. The risk: the test may pass for the wrong reason (it asserts a static helper return, not the real walk). You must falsify the new tests yourself without dirtying the tracked tree.", "environment": "Node ESM + vitest 4; tests live in a nested package (tests/) whose vitest.config.js resolves aliases relative to itself (open-sse/ -> ../open-sse, @/ -> ../src). Git tree must stay clean.", "language": "javascript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "vitest-mutation-test-proving-fallback-test-realness", "provider": "openrouter", "solved_at": "2026-09-19T00:32:32.853Z", "version": ""}