python-test-tool-discovery-path-hermeticity
I diagnosed it against the real repository (cloned totalwindupflightsystems/gitreins, inspected fix commits 37a465f and 208f2a2) and empirically reproduced both the failure and the fix.
The full write-up is at ~/solution.md. Summary:
find_tool() resolves binaries via shutil.which(...), which reads the ambient PATH. The original tests asserted positive discovery without controlling that input, so they effectively asserted "the host has mypy/pyright/cppcheck/staticcheck installed." Green on GitHub runners, red in the tool-sparse bunker.
A prior narrow fix mocked only list_available_tools, but TestFindTool calls find_tool directly, so it kept hitting the real shutil.which. There are two seams:
| Test class | Function under test | Host-reading dependency |
|---|---|---|
TestFindTool |
find_tool |
shutil.which |
TestListAvailableTools |
list_available_tools |
find_tool |
TestFindTool: patch engine.static_analysis.shutil.which, assert candidate order (["pyright", "npx"]), compound npx pyright return, unknown-tool fallback, controlled missing case.TestListAvailableTools: patch engine.static_analysis.find_tool, assert exact language→tool mapping plus negative/partial availability so the find-only-installed contract stays covered.1447 → 1449.engine/static_analysis.py unchanged.PATH=/usr/local/bin:/bin: 3 failed (reproduces assert None is not None and missing cppcheck/staticcheck).test_static_analysis.py: 84 passed.--collect-only total: 1449, matching README.# Evidence - Problem class: python-test-tool-discovery-path-hermeticity - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-14T12:14:44.936Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Symptom: a Python unit-test suite passed on GitHub-hosted CI but failed on a clean bunker because TestFindTool and TestListAvailableTools directly required mypy, pyright, cppcheck, and staticcheck on the host PATH. A first narrow fix mocked only list_available_tools and still left find_tool tests host-dependent; a restricted-PATH full-suite run exposed the residual. Root cause: positive-path tests exercised shutil.which against ambient machine state instead of controlling the discovery seam. Fix: patch engine.static_analysis.find_tool in language-mapping tests, patch engine.static_analysis.shutil.which in find_tool tests, assert candidate lookup order and negative/partial availability, and leave production discovery code byte-identical. Verification: PATH limited to the project venv and /usr/local/bin produced 12 focused passes and 84 full-file passes; the complete suite passed 1433 tests with 16 skipped; GitReins Tier 1+2 PASS; GitHub Actions run 34841372304 passed Python 3.10, 3.11, and 3.12. Secondary release discipline: adding two collected tests required synchronizing README collection-count claims from 1447 to 1449 before CI could become green.", "environment": "pytest 9.1.1 with xdist; GitHub Actions plus a tool-sparse Linux bunker", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "python-test-tool-discovery-path-hermeticity", "provider": "openrouter", "solved_at": "2026-09-14T12:14:44.937Z", "version": "3.10-3.12"}I diagnosed it against the real repository (cloned totalwindupflightsystems/gitreins, inspected fix commits 37a465f and 208f2a2) and empirically reproduced both the failure and the fix.
The full write-up is at ~/solution.md. Summary:
find_tool() resolves binaries via shutil.which(...), which reads the ambient PATH. The original tests asserted positive discovery without controlling that input, so they effectively asserted "the host has mypy/pyright/cppcheck/staticcheck installed." Green on GitHub runners, red in the tool-sparse bunker.
A prior narrow fix mocked only list_available_tools, but TestFindTool calls find_tool directly, so it kept hitting the real shutil.which. There are two seams:
| Test class | Function under test | Host-reading dependency |
|---|---|---|
TestFindTool |
find_tool |
shutil.which |
TestListAvailableTools |
list_available_tools |
find_tool |
TestFindTool: patch engine.static_analysis.shutil.which, assert candidate order (["pyright", "npx"]), compound npx pyright return, unknown-tool fallback, controlled missing case.TestListAvailableTools: patch engine.static_analysis.find_tool, assert exact language→tool mapping plus negative/partial availability so the find-only-installed contract stays covered.1447 → 1449.engine/static_analysis.py unchanged.PATH=/usr/local/bin:/bin: 3 failed (reproduces assert None is not None and missing cppcheck/staticcheck).test_static_analysis.py: 84 passed.--collect-only total: 1449, matching README.# Evidence - Problem class: python-test-tool-discovery-path-hermeticity - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-14T12:14:44.936Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Symptom: a Python unit-test suite passed on GitHub-hosted CI but failed on a clean bunker because TestFindTool and TestListAvailableTools directly required mypy, pyright, cppcheck, and staticcheck on the host PATH. A first narrow fix mocked only list_available_tools and still left find_tool tests host-dependent; a restricted-PATH full-suite run exposed the residual. Root cause: positive-path tests exercised shutil.which against ambient machine state instead of controlling the discovery seam. Fix: patch engine.static_analysis.find_tool in language-mapping tests, patch engine.static_analysis.shutil.which in find_tool tests, assert candidate lookup order and negative/partial availability, and leave production discovery code byte-identical. Verification: PATH limited to the project venv and /usr/local/bin produced 12 focused passes and 84 full-file passes; the complete suite passed 1433 tests with 16 skipped; GitReins Tier 1+2 PASS; GitHub Actions run 34841372304 passed Python 3.10, 3.11, and 3.12. Secondary release discipline: adding two collected tests required synchronizing README collection-count claims from 1447 to 1449 before CI could become green.", "environment": "pytest 9.1.1 with xdist; GitHub Actions plus a tool-sparse Linux bunker", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "python-test-tool-discovery-path-hermeticity", "provider": "openrouter", "solved_at": "2026-09-14T12:14:44.937Z", "version": "3.10-3.12"}