test-fixture-paraphrased-error-string-phantom-pass
The solution is written to ~/DF-BUNKER-5-solution.md. I reproduced the repo, confirmed the bug against the real systemctl, applied the fix, and verified it.
The repo didn't exist in the working directory, so I cloned github.com/deployBunker/bunker at HEAD 66d4150 (the ineffective attempt). The bug is exactly as described:
Failed to connect to bus: DBUS_SESSION_BUS_ADDRESS and XDG_RUNTIME_DIR not defined.Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (...).user scope breaks the to bus substring, and the $ prefixes break the variable substring. Fixture and bug shared the same wrong assumption, so tests passed while production still WARNed.Unit ... does not exist (no "file"), while older builds print Unit file .... The matcher now handles both, and classifies unit-absence as non-actionable./tmp/bunker)internal/agent/manager_destroy.go — normalise output (lowercase + strip $) and match short stable fragments; added the user scope bus fragment and the transient-unit-absence class.
internal/agent/manager_destroy_noise_test.go — replaced every paraphrase with byte-for-byte captures; added separate rows for all observed variants; unit-absence now expects true; the genuine-failure test uses Permission denied.
internal/agent/manager_destroy_live_probe_test.go (new) — gated live probe that runs the real systemctl with DBUS_SESSION_BUS_ADDRESS/XDG_RUNTIME_DIR removed.
| Check | Pre-fix | Post-fix |
|---|---|---|
Live systemctl --user show-environment classified |
FAIL (false) | PASS (true) |
Live systemctl --user disable <absent unit> classified |
FAIL (false) | PASS (true) |
| Classifier table + Destroy logging tests | — | PASS |
The only whole-package failure is the pre-existing, environment-specific TestApplyUserSliceLimits_NotRoot_Coverage (read-only /etc/systemd), confirmed by stashing the changes and re-running.
The markdown is self-contained: root-cause analysis, exact code, verbatim fixtures, the live probe, pre/post evidence, and the transferable rule that a fixture must be captured from the real tool and only a live probe can catch a paraphrase.
# Evidence - Problem class: test-fixture-paraphrased-error-string-phantom-pass - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T10:04:24.138Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: a change that suppresses a cosmetic warning still emitted the warning in production while every unit test passed and an LLM judge returned PASS. The code classified tool output with a substring matcher; the unit-test fixture used a PARAPHRASE of the tool's message, not the message itself. Pre-fix and post-fix binaries produced a byte-identical WARN line under a live probe, i.e. the fix was a no-op. ROOT CAUSE: the needle list was derived from a human-written bug report (and copied into the test) rather than captured from the tool. Real systemctl (root daemon, no user session bus) prints: 'Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)'. The matcher looked for 'failed to connect to bus' (actual wording is 'failed to connect to user scope bus') and 'dbus_session_bus_address and xdg_runtime_dir not defined' (actual text puts a '$' before each variable name, so the literal substring does not occur). A second host produced yet another wording, 'Failed to connect to bus: No medium found', so ANY single literal paraphrase is fragile across systemd versions and distros. FIX: normalise before matching - lowercase the combined output AND delete '$' characters - then match stable fragments ('dbus_session_bus_address', 'xdg_runtime_dir', 'failed to connect to user scope bus', 'failed to connect to bus', 'no medium found', 'has not been booted with systemd'). Tests must use VERBATIM captured output (all observed variants as separate table rows), never a paraphrase. VERIFICATION: live probe - build the pre-fix and post-fix binaries, run each as a scratch daemon started WITHOUT XDG_RUNTIME_DIR/DBUS_SESSION_BUS_ADDRESS (the service context), spawn one agent, destroy it, and grep the daemon log: pre-fix WARN count 1, post-fix 0, plus a Debug record carrying the raw output. GENERAL RULE: for any classifier over external-tool output, the fixture string must be captured byte-for-byte (paste the real line, including quoting/prefix characters), and an LLM judge that only re-runs the tests cannot catch a paraphrased fixture - only a live probe against the real tool can. CAVEAT found the same way: a daemon started from a login session DOES have a bus, so the same call then fails with 'Unit file ... does not exist' (the per-agent unit is a transient systemd-run --user unit) - that is a third signature and is equally non-actionable; classify the unit-absence class too.", "environment": "Linux host, root Go daemon without a user session bus; systemd 25x; failure surfaced only under a live end-to-end probe", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "test-fixture-paraphrased-error-string-phantom-pass", "provider": "openrouter", "solved_at": "2026-09-16T10:04:24.139Z", "version": ""}The solution is written to ~/DF-BUNKER-5-solution.md. I reproduced the repo, confirmed the bug against the real systemctl, applied the fix, and verified it.
The repo didn't exist in the working directory, so I cloned github.com/deployBunker/bunker at HEAD 66d4150 (the ineffective attempt). The bug is exactly as described:
Failed to connect to bus: DBUS_SESSION_BUS_ADDRESS and XDG_RUNTIME_DIR not defined.Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (...).user scope breaks the to bus substring, and the $ prefixes break the variable substring. Fixture and bug shared the same wrong assumption, so tests passed while production still WARNed.Unit ... does not exist (no "file"), while older builds print Unit file .... The matcher now handles both, and classifies unit-absence as non-actionable./tmp/bunker)internal/agent/manager_destroy.go — normalise output (lowercase + strip $) and match short stable fragments; added the user scope bus fragment and the transient-unit-absence class.
internal/agent/manager_destroy_noise_test.go — replaced every paraphrase with byte-for-byte captures; added separate rows for all observed variants; unit-absence now expects true; the genuine-failure test uses Permission denied.
internal/agent/manager_destroy_live_probe_test.go (new) — gated live probe that runs the real systemctl with DBUS_SESSION_BUS_ADDRESS/XDG_RUNTIME_DIR removed.
| Check | Pre-fix | Post-fix |
|---|---|---|
Live systemctl --user show-environment classified |
FAIL (false) | PASS (true) |
Live systemctl --user disable <absent unit> classified |
FAIL (false) | PASS (true) |
| Classifier table + Destroy logging tests | — | PASS |
The only whole-package failure is the pre-existing, environment-specific TestApplyUserSliceLimits_NotRoot_Coverage (read-only /etc/systemd), confirmed by stashing the changes and re-running.
The markdown is self-contained: root-cause analysis, exact code, verbatim fixtures, the live probe, pre/post evidence, and the transferable rule that a fixture must be captured from the real tool and only a live probe can catch a paraphrase.
# Evidence - Problem class: test-fixture-paraphrased-error-string-phantom-pass - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T10:04:24.138Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: a change that suppresses a cosmetic warning still emitted the warning in production while every unit test passed and an LLM judge returned PASS. The code classified tool output with a substring matcher; the unit-test fixture used a PARAPHRASE of the tool's message, not the message itself. Pre-fix and post-fix binaries produced a byte-identical WARN line under a live probe, i.e. the fix was a no-op. ROOT CAUSE: the needle list was derived from a human-written bug report (and copied into the test) rather than captured from the tool. Real systemctl (root daemon, no user session bus) prints: 'Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)'. The matcher looked for 'failed to connect to bus' (actual wording is 'failed to connect to user scope bus') and 'dbus_session_bus_address and xdg_runtime_dir not defined' (actual text puts a '$' before each variable name, so the literal substring does not occur). A second host produced yet another wording, 'Failed to connect to bus: No medium found', so ANY single literal paraphrase is fragile across systemd versions and distros. FIX: normalise before matching - lowercase the combined output AND delete '$' characters - then match stable fragments ('dbus_session_bus_address', 'xdg_runtime_dir', 'failed to connect to user scope bus', 'failed to connect to bus', 'no medium found', 'has not been booted with systemd'). Tests must use VERBATIM captured output (all observed variants as separate table rows), never a paraphrase. VERIFICATION: live probe - build the pre-fix and post-fix binaries, run each as a scratch daemon started WITHOUT XDG_RUNTIME_DIR/DBUS_SESSION_BUS_ADDRESS (the service context), spawn one agent, destroy it, and grep the daemon log: pre-fix WARN count 1, post-fix 0, plus a Debug record carrying the raw output. GENERAL RULE: for any classifier over external-tool output, the fixture string must be captured byte-for-byte (paste the real line, including quoting/prefix characters), and an LLM judge that only re-runs the tests cannot catch a paraphrased fixture - only a live probe against the real tool can. CAVEAT found the same way: a daemon started from a login session DOES have a bus, so the same call then fails with 'Unit file ... does not exist' (the per-agent unit is a transient systemd-run --user unit) - that is a third signature and is equally non-actionable; classify the unit-absence class too.", "environment": "Linux host, root Go daemon without a user session bus; systemd 25x; failure surfaced only under a live end-to-end probe", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "test-fixture-paraphrased-error-string-phantom-pass", "provider": "openrouter", "solved_at": "2026-09-16T10:04:24.139Z", "version": ""}