stale-installed-cli-masquerades-as-live-bug
Solution written to ~/stale-installed-cli-masquerades-as-live-bug.md.
Root cause: not a defect at HEAD. The bunker on PATH (~/go/bin/bunker) was a stale v0.1.3 build, so the already-landed rewriteSSHFSMount fix in internal/cli/sshhost.go was never executed. It kept emitting daemon-local argv (IdentityFile=/etc/bunkerd/ssh/<id>, host bunker-mvp) instead of client-local (~/.bunker/keys/<id>, resolved IP). A HEAD rebuild (v0.1.4) made the same command succeed. The same day, the server's /usr/local/bin/bunker was 4 commits behind /opt/bunker, so the E2E battery was certifying a stale binary.
Fix: no source change — rebuild/install CLI and daemon from HEAD with stamped version/commit, restart bunkerd, then re-verify.
Key diagnostic lessons documented:
1. Print the binary's own identity (bunker version, <bin> --version) and compare to git rev-parse HEAD before diagnosing code.
2. Confirm the actual code path via strace -f -qq -e trace=execve -s 600 rather than reasoning from source.
3. Prevention: force version/commit stamping in builds, log daemon identity at startup, and assert bunker version commit == HEAD at the start of E2E/CI.
The environment here independently corroborates the problem class: /usr/local/bin/bunker -> ~/go/bin/bunker is a dangling symlink and /opt/bunker (the daemon's systemd ExecStart) is missing — i.e. the executed/installed artifacts do not correspond to any current source.
The doc includes the full diagnosis workflow, exact rebuild commands, before/after strace argv expectations, and end-to-end mount verification.
# Evidence - Problem class: stale-installed-cli-masquerades-as-live-bug - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T11:01:01.379Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: `bunker mount <id> <dir>` against a healthy, running agent failed with 'read: Connection reset by peer' / 'sshfs failed: exit status 1', while a MANUAL sshfs with the same key and host mounted the same agent fine. Strace of the CLI's execve showed the argv it actually ran: sshfs -o IdentityFile=/etc/bunkerd/ssh/<id> ... bunker-<id>@bunker-mvp:~<id> <dir> - i.e. the DAEMON-local key path and the daemon's own hostname, not the client-local key (~/.bunker/keys/<id>) and the resolved address. That looked like the client-side rewrite (internal/cli/sshhost.go rewriteSSHFSMount) silently doing nothing. ROOT CAUSE: it was NOT a code defect at HEAD. The box had a stale installed CLI: ~/go/bin/bunker reported 'bunker 0.1.3 commit v0.1.3' while the repo HEAD built to 0.1.4; the rewrite fix had already landed earlier. Rebuilding from HEAD and re-running the same command produced the correct argv (IdentityFile=~/.bunker/keys/<id>, host <ip-address>) and the mount SUCCEEDED. LESSON 1: before diagnosing a fix as broken, print the binary's own version/commit and compare with repo HEAD (bunker version; <bin> --version) - a stale build makes fixed code look live-broken. LESSON 2: confirm which code path produced the argv by tracing the actual exec (strace -f -qq -e trace=execve -s 600 -o out <cmd>), instead of reasoning from the source you just read. LESSON 3: same class on the server side that same day - the demo host's /usr/local/bin/bunker was 4 commits behind (b4c3c48) while /opt/bunker ran a newer build, so the E2E battery was certifying a stale binary. Both were fixed by rebuilding/installing at HEAD.", "environment": "Linux x86_64, Go 1.26, bunker CLI v0.1.3 installed vs v0.1.4 at repo HEAD", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "stale-installed-cli-masquerades-as-live-bug", "provider": "openrouter", "solved_at": "2026-09-17T11:01:01.379Z", "version": ""}Solution written to ~/stale-installed-cli-masquerades-as-live-bug.md.
Root cause: not a defect at HEAD. The bunker on PATH (~/go/bin/bunker) was a stale v0.1.3 build, so the already-landed rewriteSSHFSMount fix in internal/cli/sshhost.go was never executed. It kept emitting daemon-local argv (IdentityFile=/etc/bunkerd/ssh/<id>, host bunker-mvp) instead of client-local (~/.bunker/keys/<id>, resolved IP). A HEAD rebuild (v0.1.4) made the same command succeed. The same day, the server's /usr/local/bin/bunker was 4 commits behind /opt/bunker, so the E2E battery was certifying a stale binary.
Fix: no source change — rebuild/install CLI and daemon from HEAD with stamped version/commit, restart bunkerd, then re-verify.
Key diagnostic lessons documented:
1. Print the binary's own identity (bunker version, <bin> --version) and compare to git rev-parse HEAD before diagnosing code.
2. Confirm the actual code path via strace -f -qq -e trace=execve -s 600 rather than reasoning from source.
3. Prevention: force version/commit stamping in builds, log daemon identity at startup, and assert bunker version commit == HEAD at the start of E2E/CI.
The environment here independently corroborates the problem class: /usr/local/bin/bunker -> ~/go/bin/bunker is a dangling symlink and /opt/bunker (the daemon's systemd ExecStart) is missing — i.e. the executed/installed artifacts do not correspond to any current source.
The doc includes the full diagnosis workflow, exact rebuild commands, before/after strace argv expectations, and end-to-end mount verification.
# Evidence - Problem class: stale-installed-cli-masquerades-as-live-bug - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T11:01:01.379Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: `bunker mount <id> <dir>` against a healthy, running agent failed with 'read: Connection reset by peer' / 'sshfs failed: exit status 1', while a MANUAL sshfs with the same key and host mounted the same agent fine. Strace of the CLI's execve showed the argv it actually ran: sshfs -o IdentityFile=/etc/bunkerd/ssh/<id> ... bunker-<id>@bunker-mvp:~<id> <dir> - i.e. the DAEMON-local key path and the daemon's own hostname, not the client-local key (~/.bunker/keys/<id>) and the resolved address. That looked like the client-side rewrite (internal/cli/sshhost.go rewriteSSHFSMount) silently doing nothing. ROOT CAUSE: it was NOT a code defect at HEAD. The box had a stale installed CLI: ~/go/bin/bunker reported 'bunker 0.1.3 commit v0.1.3' while the repo HEAD built to 0.1.4; the rewrite fix had already landed earlier. Rebuilding from HEAD and re-running the same command produced the correct argv (IdentityFile=~/.bunker/keys/<id>, host <ip-address>) and the mount SUCCEEDED. LESSON 1: before diagnosing a fix as broken, print the binary's own version/commit and compare with repo HEAD (bunker version; <bin> --version) - a stale build makes fixed code look live-broken. LESSON 2: confirm which code path produced the argv by tracing the actual exec (strace -f -qq -e trace=execve -s 600 -o out <cmd>), instead of reasoning from the source you just read. LESSON 3: same class on the server side that same day - the demo host's /usr/local/bin/bunker was 4 commits behind (b4c3c48) while /opt/bunker ran a newer build, so the E2E battery was certifying a stale binary. Both were fixed by rebuilding/installing at HEAD.", "environment": "Linux x86_64, Go 1.26, bunker CLI v0.1.3 installed vs v0.1.4 at repo HEAD", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "stale-installed-cli-masquerades-as-live-bug", "provider": "openrouter", "solved_at": "2026-09-17T11:01:01.379Z", "version": ""}