go-registry-reconcile-destroys-foreign-daemon-agents
I diagnosed the issue, verified the fix against the actual repository, and wrote the solution to ~/SOLUTION.md.
The repo (github.com/deployBunker/bunker) already contains the authoritative fix at commit e0a6093 (DF-BUNKER-13). I inspected the code, reproduced the defect, and independently verified the fix.
Root cause: reconciliation is host-scoped. A second daemon enumerates every bunker-* user, treats the first daemon's agents as orphans, and fails closed when their persisted port range (<home>/.bunker/ports) falls outside its own pool — force-destroying a healthy agent that could never collide with anything it allocates.
Exact fix (4 code changes + README):
1. PortAllocator.Bounds() — read-only, mutex-guarded pool span accessor.
2. ReconcileReport.Foreign counter.
3. orphanIsForeign(sa) helper + a foreign check before the mode branch (so it applies in both adopt and destroy mode) that skips agents whose readable persisted range is provably disjoint from the pool (end < poolStart || start > poolEnd). It deliberately does not use ValidateRange, so in-pool-but-unaligned ranges keep their fail-closed treatment; malformed metadata is never foreign.
4. Startup summary logs foreign; README documents the isolation requirement.
Verification performed:
- Repo's own DF-BUNKER-13 tests all pass on the fixed tree (including the moved outside the pool row with flipped expectations).
- I wrote an independent probe that compiles on both trees (uses no fix-introduced symbols) and ran it against a pre-fix worktree at e0a6093~1:
- Pre-fix: FAILS with destroy seam called 1 time(s): [foreign-agent] in both adopt and destroy mode; controls pass.
- Fixed: PASSES with 0 destroy calls; the two controls (in-pool held range, malformed metadata) destroy exactly once on both trees.
The repo is left clean, and the worktree was removed.
# Evidence - Problem class: go-registry-reconcile-destroys-foreign-daemon-agents - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T15:44:37.562Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: starting a SECOND daemon instance on a host that already runs one silently DESTROYS the first daemon's healthy agents. Observed verbatim in the daemon log: 'registry reconcile: adopt failed, destroying orphan instead' followed by 'registry reconcile: destroyed orphan agent ... agent_id=bunker-2eae301d'. The agent's rootless dockerd was SIGTERMed and the agent was unrecoverable, even with agent.reconciliation.mode: adopt configured. It reproduces on the documented multi-instance path (README 'Non-default ports' tells you to start a second daemon with different listen ports).\n\nROOT CAUSE: reconciliation is HOST-scoped, not registry-scoped. At startup it enumerates every bunker-* user in /etc/passwd, subtracts the durable registry records it knows, and treats the remainder as orphans. An orphan whose persisted port range cannot be re-established in THIS daemon's pool is force-destroyed as 'unmanageable'. But the persisted range comes from the agent's own <home>/.bunker/ports file ('<start>-<end>' written at spawn), and a second daemon with a different pool geometry (agent.port_range_start/end) sees the first daemon's in-range-for-daemon-1 ranges as OUTSIDE ITS OWN POOL: PortAllocator.ValidateRange -> validateRangeLocked returns 'port range %d-%d outside pool %d-%d', adoptAgent fails, and the destroy path runs. The daemon destroys an agent that provably cannot collide with anything it allocates.\n\nFIX: classify orphans FOREIGN BEFORE the mode branch (applies to BOTH adopt and destroy mode). Foreign == the agent's persisted range is readable AND DISJOINT from this daemon's pool (end < poolStart || start > poolEnd). A disjoint agent cannot collide with any port this daemon will ever allocate, so destroying it buys nothing and costs another daemon's agent: skip it, emit one loud warning naming the agent id, its persisted range, this daemon's pool bounds and the cause ('owned by another daemon instance or an older pool geometry; destroy it from the daemon that owns it'), count it in a new ReconcileReport.Foreign field, and leave the user, its tracker slots and the pool untouched. Deliberately do NOT use ValidateRange for the foreign test: it also rejects IN-POOL-but-unaligned ranges, which must keep their fail-closed treatment, because two daemons with OVERLAPPING pools remain unsupported. Unreadable/malformed metadata is NOT foreign either (the daemon cannot prove it is safe to leave). Implementation: a read-only PortAllocator.Bounds() accessor (take the mutex like the other accessors) plus a small orphanIsForeign(sa) helper; add 'foreign' to the startup reconciliation summary log; document the isolation requirement in the README (shared bunker-* user namespace + shared agent.registry.path, give each instance its own pool slice and registry path).\n\nVERIFICATION (this is the part that matters, because the repo's OWN test suite had the defect encoded as expected behaviour): the pre-existing table row 'outside the pool' (persisted 30000-30099 against the default 10000-19999 pool, expected-destroy) was byte-for-byte the bug, so it had to be MOVED to a new test file with flipped expectations rather than left asserting the defect. The load-bearing proof is an INDEPENDENT probe run against a pre-fix git worktree (HEAD~1): the probe FAILS there with 'destroy seam called 1 time(s): [foreign-agent]' in BOTH adopt and destroy mode (the defect reproduced), and PASSES on the fixed commit with the same assertions; two controls (in-pool range held by another agent -> still destroyed; malformed metadata -> still destroyed) behave identically in both trees, so the probe is testing the fix and not a global behaviour change. Assert the seam call COUNT (0 for foreign), report.Foreign == 1, and that no tracker record or port reservation is left behind.", "environment": "", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-registry-reconcile-destroys-foreign-daemon-agents", "provider": "openrouter", "solved_at": "2026-09-16T15:44:37.563Z", "version": ""}I diagnosed the issue, verified the fix against the actual repository, and wrote the solution to ~/SOLUTION.md.
The repo (github.com/deployBunker/bunker) already contains the authoritative fix at commit e0a6093 (DF-BUNKER-13). I inspected the code, reproduced the defect, and independently verified the fix.
Root cause: reconciliation is host-scoped. A second daemon enumerates every bunker-* user, treats the first daemon's agents as orphans, and fails closed when their persisted port range (<home>/.bunker/ports) falls outside its own pool — force-destroying a healthy agent that could never collide with anything it allocates.
Exact fix (4 code changes + README):
1. PortAllocator.Bounds() — read-only, mutex-guarded pool span accessor.
2. ReconcileReport.Foreign counter.
3. orphanIsForeign(sa) helper + a foreign check before the mode branch (so it applies in both adopt and destroy mode) that skips agents whose readable persisted range is provably disjoint from the pool (end < poolStart || start > poolEnd). It deliberately does not use ValidateRange, so in-pool-but-unaligned ranges keep their fail-closed treatment; malformed metadata is never foreign.
4. Startup summary logs foreign; README documents the isolation requirement.
Verification performed:
- Repo's own DF-BUNKER-13 tests all pass on the fixed tree (including the moved outside the pool row with flipped expectations).
- I wrote an independent probe that compiles on both trees (uses no fix-introduced symbols) and ran it against a pre-fix worktree at e0a6093~1:
- Pre-fix: FAILS with destroy seam called 1 time(s): [foreign-agent] in both adopt and destroy mode; controls pass.
- Fixed: PASSES with 0 destroy calls; the two controls (in-pool held range, malformed metadata) destroy exactly once on both trees.
The repo is left clean, and the worktree was removed.
# Evidence - Problem class: go-registry-reconcile-destroys-foreign-daemon-agents - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T15:44:37.562Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: starting a SECOND daemon instance on a host that already runs one silently DESTROYS the first daemon's healthy agents. Observed verbatim in the daemon log: 'registry reconcile: adopt failed, destroying orphan instead' followed by 'registry reconcile: destroyed orphan agent ... agent_id=bunker-2eae301d'. The agent's rootless dockerd was SIGTERMed and the agent was unrecoverable, even with agent.reconciliation.mode: adopt configured. It reproduces on the documented multi-instance path (README 'Non-default ports' tells you to start a second daemon with different listen ports).\n\nROOT CAUSE: reconciliation is HOST-scoped, not registry-scoped. At startup it enumerates every bunker-* user in /etc/passwd, subtracts the durable registry records it knows, and treats the remainder as orphans. An orphan whose persisted port range cannot be re-established in THIS daemon's pool is force-destroyed as 'unmanageable'. But the persisted range comes from the agent's own <home>/.bunker/ports file ('<start>-<end>' written at spawn), and a second daemon with a different pool geometry (agent.port_range_start/end) sees the first daemon's in-range-for-daemon-1 ranges as OUTSIDE ITS OWN POOL: PortAllocator.ValidateRange -> validateRangeLocked returns 'port range %d-%d outside pool %d-%d', adoptAgent fails, and the destroy path runs. The daemon destroys an agent that provably cannot collide with anything it allocates.\n\nFIX: classify orphans FOREIGN BEFORE the mode branch (applies to BOTH adopt and destroy mode). Foreign == the agent's persisted range is readable AND DISJOINT from this daemon's pool (end < poolStart || start > poolEnd). A disjoint agent cannot collide with any port this daemon will ever allocate, so destroying it buys nothing and costs another daemon's agent: skip it, emit one loud warning naming the agent id, its persisted range, this daemon's pool bounds and the cause ('owned by another daemon instance or an older pool geometry; destroy it from the daemon that owns it'), count it in a new ReconcileReport.Foreign field, and leave the user, its tracker slots and the pool untouched. Deliberately do NOT use ValidateRange for the foreign test: it also rejects IN-POOL-but-unaligned ranges, which must keep their fail-closed treatment, because two daemons with OVERLAPPING pools remain unsupported. Unreadable/malformed metadata is NOT foreign either (the daemon cannot prove it is safe to leave). Implementation: a read-only PortAllocator.Bounds() accessor (take the mutex like the other accessors) plus a small orphanIsForeign(sa) helper; add 'foreign' to the startup reconciliation summary log; document the isolation requirement in the README (shared bunker-* user namespace + shared agent.registry.path, give each instance its own pool slice and registry path).\n\nVERIFICATION (this is the part that matters, because the repo's OWN test suite had the defect encoded as expected behaviour): the pre-existing table row 'outside the pool' (persisted 30000-30099 against the default 10000-19999 pool, expected-destroy) was byte-for-byte the bug, so it had to be MOVED to a new test file with flipped expectations rather than left asserting the defect. The load-bearing proof is an INDEPENDENT probe run against a pre-fix git worktree (HEAD~1): the probe FAILS there with 'destroy seam called 1 time(s): [foreign-agent]' in BOTH adopt and destroy mode (the defect reproduced), and PASSES on the fixed commit with the same assertions; two controls (in-pool range held by another agent -> still destroyed; malformed metadata -> still destroyed) behave identically in both trees, so the probe is testing the fix and not a global behaviour change. Assert the seam call COUNT (0 for foreign), report.Foreign == 1, and that no tracker record or port reservation is left behind.", "environment": "", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-registry-reconcile-destroys-foreign-daemon-agents", "provider": "openrouter", "solved_at": "2026-09-16T15:44:37.563Z", "version": ""}