go-lease-fenced-shard-rebalance
Done. The verified solution is written to ~/SOLUTION.md (936 lines: title, root-cause analysis, full fix, verification), and the runnable module lives in ~/shardrebalancer/.
Root causes identified
1. Paused old owners write after a handoff → fixed with monotonic fencing tokens bumped at BeginTransfer.
2. Leases trusting worker clocks → fixed by evaluating leases only in authority time.
3. Lost acknowledged writes during copy → fixed with revoke → fence → snapshot → install → activate ordering + lease-check/apply under the same mutex the snapshot takes.
4. Double-applies / duplicate storage → fixed with idempotent writes by op ID and idempotent installs by seq.
5. Coordinator crashes mid-handoff → fixed with a write-ahead TransferRecord and idempotent phase replay.
Linearization point: Authority.BeginTransfer atomically bumps the fence and puts the destination in a dead epoch; Activate opens it only after the snapshot is installed.
Verification (all passing, including -race -count=3)
- TestStaleOwnerFencedAfterTransfer
- TestLeaseExpiryFencesPausedOwner
- TestCrashRecoveryAtEveryHandoffStep — subtests for crash after revoke, fence, snapshot, install, activate, drop
- TestIdempotentReplay
- TestNoAckedWriteLostDuringHandoff
- TestSingleCopyAfterTransfer
$ go test ./... -count=1
ok shardrebalancer 0.008s
$ go test ./... -race -count=3
ok shardrebalancer 1.029s
The doc's §3.1 also maps each in-memory component (Authority, RecordStore, worker store) to its production counterpart (etcd/ZK, WAL/consensus, partitioned storage) so the same invariants carry over.
# Evidence - Problem class: go-lease-fenced-shard-rebalance - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-10-04T22:09:10.952Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Build a shard rebalancer that migrates partitioned state between workers while leases may expire, clocks drift, and old owners continue issuing writes after a pause. Design fencing and an idempotent transfer protocol that preserves linearizable reads, bounds duplicate storage, and recovers correctly from crashes at every handoff step.", "environment": "go1.26", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-lease-fenced-shard-rebalance", "provider": "openrouter", "solved_at": "2026-10-04T22:09:10.953Z", "version": "1.26"}