raft-snapshot-repair
The implementation provides a complete Raft snapshot installation handler in raft_snapshot.go (457 lines) with 17 tests in raft_snapshot_test.go (945 lines). The core design addresses all edge cases:
Snapshot Assembly — Each leader's snapshot transfer is tracked independently via a snapshotAssembly struct keyed by leaderID. Chunks are stored in a map[uint64][]byte keyed by chunk index, enabling out-of-order delivery and natural deduplication (duplicate chunks overwrite with the same data, which is idempotent).
Atomic Log Truncation & State Machine Update — When all chunks are received, the handler: 1. Reassembles snapshot data by concatenating chunks in order 2. Decodes the state machine from the snapshot 3. Atomically (under a single mutex lock) truncates the log, installs the snapshot metadata, replaces the state machine, and advances commit/last-applied indices
Log Conflict Resolution — The log is truncated at the snapshot boundary. Entries strictly after lastIncludedIndex are preserved only if the entry at lastIncludedIndex matches the snapshot's term (indicating consistency). Conflicting entries are discarded, letting the leader re-replicate them.
Edge case handling:
- Stale snapshots (older than current snapshot) → return success (already caught up)
- Stale leader terms → reject with current term
- Zero chunk count → reject immediately
- Interrupted transfers → assembly state persists; retransmission from leader resets the assembly (different LastIncludedIndex triggers fresh state)
- Multiple concurrent leaders → each leader's transfer is tracked separately by leaderID
// Core handler signature — the heart of the solution
func (rn *RaftNode) HandleInstallSnapshot(req *InstallSnapshotRequest) *InstallSnapshotResponse {
rn.mu.Lock()
defer rn.mu.Unlock()
// 1. Term check
// 2. Stale snapshot check
// 3. Zero chunk count rejection
// 4. Assembly management (get or create per leader)
// 5. Chunk storage (idempotent)
// 6. Wait for all chunks
// 7. Reassemble and decode
// 8. Atomic: truncate log + install snapshot + replace state machine
}
**All 17 tests pass** with race detector enabled:
| Test | What it covers |
|------|---------------|
| `TestBasicSnapshotInstall` | Single-chunk snapshot, verifies metadata, state machine, commit/lastApplied indices |
| `TestChunkedSnapshotInstall` | Out-of-order chunk delivery across 4+ chunks |
| `TestDuplicateChunks` | 3 full passes of duplicate chunk sends — verifies idempotency |
| `TestLogConflictResolution` | Snapshot at (3,1) where log entry at (3,1) matches → keeps entries after index 3 |
| `TestLogConflictNoMatch` | Snapshot at (3,3) where log has entry at (2,2) — no match → discards conflicting entries |
| `TestStaleSnapshotRejection` | Newer snapshot installed → older snapshot returns success (already ahead) |
| `TestInterruptedTransferResume` | Only 2 of 4 chunks sent → no snapshot; retransmit all 4 → snapshot installed |
| `TestSendSnapshot` | End-to-end leader → follower via `SendSnapshot` with `sendFn` |
| `TestOutOfOrderWithDuplicates` | Chaotic order: 3,1,1,0,2,0,3 — never sends final chunk → snapshot NOT installed (correct) |
| `TestMultipleLeadersSnapshot` | Two leaders simultaneously sending different snapshots |
| `TestEmptyStateMachineSnapshot` | `map[string]string{}` encoded and installed correctly |
| `TestSnapshotAtLogBoundary` | Snapshot at the exact last log entry index |
| `TestZeroChunkCount` | `ChunkCount=0` rejected with `Success=false` |
| `TestStaleLeaderTerm` | Leader term 5 < follower term 10 → rejected with term=10 |
| `TestInterleavedSnapshotsFromDifferentLeaders` | Leaders 10 and 20 interleave chunks; leader 10 completes first |
| `TestConcurrentSnapshotOperations` | 5 goroutines send snapshots concurrently — no races or deadlocks |
| `TestLogPreservationAfterSnapshot` | Entries after snapshot point preserved only if term matches |
**Benchmark**: `BenchmarkChunkedSnapshotInstall` completes in ~11.5µs per operation.
**Race detector**: Clean — no data races detected across all concurrent tests.
---{"model": "claude-sonnet-4-20250514", "problem_class": "raft-snapshot-repair", "result": "passed", "tests": 17}