shell-20261008-raft-snapshot-install-fsync-recovery

Answer

Done. The verified solution is at /workspace/SOLUTION.md (31 KB, self-contained, with the full implementation and harness embedded). Supporting files: /workspace/raft_snapshot.sh and /workspace/verify.sh.

Root cause

The classic temp → rename → manifest → truncate log sequence has seven reachable crash states (F1–F7). The fatal ones are:

• F4/F5 — trusting a filename (snap-<idx>) or a stale .tmp/directory entry, so a torn or leftover file is loaded as a valid snapshot and commitIndex is set from its name.
• F6 — a separate manifest can be fsynced before/after the data it references; no write ordering makes that safe across filesystems.
• F7 — when the chosen snapshot fails validation, code clamps commitIndex downward. That silently loses committed entries — the actual safety violation.

The fix

Make the snapshot one self-describing, checksummed file (512-byte header with index/term/length/sha256/commit, then payload, then DATAEND trailer), so data and its commit metadata are durable or torn together. The single commit point is one atomic rename inside one directory:

fsync(temp)      # bytes durable BEFORE the name      (Rule 1)
rename -> final  # atomic, same FS
fsync(dir)       # name durable                       (Rule 2)
write_stable(commit = max(commit, idx))               # only after Rule 2
truncate log > idx                                    # idempotent, last

Recovery wipes staging, validates every candidate, keeps the highest valid snapshot, quarantines the rest, discards the torn log tail, and computes:

commit   = max(stable.commit, snapshot.index)
coverage = max(snapshot.index, lastIntactLogIndex)
if commit > coverage: NEEDS_RESYNC = 1   # I6 — never clamp, never roll back

Verification (bash /workspace/verify.sh)

==== 22 passed, 0 failed ====

Covers: baseline restart; six torn-write shapes (truncate, payload flip, trailer flip, bad magic, trailing junk, zeroed length) — each rejected with commit preserved and NEEDS_RESYNC set; stale staging + invalid final entries; crash injected at each of the 5 durable install steps; torn stable record; and the ordering adversary where a rename is durable but bytes are not (commit kept, NEEDS_RESYNC flagged). I also extracted the embedded code blocks from SOLUTION.md into a clean directory and re-ran the whole suite against them, confirming the document is truly self-contained (embedded script == tested workspace script).