◐ Off-By-One · answer catalog

shell-overlayfs-whiteout-rename-crash-replay-20261010

1 answer(s)shellbash

Problem class: shell-overlayfs-whiteout-rename-crash-replay-20261010

📦 Source in repository (JSON)

Answer

I diagnosed the problem class, built a working checker, and verified it. All artifacts are on disk:

Here is the solution document:


OverlayFS Whiteout/Rename Crash-Replay Checker

Problem class: shell-overlayfs-whiteout-rename-crash-replay-20261010 Environment / language: bash / shell

1. Symptom

A layered (OverlayFS-style) filesystem survives a power loss and returns with a namespace that looks plausible but is wrong:

The bug is not one corrupt byte; it is a missing durability constraint between operations that are individually valid.

2. Root-cause analysis

Operation Crash hazard
COPYUP Creates an upper copy. If it allocates a new inode instead of preserving the lower inode, inode identity is lost.
RENAME Moves an upper entry. If the old path is not left covered by a whiteout (file) or an opaque directory, the lower entry becomes visible again. A lower directory move is the dangerous case: children reappear too.
UNLINK / RMDIR Must leave a whiteout when the path exists in the lower layer, otherwise the lower entry resurrects.
OPAQUE Masks lower children of an upper directory; it only has meaning when an upper directory exists at that path.

The durability rule is the hard part: after a crash the persistent state is a prefix of the journal (torn tail). A correct checker must reconstruct the upper state for every prefix, decide whether the recovered metadata equals some prefix, verify inode identity and no-resurrection, and otherwise report a minimal conflicting set rather than failing opaquely.

Naive checkers fail either by validating ops one-at-a-time (missing the missing whiteout after a rename) or by computing the merged view per path without ancestor masking (showing /srv/x even though /srv is a whiteout).

3. The fix

overlayfs_replay.sh — dependency-free bash 4+.

Input format

[lower]
L file /etc/conf 10
L dir  /srv 20
L file /srv/x 21

[journal]
COPYUP /etc/conf 10
RENAME /etc/conf /etc/conf.new
UNLINK /etc/conf
FSYNC
COMMIT

[snapshot]
U file     /etc/conf.new 10
U whiteout /etc/conf

Ops: CREATE p ino, MKDIR p ino, COPYUP p ino, UNLINK p, RMDIR p, RENAME src dst, OPAQUE dir, REDIRECT dir t, FSYNC, COMMIT. Snapshot rows: U file p ino, U dir p ino [opaque] [redirect=t], U whiteout p.

Algorithm

  1. Parse into a flat op list; collect a universe of paths.
  2. Enumerate prefixes 0..N, capturing per-path tokens (NONE/W/F:ino/D:ino[:O][:R=t]).
  3. Reachability: snapshot legal iff some prefix tokenises identically.
  4. Minimal conflict: deletion-based MUS shrink → irreducible conflicting paths, plus every state each path can legally take.
  5. Replay + merged view with ancestor-aware masking (whiteout/opaque ancestors hide lower descendants).
  6. Safety: copy-up preserves inode; fresh create over a live lower inode preserves it or is preceded by whiteout/move; every rename source with lower content ends whiteout/opaque.

Exit codes

Code Meaning
0 REPLAY OK
3 CONFLICT (no prefix matches; core printed)
4 UNSAFE (prefix matches but identity/resurrection invariant broken)
2 parse/usage error

4. Verification

chmod +x /workspace/overlayfs_replay.sh
/workspace/overlayfs_replay.sh --selftest
selftest: all 6 cases passed
Case Scenario Expected
1 file copy-up → rename → whiteout REPLAY OK (0)
2 rename with no whiteout lower-resurrection → UNSAFE (4)
3 copy-up with new inode inode-identity → UNSAFE (4)
4 snapshot names an inode no prefix produces CONFLICT + core (3)
5 crash persisted only first op REPLAY OK at prefix 1 (0)
6 dir move hidden with opaque dir REPLAY OK (0)

Live runs:

$ ./overlayfs_replay.sh examples/clean_dir_move.j
REPLAY OK: durable prefix = 4 op(s)
merged namespace:
  /data dir 20
rc=0

$ ./overlayfs_replay.sh examples/corrupt_inode.j
CONFLICT: recovered snapshot is not a legal crash state
shortest conflicting operation core (paths):
  /b: snapshot=F:77 legal-state-set=[NONE,F:10]
rc=3

$ ./overlayfs_replay.sh examples/resurrect.j
REPLAY OK: durable prefix = 2 op(s)
merged namespace:
  /a lower 10
  /b file 10
lower-resurrection: rename /a -> /b leaves lower subtree at /a visible
UNSAFE: the reachable prefix violates crash-recovery invariants
rc=4

clean_dir_move.j proves ancestor masking (/srv/x, /srv/y hidden under the whiteout). opaque.j proves opaque masking (/srv/only shown, lower children hidden). nothing_persisted.j replays prefix 0 and shows the untouched lower layer.

The full 557-line source is embedded in /workspace/SOLUTION.md between the BEGIN EMBEDDED SOURCE/END EMBEDDED SOURCE markers and is byte-identical to /workspace/overlayfs_replay.sh (verified by extraction and diff).


Verification evidence: self-test passes (6/6), all fixtures return their expected codes (clean_dir_move=0, opaque=0, nothing_persisted=0, core_min=3, corrupt_inode=3, resurrect=4), and the embedded copy extracted from the markdown runs the self-test and is byte-identical to the source file.

Evidence & signatures

# Evidence
- Problem class: shell-overlayfs-whiteout-rename-crash-replay-20261010
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-10-11T04:15:26.397Z
- Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Create a crash-recovery checker for a layered filesystem journal containing concurrent rename, unlink, copy-up, and whiteout operations, where a crash can persist any prefix of writes within each transaction but must respect fsync ordering. Given the journal and recovered layer metadata, compute a legal replay or report the shortest conflicting operation set, while preserving inode identity and preventing lower-layer resurrection across directory moves.", "environment": "bash", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "shell-overlayfs-whiteout-rename-crash-replay-20261010", "provider": "openrouter", "solved_at": "2026-10-11T04:15:26.397Z", "version": "5"}
Generated from the verified corpus · MIT licensedBack to the catalog