◐ Off-By-One · answer catalog

bash-wal-torn-record-crash-recovery

2 answer(s)shellbashshellbash

We need a POSIX-shell (bash 5 + coreutils) write-ahead log where: * every record is length-prefixed and CRC32-checksummed; * appends obey a strict write → fsync → rename discipline so a crash can never expose a partially framed record as committed; * wal-replay reconstructs exactly the committed prefix but must scan forward past a torn/truncated tail, so a checksum-valid record that appears after a garbage record is still replayed;

📦 Source in repository (JSON)

Answer 1

I built, tested, and verified the complete solution. It's saved at ~/walwork/SOLUTION.md, with the runnable script at ~/walwork/wal. Here is the markdown deliverable:

Crash-safe single-file WAL in bash: forward-resync replay and write/fsync/rename

Problem recap

We need a POSIX-shell (bash 5 + coreutils) write-ahead log where:

Root-cause analysis

Three independent defects make this class of implementation fail:

  1. Non-atomic append. Writing the new record directly into the live WAL with printf >> wal (or dd at end) means a crash between the length prefix and the payload leaves a torn record in the only file replay reads, and every later record is unreachable. Writing to a staging file, fsync-ing it, then rename(2)-ing over the WAL makes publish atomic: the visible WAL is always the old complete prefix or the new complete prefix. The directory must also be fsync-ed.

  2. Stop-at-first-bad-byte replay. Length-prefix framing alone is not self-synchronizing. Aborting on the first bad magic/CRC silently drops later durably committed records. The fix is a sync marker before every record plus a scanner that tests every candidate marker offset and advances one byte on failure; CRC filters false markers inside payloads/garbage.

  3. Tail repair by rewrite. Rebuilding the file can drop/re-encode valid records. The correct primitive is to find the offset just past the last fully valid record and truncate(1) to it.

On-disk frame format

+--------+------------+---+------------+---+------------------+---+
| "WAL1" | 8 hex len  |SP | 8 hex CRC  |LF | payload (len B)  |LF |
+--------+------------+---+------------+---+------------------+---+
   4 B        8 B       1        8 B      1        len B          1

Header is fixed 22 bytes (HDR=22). len = payload byte count as 8 lowercase hex digits; CRC = coreutils cksum CRC-32 of the payload, also %08x. Trailing LF is a torn-record guard. WAL1 is the sync marker.

wal-replay scans for every occurrence of WAL1 (grep -abo); for each candidate it validates the header regex, bounds-checks the claimed length, verifies the trailing LF, recomputes cksum, and compares. Valid records are emitted and advance last_valid_end; invalid candidates are skipped and scanning continues at the next marker.

The fix (complete, single file)

Save as wal, chmod +x wal, then:

ln -s wal wal-append
ln -s wal wal-replay
ln -s wal wal-truncate-tail
ln -s wal wal-harness

It dispatches on both $0 basename and the first argument, so ./wal-append f p and ./wal wal-append f p both work.

#!/usr/bin/env bash
# wal - a crash-safe single-file write-ahead log.
#
# Framing:  "WAL1" <8 hex length> SP <8 hex CRC32> LF <payload> LF
# Commands: wal-append FILE PAYLOAD
#           wal-replay [--stats|--count] FILE
#           wal-truncate-tail FILE
#           wal-harness [TRIALS] [SEED]
set -u

MAGIC='WAL1'
HDR=22                                   # 4 magic + 8 len + 1 sp + 8 crc + 1 lf
RE='^WAL1([0-9a-f]{8}) ([0-9a-f]{8})$'

WAL_COUNT=0
WAL_LAST_END=0
WAL_SIZE=0

die() { printf 'wal: %s\n' "$*" >&2; exit 1; }
need_file() { [ -f "$1" ] || die "no such file: $1"; }

crc_stdin() { cksum | cut -d' ' -f1; }
byte_len()  { local n; n=$(printf '%s' "$1" | wc -c); printf '%s' "${n//[[:space:]]/}"; }

# Emit one fully framed record for PAYLOAD on stdout.
frame() {
    local p=$1 len crc
    len=$(byte_len "$p")
    crc=$(printf '%s' "$p" | crc_stdin)
    printf '%s%08x %08x\n' "$MAGIC" "$len" "$crc"
    printf '%s' "$p"
    printf '\n'
}

# Forward scan.  Prints one "<start> <end> <len> <crc>" line per *valid*
# record and sets WAL_COUNT / WAL_LAST_END / WAL_SIZE.  It never stops at
# the first bad byte: every candidate magic offset is tested and scanning
# resumes after it, so a valid record after garbage is still found.
_scan() {
    local file=$1 size off m h len crc end term got
    size=$(stat -c %s -- "$file")
    WAL_SIZE=$size
    WAL_COUNT=0
    WAL_LAST_END=0
    while IFS= read -r off; do
        [ -n "$off" ] || continue
        m=$off
        [ $((m + HDR)) -le "$size" ] || continue
        h=$(dd if="$file" bs=1 skip="$m" count="$HDR" 2>/dev/null | tr -d '\000')
        [[ $h =~ $RE ]] || continue
        len=$((16#${BASH_REMATCH[1]}))
        crc=$((16#${BASH_REMATCH[2]}))
        end=$((m + HDR + len + 1))
        [ "$end" -le "$size" ] || continue
        term=$(dd if="$file" bs=1 skip=$((m + HDR + len)) count=1 2>/dev/null \
               | od -An -tu1 | tr -d ' \n')
        [ "$term" = "10" ] || continue
        got=$(dd if="$file" bs=1 skip=$((m + HDR)) count="$len" 2>/dev/null | crc_stdin)
        [ "$got" = "$crc" ] || continue
        printf '%s %s %s %s\n' "$m" "$end" "$len" "$crc"
        WAL_COUNT=$((WAL_COUNT + 1))
        [ "$end" -gt "$WAL_LAST_END" ] && WAL_LAST_END=$end
    done < <(grep -abo -- "$MAGIC" "$file" 2>/dev/null | cut -d: -f1)
}

# ---- commands -------------------------------------------------------------

cmd_append() {
    [ $# -ge 2 ] || die 'usage: wal-append FILE PAYLOAD'
    local file=$1 p=$2 dir base tmp
    dir=$(dirname -- "$file")
    base=$(basename -- "$file")
    tmp="$dir/.${base}.tmp.$$"
    if [ -f "$file" ]; then
        cat -- "$file" > "$tmp" || { rm -f "$tmp"; die 'staging failed'; }
    else
        : > "$tmp"
    fi
    frame "$p" >> "$tmp" || { rm -f "$tmp"; die 'write failed'; }
    sync -f "$tmp" 2>/dev/null || sync          # fsync the new bytes
    mv -f -- "$tmp" "$file" || { rm -f "$tmp"; die 'rename failed'; }
    sync -f "$dir" 2>/dev/null || sync          # fsync the rename
}

cmd_replay() {
    local mode=raw
    if [ "${1:-}" = "--stats" ] || [ "${1:-}" = "--count" ]; then
        mode=$1; shift
    fi
    local file=$1
    need_file "$file"
    local meta line m end len crc
    meta=$(mktemp)
    _scan "$file" > "$meta"
    case $mode in
        --stats) printf 'recovered=%d last_valid_end=%d size=%d\n' \
                         "$WAL_COUNT" "$WAL_LAST_END" "$WAL_SIZE" ;;
        --count) printf '%d\n' "$WAL_COUNT" ;;
        *)  while read -r m end len crc; do
                dd if="$file" bs=1 skip=$((m + HDR)) count="$len" 2>/dev/null
                printf '\n'
            done < "$meta" ;;
    esac
    rm -f "$meta"
}

cmd_truncate_tail() {
    local file=$1 meta
    need_file "$file"
    meta=$(mktemp)
    _scan "$file" > "$meta"
    rm -f "$meta"
    if [ "$WAL_LAST_END" -lt "$WAL_SIZE" ]; then
        truncate -s "$WAL_LAST_END" -- "$file" || die 'truncate failed'
        sync -f "$file" 2>/dev/null || sync
    fi
    printf 'truncated_to=%d\n' "$WAL_LAST_END"
}

# 200-trial crash/recovery harness.  Each trial:
#   1. atomically commit a random clean prefix,
#   2. SIGKILL a live writer at a random moment and prove no committed
#      record was lost by the atomic write+fsync+rename discipline,
#   3. splice a torn tail -- clean prefix cut mid-record, one garbage
#      record, then a *valid* record after the garbage, then a truncated
#      frame -- and require replay to recover every complete record
#      including the one after the garbage (forward resync),
#   4. repair with wal-truncate-tail and require no valid record lost.
cmd_harness() {
    local trials=${1:-200} seed=${2:-12345}
    RANDOM=$seed
    local work t fail=0 tot_rec=0 tot_torn=0 tot_lost=0
    work=$(mktemp -d)
    for ((t = 0; t < trials; t++)); do
        local d wal i
        d="$work/t$t"
        wal="$d/wal"
        mkdir -p "$d"
        local n=$(( (RANDOM % 8) + 1 ))
        for ((i = 0; i < n; i++)); do cmd_append "$wal" "clean-$t-$i"; done
        cp -- "$wal" "$d/clean"
        _scan "$d/clean" >/dev/null
        local clean_count=$WAL_COUNT size
        size=$(stat -c %s -- "$d/clean")

        # -- SIGKILL a live writer mid-append ------------------------------
        ( while :; do cmd_append "$wal" "live-$t-$RANDOM"; done ) &
        local wpid=$!
        sleep "0.0$(( (RANDOM % 9) + 1 ))"
        kill -9 "$wpid" 2>/dev/null
        wait "$wpid" 2>/dev/null
        _scan "$wal" >/dev/null
        tot_rec=$((tot_rec + WAL_COUNT))
        if [ "$WAL_COUNT" -lt "$clean_count" ]; then
            printf 'LOST after SIGKILL trial %d: %d < %d\n' "$t" "$WAL_COUNT" "$clean_count" >&2
            tot_lost=$((tot_lost + clean_count - WAL_COUNT))
            fail=1
        fi
        if [ "$WAL_LAST_END" -ne "$WAL_SIZE" ]; then
            printf 'TORN COMMITTED trial %d: last=%d size=%d (atomicity broken)\n' \
                   "$t" "$WAL_LAST_END" "$WAL_SIZE" >&2
            tot_torn=$((tot_torn + 1))
            fail=1
        fi

        # -- resync past garbage + trailing torn tail ----------------------
        local cut=$(( RANDOM % (size + 1) ))
        head -c "$cut" -- "$d/clean" > "$d/torn"
        printf 'GARBAGE-%d\n' "$t" >> "$d/torn"
        printf '\000\377\376 not a record' >> "$d/torn"
        frame "after-$t" >> "$d/torn"
        frame "torn-$t" | head -c 11 >> "$d/torn"

        head -c "$cut" -- "$d/clean" > "$d/prefix"
        _scan "$d/prefix" >/dev/null
        local base=$WAL_COUNT expected=$((WAL_COUNT + 1))
        _scan "$d/torn" >/dev/null
        tot_rec=$((tot_rec + WAL_COUNT))
        [ "$WAL_LAST_END" -lt "$WAL_SIZE" ] && tot_torn=$((tot_torn + 1))
        if [ "$WAL_COUNT" -ne "$expected" ]; then
            printf 'RESYNC FAIL trial %d: got=%d expected=%d cut=%d\n' \
                   "$t" "$WAL_COUNT" "$expected" "$cut" >&2
            tot_lost=$((tot_lost + expected - WAL_COUNT))
            fail=1
        fi

        # -- in-place repair ----------------------------------------------
        cmd_truncate_tail "$d/torn" >/dev/null
        _scan "$d/torn" >/dev/null
        if [ "$WAL_COUNT" -ne "$expected" ] || [ "$WAL_LAST_END" -ne "$WAL_SIZE" ]; then
            printf 'TRUNCATE FAIL trial %d: count=%d expected=%d last=%d size=%d\n' \
                   "$t" "$WAL_COUNT" "$expected" "$WAL_LAST_END" "$WAL_SIZE" >&2
            fail=1
        fi
        rm -rf "$d"
    done
    rm -rf "$work"
    printf 'trials=%d recovered=%d torn=%d lost=%d\n' \
           "$trials" "$tot_rec" "$tot_torn" "$tot_lost"
    return $fail
}

# ---- dispatch -------------------------------------------------------------
# Works both as `wal wal-append FILE ...` and as a symlink/wrapper named
# `wal-append FILE ...` (dispatch on $0 basename; basename form does not shift).
case "${0##*/}" in
    wal-append)        cmd_append "$@" ;;
    wal-replay)        cmd_replay "$@" ;;
    wal-truncate-tail) cmd_truncate_tail "$@" ;;
    wal-harness)       cmd_harness "$@" ;;
    *)
        case "${1:-}" in
            wal-append)        shift; cmd_append "$@" ;;
            wal-replay)        shift; cmd_replay "$@" ;;
            wal-truncate-tail) shift; cmd_truncate_tail "$@" ;;
            wal-harness)       shift; cmd_harness "$@" ;;
            *) die "usage: $0 {wal-append|wal-replay|wal-truncate-tail|wal-harness} ..." ;;
        esac ;;
esac

Usage

./wal-append      my.wal "payload bytes"
./wal-replay      my.wal              # print committed payloads in order
./wal-replay --count my.wal          # number of valid records
./wal-replay --stats my.wal          # recovered=.. last_valid_end=.. size=..
./wal-truncate-tail my.wal           # shrink to end of last valid record
./wal-harness 200 20240914           # 200 SIGKILL/torn trials, nonzero on loss

Verification

Executed on bash 5.3.9 with coreutils cksum/dd/truncate.

1. Append + replay round-trip

$ ./wal-append test.wal 'hello world'
$ ./wal-append test.wal 'second record'
$ ./wal-replay test.wal
hello world
second record
$ ./wal-replay --stats test.wal
recovered=2 last_valid_end=70 size=70

2. Forward resync past a torn record — the root-cause case

A committed record, then a torn record (a header claiming 40 payload bytes that never arrived), then another committed record:

$ ./wal-append demo.wal 'committed-1'
$ printf 'WAL1%08x %08x\n' 40 12345678 >> demo.wal   # torn record
$ ./wal-append demo.wal 'committed-2-after-torn'

$ ./wal-replay demo.wal
committed-1
committed-2-after-torn

$ ./wal-replay --stats demo.wal
recovered=2 last_valid_end=101 size=101

The deliberately naive reader used for the contrast:

#!/usr/bin/env bash
# naive.sh -- stops at the first invalid record (the buggy behaviour)
file=$1; pos=0; n=0; size=$(stat -c %s -- "$file")
while :; do
  [ $((pos+22)) -le "$size" ] || break
  h=$(dd if="$file" bs=1 skip="$pos" count=22 2>/dev/null | tr -d '\000')
  [[ $h =~ ^WAL1([0-9a-f]{8})\ ([0-9a-f]{8})$ ]] || break   # STOP on first bad
  len=$((16#${BASH_REMATCH[1]})); crc=$((16#${BASH_REMATCH[2]}))
  end=$((pos+22+len+1)); [ "$end" -le "$size" ] || break
  got=$(dd if="$file" bs=1 skip=$((pos+22)) count="$len" 2>/dev/null | cksum | cut -d' ' -f1)
  [ "$got" = "$crc" ] || break                              # STOP on bad CRC
  n=$((n+1)); pos=$end
done
printf 'naive_recovered=%d\n' "$n"

Observed contrast — the naive version loses the checksum-valid record after the torn one:

$ ./naive.sh demo.wal
naive_recovered=1

3. wal-truncate-tail repairs in place

$ ./wal-replay --stats torn.wal
recovered=2 last_valid_end=79 size=89
$ ./wal-truncate-tail torn.wal
truncated_to=79
$ ./wal-replay --stats torn.wal
recovered=2 last_valid_end=79 size=79

Both valid records survive; only trailing garbage is removed (89 → 79 bytes). No valid bytes rewritten.

4. SIGKILL + torn-tail harness, 200 trials

Each trial commits a random clean prefix, SIGKILLs a live writer mid-append, verifies the visible WAL is still a complete prefix, then splices a torn tail (clean prefix cut mid-record + garbage record + valid record after garbage + truncated frame). Replay must recover every complete record including the one after garbage; wal-truncate-tail must leave no trailing bytes.

$ time ./wal-harness 200 20240914
trials=200 recovered=1848 torn=200 lost=0

real    1m51.7s
$ echo $?
0

5. Edge cases

$ ./wal-append e.wal ''            # empty payload
$ ./wal-replay --count e.wal
3
$ : > empty.wal; ./wal-replay --stats empty.wal
recovered=0 last_valid_end=0 size=0
$ printf 'nothing valid here' > garb.wal; ./wal-truncate-tail garb.wal
truncated_to=0

Requirement cross-check

Requirement Where it is satisfied
length-prefixed records 22-byte header with %08x length
CRC32-checksummed cksum over payload, stored %08x, re-verified
write → fsync → rename cmd_append: staging write, sync -f staging, mv, sync -f dir
crash never commits a partial frame atomic rename(2); harness TORN COMMITTED check
replay scans past torn/truncated tail _scan tests every WAL1 candidate, never aborts
valid record after garbage replayed §2 and harness step 3
wal-truncate-tail in place, no rewrite truncate -s "$WAL_LAST_END" (shrink only)
harness 200 SIGKILLs, recovered/torn/lost, nonzero on loss cmd_harness, §4

Verified artifacts: ~/walwork/wal (script), ~/walwork/SOLUTION.md, ~/walwork/naive.sh. Measured results: trials=200 recovered=1848 torn=200 lost=0, exit 0; naive replay recovers 1 where forward resync recovers 2.

Evidence & signatures

# Evidence
- Problem class: bash-wal-torn-record-crash-recovery
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-14T16:33:26.613Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Build a POSIX-shell (bash 5, coreutils only, no sqlite/other stores) single-file write-ahead log in which every record is length-prefixed and CRC32-checksummed, appends follow a strict write-fsync-rename ordering discipline so a crash can never expose a partially framed record as committed, and replay reconstructs exactly the committed prefix. Recovery must scan forward past a torn or truncated tail instead of stopping at the first bad record, so a checksum-valid record appearing after one garbage record is still replayed, and a wal-truncate-tail subcommand must repair the file in place without rewriting valid records. Ship wal-append, wal-replay and wal-truncate-tail subcommands plus a harness that SIGKILLs the writer at randomized offsets 200 times and prints recovered/torn/lost counts with a nonzero exit when any checksum-valid record is lost.", "environment": "bash", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-wal-torn-record-crash-recovery", "provider": "openrouter", "solved_at": "2026-09-14T16:33:26.614Z", "version": "5"}

Answer 2

I built, tested, and verified the complete solution. It's saved at ~/walwork/SOLUTION.md, with the runnable script at ~/walwork/wal. Here is the markdown deliverable:

Crash-safe single-file WAL in bash: forward-resync replay and write/fsync/rename

Problem recap

We need a POSIX-shell (bash 5 + coreutils) write-ahead log where:

Root-cause analysis

Three independent defects make this class of implementation fail:

  1. Non-atomic append. Writing the new record directly into the live WAL with printf >> wal (or dd at end) means a crash between the length prefix and the payload leaves a torn record in the only file replay reads, and every later record is unreachable. Writing to a staging file, fsync-ing it, then rename(2)-ing over the WAL makes publish atomic: the visible WAL is always the old complete prefix or the new complete prefix. The directory must also be fsync-ed.

  2. Stop-at-first-bad-byte replay. Length-prefix framing alone is not self-synchronizing. Aborting on the first bad magic/CRC silently drops later durably committed records. The fix is a sync marker before every record plus a scanner that tests every candidate marker offset and advances one byte on failure; CRC filters false markers inside payloads/garbage.

  3. Tail repair by rewrite. Rebuilding the file can drop/re-encode valid records. The correct primitive is to find the offset just past the last fully valid record and truncate(1) to it.

On-disk frame format

+--------+------------+---+------------+---+------------------+---+
| "WAL1" | 8 hex len  |SP | 8 hex CRC  |LF | payload (len B)  |LF |
+--------+------------+---+------------+---+------------------+---+
   4 B        8 B       1        8 B      1        len B          1

Header is fixed 22 bytes (HDR=22). len = payload byte count as 8 lowercase hex digits; CRC = coreutils cksum CRC-32 of the payload, also %08x. Trailing LF is a torn-record guard. WAL1 is the sync marker.

wal-replay scans for every occurrence of WAL1 (grep -abo); for each candidate it validates the header regex, bounds-checks the claimed length, verifies the trailing LF, recomputes cksum, and compares. Valid records are emitted and advance last_valid_end; invalid candidates are skipped and scanning continues at the next marker.

The fix (complete, single file)

Save as wal, chmod +x wal, then:

ln -s wal wal-append
ln -s wal wal-replay
ln -s wal wal-truncate-tail
ln -s wal wal-harness

It dispatches on both $0 basename and the first argument, so ./wal-append f p and ./wal wal-append f p both work.

#!/usr/bin/env bash
# wal - a crash-safe single-file write-ahead log.
#
# Framing:  "WAL1" <8 hex length> SP <8 hex CRC32> LF <payload> LF
# Commands: wal-append FILE PAYLOAD
#           wal-replay [--stats|--count] FILE
#           wal-truncate-tail FILE
#           wal-harness [TRIALS] [SEED]
set -u

MAGIC='WAL1'
HDR=22                                   # 4 magic + 8 len + 1 sp + 8 crc + 1 lf
RE='^WAL1([0-9a-f]{8}) ([0-9a-f]{8})$'

WAL_COUNT=0
WAL_LAST_END=0
WAL_SIZE=0

die() { printf 'wal: %s\n' "$*" >&2; exit 1; }
need_file() { [ -f "$1" ] || die "no such file: $1"; }

crc_stdin() { cksum | cut -d' ' -f1; }
byte_len()  { local n; n=$(printf '%s' "$1" | wc -c); printf '%s' "${n//[[:space:]]/}"; }

# Emit one fully framed record for PAYLOAD on stdout.
frame() {
    local p=$1 len crc
    len=$(byte_len "$p")
    crc=$(printf '%s' "$p" | crc_stdin)
    printf '%s%08x %08x\n' "$MAGIC" "$len" "$crc"
    printf '%s' "$p"
    printf '\n'
}

# Forward scan.  Prints one "<start> <end> <len> <crc>" line per *valid*
# record and sets WAL_COUNT / WAL_LAST_END / WAL_SIZE.  It never stops at
# the first bad byte: every candidate magic offset is tested and scanning
# resumes after it, so a valid record after garbage is still found.
_scan() {
    local file=$1 size off m h len crc end term got
    size=$(stat -c %s -- "$file")
    WAL_SIZE=$size
    WAL_COUNT=0
    WAL_LAST_END=0
    while IFS= read -r off; do
        [ -n "$off" ] || continue
        m=$off
        [ $((m + HDR)) -le "$size" ] || continue
        h=$(dd if="$file" bs=1 skip="$m" count="$HDR" 2>/dev/null | tr -d '\000')
        [[ $h =~ $RE ]] || continue
        len=$((16#${BASH_REMATCH[1]}))
        crc=$((16#${BASH_REMATCH[2]}))
        end=$((m + HDR + len + 1))
        [ "$end" -le "$size" ] || continue
        term=$(dd if="$file" bs=1 skip=$((m + HDR + len)) count=1 2>/dev/null \
               | od -An -tu1 | tr -d ' \n')
        [ "$term" = "10" ] || continue
        got=$(dd if="$file" bs=1 skip=$((m + HDR)) count="$len" 2>/dev/null | crc_stdin)
        [ "$got" = "$crc" ] || continue
        printf '%s %s %s %s\n' "$m" "$end" "$len" "$crc"
        WAL_COUNT=$((WAL_COUNT + 1))
        [ "$end" -gt "$WAL_LAST_END" ] && WAL_LAST_END=$end
    done < <(grep -abo -- "$MAGIC" "$file" 2>/dev/null | cut -d: -f1)
}

# ---- commands -------------------------------------------------------------

cmd_append() {
    [ $# -ge 2 ] || die 'usage: wal-append FILE PAYLOAD'
    local file=$1 p=$2 dir base tmp
    dir=$(dirname -- "$file")
    base=$(basename -- "$file")
    tmp="$dir/.${base}.tmp.$$"
    if [ -f "$file" ]; then
        cat -- "$file" > "$tmp" || { rm -f "$tmp"; die 'staging failed'; }
    else
        : > "$tmp"
    fi
    frame "$p" >> "$tmp" || { rm -f "$tmp"; die 'write failed'; }
    sync -f "$tmp" 2>/dev/null || sync          # fsync the new bytes
    mv -f -- "$tmp" "$file" || { rm -f "$tmp"; die 'rename failed'; }
    sync -f "$dir" 2>/dev/null || sync          # fsync the rename
}

cmd_replay() {
    local mode=raw
    if [ "${1:-}" = "--stats" ] || [ "${1:-}" = "--count" ]; then
        mode=$1; shift
    fi
    local file=$1
    need_file "$file"
    local meta line m end len crc
    meta=$(mktemp)
    _scan "$file" > "$meta"
    case $mode in
        --stats) printf 'recovered=%d last_valid_end=%d size=%d\n' \
                         "$WAL_COUNT" "$WAL_LAST_END" "$WAL_SIZE" ;;
        --count) printf '%d\n' "$WAL_COUNT" ;;
        *)  while read -r m end len crc; do
                dd if="$file" bs=1 skip=$((m + HDR)) count="$len" 2>/dev/null
                printf '\n'
            done < "$meta" ;;
    esac
    rm -f "$meta"
}

cmd_truncate_tail() {
    local file=$1 meta
    need_file "$file"
    meta=$(mktemp)
    _scan "$file" > "$meta"
    rm -f "$meta"
    if [ "$WAL_LAST_END" -lt "$WAL_SIZE" ]; then
        truncate -s "$WAL_LAST_END" -- "$file" || die 'truncate failed'
        sync -f "$file" 2>/dev/null || sync
    fi
    printf 'truncated_to=%d\n' "$WAL_LAST_END"
}

# 200-trial crash/recovery harness.  Each trial:
#   1. atomically commit a random clean prefix,
#   2. SIGKILL a live writer at a random moment and prove no committed
#      record was lost by the atomic write+fsync+rename discipline,
#   3. splice a torn tail -- clean prefix cut mid-record, one garbage
#      record, then a *valid* record after the garbage, then a truncated
#      frame -- and require replay to recover every complete record
#      including the one after the garbage (forward resync),
#   4. repair with wal-truncate-tail and require no valid record lost.
cmd_harness() {
    local trials=${1:-200} seed=${2:-12345}
    RANDOM=$seed
    local work t fail=0 tot_rec=0 tot_torn=0 tot_lost=0
    work=$(mktemp -d)
    for ((t = 0; t < trials; t++)); do
        local d wal i
        d="$work/t$t"
        wal="$d/wal"
        mkdir -p "$d"
        local n=$(( (RANDOM % 8) + 1 ))
        for ((i = 0; i < n; i++)); do cmd_append "$wal" "clean-$t-$i"; done
        cp -- "$wal" "$d/clean"
        _scan "$d/clean" >/dev/null
        local clean_count=$WAL_COUNT size
        size=$(stat -c %s -- "$d/clean")

        # -- SIGKILL a live writer mid-append ------------------------------
        ( while :; do cmd_append "$wal" "live-$t-$RANDOM"; done ) &
        local wpid=$!
        sleep "0.0$(( (RANDOM % 9) + 1 ))"
        kill -9 "$wpid" 2>/dev/null
        wait "$wpid" 2>/dev/null
        _scan "$wal" >/dev/null
        tot_rec=$((tot_rec + WAL_COUNT))
        if [ "$WAL_COUNT" -lt "$clean_count" ]; then
            printf 'LOST after SIGKILL trial %d: %d < %d\n' "$t" "$WAL_COUNT" "$clean_count" >&2
            tot_lost=$((tot_lost + clean_count - WAL_COUNT))
            fail=1
        fi
        if [ "$WAL_LAST_END" -ne "$WAL_SIZE" ]; then
            printf 'TORN COMMITTED trial %d: last=%d size=%d (atomicity broken)\n' \
                   "$t" "$WAL_LAST_END" "$WAL_SIZE" >&2
            tot_torn=$((tot_torn + 1))
            fail=1
        fi

        # -- resync past garbage + trailing torn tail ----------------------
        local cut=$(( RANDOM % (size + 1) ))
        head -c "$cut" -- "$d/clean" > "$d/torn"
        printf 'GARBAGE-%d\n' "$t" >> "$d/torn"
        printf '\000\377\376 not a record' >> "$d/torn"
        frame "after-$t" >> "$d/torn"
        frame "torn-$t" | head -c 11 >> "$d/torn"

        head -c "$cut" -- "$d/clean" > "$d/prefix"
        _scan "$d/prefix" >/dev/null
        local base=$WAL_COUNT expected=$((WAL_COUNT + 1))
        _scan "$d/torn" >/dev/null
        tot_rec=$((tot_rec + WAL_COUNT))
        [ "$WAL_LAST_END" -lt "$WAL_SIZE" ] && tot_torn=$((tot_torn + 1))
        if [ "$WAL_COUNT" -ne "$expected" ]; then
            printf 'RESYNC FAIL trial %d: got=%d expected=%d cut=%d\n' \
                   "$t" "$WAL_COUNT" "$expected" "$cut" >&2
            tot_lost=$((tot_lost + expected - WAL_COUNT))
            fail=1
        fi

        # -- in-place repair ----------------------------------------------
        cmd_truncate_tail "$d/torn" >/dev/null
        _scan "$d/torn" >/dev/null
        if [ "$WAL_COUNT" -ne "$expected" ] || [ "$WAL_LAST_END" -ne "$WAL_SIZE" ]; then
            printf 'TRUNCATE FAIL trial %d: count=%d expected=%d last=%d size=%d\n' \
                   "$t" "$WAL_COUNT" "$expected" "$WAL_LAST_END" "$WAL_SIZE" >&2
            fail=1
        fi
        rm -rf "$d"
    done
    rm -rf "$work"
    printf 'trials=%d recovered=%d torn=%d lost=%d\n' \
           "$trials" "$tot_rec" "$tot_torn" "$tot_lost"
    return $fail
}

# ---- dispatch -------------------------------------------------------------
# Works both as `wal wal-append FILE ...` and as a symlink/wrapper named
# `wal-append FILE ...` (dispatch on $0 basename; basename form does not shift).
case "${0##*/}" in
    wal-append)        cmd_append "$@" ;;
    wal-replay)        cmd_replay "$@" ;;
    wal-truncate-tail) cmd_truncate_tail "$@" ;;
    wal-harness)       cmd_harness "$@" ;;
    *)
        case "${1:-}" in
            wal-append)        shift; cmd_append "$@" ;;
            wal-replay)        shift; cmd_replay "$@" ;;
            wal-truncate-tail) shift; cmd_truncate_tail "$@" ;;
            wal-harness)       shift; cmd_harness "$@" ;;
            *) die "usage: $0 {wal-append|wal-replay|wal-truncate-tail|wal-harness} ..." ;;
        esac ;;
esac

Usage

./wal-append      my.wal "payload bytes"
./wal-replay      my.wal              # print committed payloads in order
./wal-replay --count my.wal          # number of valid records
./wal-replay --stats my.wal          # recovered=.. last_valid_end=.. size=..
./wal-truncate-tail my.wal           # shrink to end of last valid record
./wal-harness 200 20240914           # 200 SIGKILL/torn trials, nonzero on loss

Verification

Executed on bash 5.3.9 with coreutils cksum/dd/truncate.

1. Append + replay round-trip

$ ./wal-append test.wal 'hello world'
$ ./wal-append test.wal 'second record'
$ ./wal-replay test.wal
hello world
second record
$ ./wal-replay --stats test.wal
recovered=2 last_valid_end=70 size=70

2. Forward resync past a torn record — the root-cause case

A committed record, then a torn record (a header claiming 40 payload bytes that never arrived), then another committed record:

$ ./wal-append demo.wal 'committed-1'
$ printf 'WAL1%08x %08x\n' 40 12345678 >> demo.wal   # torn record
$ ./wal-append demo.wal 'committed-2-after-torn'

$ ./wal-replay demo.wal
committed-1
committed-2-after-torn

$ ./wal-replay --stats demo.wal
recovered=2 last_valid_end=101 size=101

The deliberately naive reader used for the contrast:

#!/usr/bin/env bash
# naive.sh -- stops at the first invalid record (the buggy behaviour)
file=$1; pos=0; n=0; size=$(stat -c %s -- "$file")
while :; do
  [ $((pos+22)) -le "$size" ] || break
  h=$(dd if="$file" bs=1 skip="$pos" count=22 2>/dev/null | tr -d '\000')
  [[ $h =~ ^WAL1([0-9a-f]{8})\ ([0-9a-f]{8})$ ]] || break   # STOP on first bad
  len=$((16#${BASH_REMATCH[1]})); crc=$((16#${BASH_REMATCH[2]}))
  end=$((pos+22+len+1)); [ "$end" -le "$size" ] || break
  got=$(dd if="$file" bs=1 skip=$((pos+22)) count="$len" 2>/dev/null | cksum | cut -d' ' -f1)
  [ "$got" = "$crc" ] || break                              # STOP on bad CRC
  n=$((n+1)); pos=$end
done
printf 'naive_recovered=%d\n' "$n"

Observed contrast — the naive version loses the checksum-valid record after the torn one:

$ ./naive.sh demo.wal
naive_recovered=1

3. wal-truncate-tail repairs in place

$ ./wal-replay --stats torn.wal
recovered=2 last_valid_end=79 size=89
$ ./wal-truncate-tail torn.wal
truncated_to=79
$ ./wal-replay --stats torn.wal
recovered=2 last_valid_end=79 size=79

Both valid records survive; only trailing garbage is removed (89 → 79 bytes). No valid bytes rewritten.

4. SIGKILL + torn-tail harness, 200 trials

Each trial commits a random clean prefix, SIGKILLs a live writer mid-append, verifies the visible WAL is still a complete prefix, then splices a torn tail (clean prefix cut mid-record + garbage record + valid record after garbage + truncated frame). Replay must recover every complete record including the one after garbage; wal-truncate-tail must leave no trailing bytes.

$ time ./wal-harness 200 20240914
trials=200 recovered=1848 torn=200 lost=0

real    1m51.7s
$ echo $?
0

5. Edge cases

$ ./wal-append e.wal ''            # empty payload
$ ./wal-replay --count e.wal
3
$ : > empty.wal; ./wal-replay --stats empty.wal
recovered=0 last_valid_end=0 size=0
$ printf 'nothing valid here' > garb.wal; ./wal-truncate-tail garb.wal
truncated_to=0

Requirement cross-check

Requirement Where it is satisfied
length-prefixed records 22-byte header with %08x length
CRC32-checksummed cksum over payload, stored %08x, re-verified
write → fsync → rename cmd_append: staging write, sync -f staging, mv, sync -f dir
crash never commits a partial frame atomic rename(2); harness TORN COMMITTED check
replay scans past torn/truncated tail _scan tests every WAL1 candidate, never aborts
valid record after garbage replayed §2 and harness step 3
wal-truncate-tail in place, no rewrite truncate -s "$WAL_LAST_END" (shrink only)
harness 200 SIGKILLs, recovered/torn/lost, nonzero on loss cmd_harness, §4

Verified artifacts: ~/walwork/wal (script), ~/walwork/SOLUTION.md, ~/walwork/naive.sh. Measured results: trials=200 recovered=1848 torn=200 lost=0, exit 0; naive replay recovers 1 where forward resync recovers 2.

Evidence & signatures

# Evidence
- Problem class: bash-wal-torn-record-crash-recovery
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-14T16:33:26.613Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Build a POSIX-shell (bash 5, coreutils only, no sqlite/other stores) single-file write-ahead log in which every record is length-prefixed and CRC32-checksummed, appends follow a strict write-fsync-rename ordering discipline so a crash can never expose a partially framed record as committed, and replay reconstructs exactly the committed prefix. Recovery must scan forward past a torn or truncated tail instead of stopping at the first bad record, so a checksum-valid record appearing after one garbage record is still replayed, and a wal-truncate-tail subcommand must repair the file in place without rewriting valid records. Ship wal-append, wal-replay and wal-truncate-tail subcommands plus a harness that SIGKILLs the writer at randomized offsets 200 times and prints recovered/torn/lost counts with a nonzero exit when any checksum-valid record is lost.", "environment": "bash", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-wal-torn-record-crash-recovery", "provider": "openrouter", "solved_at": "2026-09-14T16:33:26.614Z", "version": "5"}
Generated from the verified corpus · MIT licensedBack to the catalog