◐ Off-By-One

shell-ext4-jbd2-journal-replay-idempotent-torn-commit

2 answer(s)shellbashshellbash

A post-mortem tool that replays an extracted ext4 jbd2 journal onto a target block range with a naive "walk the log and dd every tagged block" loop produces metadata that is wrong and that changes on a second run. The symptom is stale metadata resurrected on power-loss recovery, and a target whose bytes depend on how many times recovery ran.

📦 Source in repository (JSON)

Answer 1

The solution is implemented, tested, and written to ~/SOLUTION.md (self-contained with both scripts). Here is the document:

Post-mortem jbd2 journal replay: idempotent, revoke-aware, torn-tail-safe

1. The failure being diagnosed

A post-mortem tool that replays an extracted ext4 jbd2 journal onto a target block range with a naive "walk the log and dd every tagged block" loop produces metadata that is wrong and that changes on a second run. The symptom is stale metadata resurrected on power-loss recovery, and a target whose bytes depend on how many times recovery ran.

2. Root cause analysis

The journal is not a flat write-ahead byte stream; it is a set of transactions that must be reduced to an effective final state before a single byte is written.

Data blocks are anonymous. A descriptor carries a tag list with per-block checksums; the payloads are the blocks immediately following the descriptor. A walker that does not consume N trailing data blocks per descriptor mistakes payload for metadata.
Revoke records undo earlier writes. A revoke for block B invalidates writes to B whose sequence is <= the revoke's sequence. Applying the log left-to-right (or applying every tag) resurrects those stale writes. The correct result is a monotone merge keyed on the highest sequence number: for each block keep the write with the highest sequence Sw such that Sw > max(revoke sequences for B). Because the merge is a pure function of the record set, order does not matter.
A transaction is only durable when its commit validates. Power loss can leave a descriptor plus data blocks with a missing/corrupt commit block (torn tail). Partially applying it corrupts metadata; the whole transaction must be discarded. Per-block tag checksums catch corrupt payloads; the commit checksum catches a torn descriptor/commit group.
Idempotency requires applying a precomputed survivor set. If replay writes while scanning, or reapplies obsolete records on the second run, run-twice differs from run-once. Computing the final block -> payload map first, then writing each survivor exactly once, makes replay a deterministic function of the journal.
An inconsistent header must abort before touching the target. All superblock validation happens before any dd to the target.

3. Format handled

Superblock: h_magic = 0xC03B3998, h_blocktype = 4, s_blocksize @12, s_maxlen @16, s_first @20, s_sequence @24, s_start @28.
Descriptor 1: 12-byte header, then 8-byte tags t_blocknr(4) t_checksum(2) t_flags(2) (--tag64 adds t_blocknr_high(4) => 12-byte tags), terminated by JBD2_FLAG_LAST_TAG (8); the next N blocks are the tagged payloads.
Revoke 5: r_count @12, block numbers from @16.
Commit 2: CRC32 of the whole block with the 4-byte checksum at @16 zeroed.
The ring wraps from s_maxlen-1 back to s_first.

The harness substitutes coreutils cksum (CRC-32) for the kernel's CRC-32C so the deliverable needs no crypto library. Recovery logic is identical; swap crc_range/crc_block_zeroed for CRC-32C to run against a real image.

4. Exact fix

4.1 `jbd2_replay.sh`

#!/usr/bin/env bash
#
# jbd2_replay.sh - deterministic, idempotent post-mortem replay of an
#                  extracted ext4/jbd2 journal image onto a target block range.
#
set -u

MAGIC=$((0xC03B3998))
TYPE_DESC=1; TYPE_COMMIT=2; TYPE_SB=4; TYPE_REVOKE=5
FLAG_ESCAPE=1; FLAG_SAME_UUID=2; FLAG_DELETED=4; FLAG_LAST_TAG=8

PROG=${0##*/}; TAG64=0; DRY_RUN=0; QUIET=0
die()     { printf '%s: error: %s\n' "$PROG" "$*" >&2; exit 3; }
hdr_die() { printf '%s: inconsistent journal header: %s\n' "$PROG" "$*" >&2; exit 2; }
note()    { [ "$QUIET" -eq 1 ] || printf '%s\n' "$*"; }

usage() { cat >&2 <<EOF
usage: $PROG --journal FILE --target FILE [--tag64] [--dry-run] [--quiet]
EOF
    exit 64; }

read_u16() { local f=$1 o=$2 a; read -r -a a < <(od -An -tu1 -j "$o" -N2 "$f")
    [ "${#a[@]}" -lt 2 ] && { echo 0; return; }; echo $(( (a[0]<<8)|a[1] )); }
read_u32() { local f=$1 o=$2 a; read -r -a a < <(od -An -tu1 -j "$o" -N4 "$f")
    [ "${#a[@]}" -lt 4 ] && { echo 0; return; }; echo $(( (a[0]<<24)|(a[1]<<16)|(a[2]<<8)|a[3] )); }
crc_range() { dd if="$1" bs=1 skip="$2" count="$3" status=none | cksum | awk '{print $1}'; }
crc_block_zeroed() { local f=$1 off=$2 bs=$3 zoff=$4
    { dd if="$f" bs=1 skip="$off" count="$zoff" status=none; printf '\0\0\0\0'
      dd if="$f" bs=1 skip=$((off+zoff+4)) count=$((bs-zoff-4)) status=none; } | cksum | awk '{print $1}'; }

JOURNAL=""; TARGET=""
while [ $# -gt 0 ]; do case "$1" in
    --journal) JOURNAL=${2:-}; shift ;; --target) TARGET=${2:-}; shift ;;
    --tag64) TAG64=1 ;; --dry-run) DRY_RUN=1 ;; --quiet) QUIET=1 ;;
    -h|--help) usage ;; *) die "unknown argument: $1" ;; esac; shift; done
[ -n "$JOURNAL" ] || usage; [ -n "$TARGET" ] || usage; [ -r "$JOURNAL" ] || die "cannot read journal: $JOURNAL"

sb_magic=$(read_u32 "$JOURNAL" 0);   sb_type=$(read_u32 "$JOURNAL" 4)
sb_bs=$(read_u32 "$JOURNAL" 12);     sb_maxlen=$(read_u32 "$JOURNAL" 16)
sb_first=$(read_u32 "$JOURNAL" 20);  sb_seq=$(read_u32 "$JOURNAL" 24)
sb_start=$(read_u32 "$JOURNAL" 28)

[ "$sb_magic" -eq "$MAGIC" ]      || hdr_die "bad magic $(printf 0x%08x "$sb_magic")"
[ "$sb_type" -eq "$TYPE_SB" ]     || hdr_die "superblock blocktype is $sb_type"
[ "$sb_bs" -ge 1024 ]             || hdr_die "block size $sb_bs < 1024"
(( (sb_bs & (sb_bs-1)) == 0 ))    || hdr_die "block size $sb_bs not a power of two"
[ "$sb_maxlen" -gt "$sb_first" ]  || hdr_die "maxlen $sb_maxlen <= first $sb_first"
[ "$sb_first" -ge 1 ]             || hdr_die "first log block $sb_first < 1"
[ "$sb_seq" -ge 1 ]               || hdr_die "first sequence $sb_seq < 1"

BS=$sb_bs; MAXLEN=$sb_maxlen; FIRST=$sb_first
jsize=$(stat -c %s "$JOURNAL" 2>/dev/null || echo 0)
[ "$jsize" -ge $((MAXLEN*BS)) ] || hdr_die "journal image too small"
[ "$sb_start" -eq 0 ] && { note "journal is clean (s_start=0); nothing to replay"; exit 0; }
[ "$sb_start" -ge "$FIRST" ] && [ "$sb_start" -lt "$MAXLEN" ] || hdr_die "s_start $sb_start outside [$FIRST,$MAXLEN)"

CAP=$(( MAXLEN - FIRST )); TAGSZ=8; [ "$TAG64" -eq 1 ] && TAGSZ=12
pos=$sb_start; seen=0
adv() { pos=$(( pos+1 )); [ "$pos" -ge "$MAXLEN" ] && pos=$FIRST; seen=$(( seen+1 )); }

WRITES=""; REVOKES=""; ALLSEQ=""; TORNSEQ=""
declare -A TXN_TORN

parse_descriptor() {
    local base=$(( pos*BS )) o=$(( base+12 )) end=$(( base+BS )) blk flags csum hi
    TAG_BLK=(); TAG_CSUM=(); TAG_FLAGS=()
    while [ $(( o+TAGSZ )) -le "$end" ]; do
        blk=$(read_u32 "$JOURNAL" "$o"); csum=$(read_u16 "$JOURNAL" $((o+4))); flags=$(read_u16 "$JOURNAL" $((o+6)))
        if [ "$TAG64" -eq 1 ]; then hi=$(read_u32 "$JOURNAL" $((o+8))); blk=$(( hi*4294967296 + blk )); fi
        TAG_BLK+=("$blk"); TAG_CSUM+=("$csum"); TAG_FLAGS+=("$flags")
        o=$(( o+TAGSZ )); (( flags & FLAG_LAST_TAG )) && break
    done
}

cur_writes=""; cur_revokes=""; have_desc=0; commit_ok=0; torn=0; cur_seq=$sb_seq
while [ "$seen" -lt "$CAP" ]; do
    hm=$(read_u32 "$JOURNAL" $(( pos*BS ))); [ "$hm" -ne "$MAGIC" ] && break
    ht=$(read_u32 "$JOURNAL" $(( pos*BS+4 ))); hs=$(read_u32 "$JOURNAL" $(( pos*BS+8 )))
    [ "$hs" -ne "$cur_seq" ] && break
    case "$ht" in
        "$TYPE_DESC")
            parse_descriptor; n=${#TAG_BLK[@]}; adv; i=0
            while [ "$i" -lt "$n" ]; do
                [ "$seen" -ge "$CAP" ] && { overrun=1; break; }
                src=$pos; blk=${TAG_BLK[$i]}; csum=${TAG_CSUM[$i]}; flags=${TAG_FLAGS[$i]}
                if (( flags & FLAG_DELETED )); then :; else cur_writes+="$cur_seq|$blk|$src|$csum"$'\n'; fi
                adv; i=$(( i+1 ))
            done
            have_desc=1; [ "${overrun:-0}" -eq 1 ] && break ;;
        "$TYPE_REVOKE")
            base=$(( pos*BS )); rcount=$(read_u32 "$JOURNAL" $((base+12))); o=$(( base+16 )); rend=$(( base+rcount ))
            [ "$rend" -gt $(( base+BS )) ] && rend=$(( base+BS ))
            while [ $(( o+4 )) -le "$rend" ]; do blk=$(read_u32 "$JOURNAL" "$o"); [ "$blk" -eq 0 ] && break
                cur_revokes+="$cur_seq|$blk"$'\n'; o=$(( o+4 )); done
            adv ;;
        "$TYPE_COMMIT")
            base=$(( pos*BS )); stored=$(read_u32 "$JOURNAL" $((base+16))); calc=$(crc_block_zeroed "$JOURNAL" "$base" "$BS" 16)
            [ "$stored" -eq "$calc" ] && commit_ok=1 || torn=1
            adv
            if [ "$commit_ok" -eq 1 ]; then
                bad=0
                if [ -n "$cur_writes" ]; then while IFS='|' read -r s b src c; do
                    [ -z "$s" ] && continue
                    rc=$(crc_range "$JOURNAL" $(( src*BS )) "$BS")
                    [ $(( rc & 0xffff )) -eq "$c" ] || { bad=1; break; }
                done <<< "$cur_writes"; fi
                if [ "$bad" -eq 1 ]; then torn=1; else
                    WRITES+="$cur_writes"; REVOKES+="$cur_revokes"; ALLSEQ+="$cur_seq"$'\n'; fi
            fi
            stop=0
            if [ "$torn" -eq 1 ]; then TORNSEQ+="$cur_seq"$'\n'; TXN_TORN[$cur_seq]=1; stop=1; fi
            cur_writes=""; cur_revokes=""; have_desc=0; commit_ok=0; torn=0; cur_seq=$(( cur_seq+1 ))
            [ "$stop" -eq 1 ] && break ;;
        *) break ;;
    esac
done
if [ -n "$cur_writes" ] || [ -n "$cur_revokes" ] || [ "$have_desc" -eq 1 ]; then
    TORNSEQ+="$cur_seq"$'\n'; TXN_TORN[$cur_seq]=1; fi

# monotone merge keyed on highest sequence
declare -A BEST_SEQ BEST_SRC MAXREV
if [ -n "$WRITES" ]; then while IFS='|' read -r s b src c; do [ -z "$s" ] && continue
    [ -z "${BEST_SEQ[$b]:-}" ] || [ "$s" -gt "${BEST_SEQ[$b]}" ] && { BEST_SEQ[$b]=$s; BEST_SRC[$b]=$src; }
done <<< "$WRITES"; fi
if [ -n "$REVOKES" ]; then while IFS='|' read -r s b; do [ -z "$s" ] && continue
    [ -z "${MAXREV[$b]:-}" ] || [ "$s" -gt "${MAXREV[$b]}" ] && MAXREV[$b]=$s
done <<< "$REVOKES"; fi
SURVIVORS=""
for b in "${!BEST_SEQ[@]}"; do sw=${BEST_SEQ[$b]}; sr=${MAXREV[$b]:-0}
    [ "$sw" -gt "$sr" ] && SURVIVORS+="$b|${BEST_SRC[$b]}|$sw"$'\n'; done

# per-transaction verdicts
declare -A VERDICT T_HASSURV T_HASPRIORREVOKE T_HASREVOKEDWRITE
while IFS= read -r s; do [ -z "$s" ] && continue; VERDICT[$s]="applied"; done <<< "$ALLSEQ"
if [ -n "$WRITES" ]; then while IFS='|' read -r s b src c; do [ -z "$s" ] && continue
    bs=${BEST_SEQ[$b]}; sr=${MAXREV[$b]:-0}
    if [ "$s" -eq "$bs" ] && [ "$bs" -gt "$sr" ]; then T_HASSURV[$s]=1; [ "$sr" -gt 0 ] && T_HASPRIORREVOKE[$s]=1
    elif [ "$sr" -ge "$s" ]; then T_HASREVOKEDWRITE[$s]=1; fi
done <<< "$WRITES"; fi
for s in "${!VERDICT[@]}"; do
    if [ "${T_HASPRIORREVOKE[$s]:-0}" -eq 1 ]; then VERDICT[$s]="revoked-then-applied"
    elif [ "${T_HASSURV[$s]:-0}" -eq 1 ]; then VERDICT[$s]="applied"
    elif [ "${T_HASREVOKEDWRITE[$s]:-0}" -eq 1 ]; then VERDICT[$s]="revoked"; fi
done

if [ "$QUIET" -eq 0 ]; then
    allseq_sorted=$(printf '%s\n' "$ALLSEQ" "$TORNSEQ" | grep -v '^$' | sort -n -u)
    while IFS= read -r s; do [ -z "$s" ] && continue
        v=${VERDICT[$s]:-applied}; [ "${TXN_TORN[$s]:-0}" -eq 1 ] && v="torn-discarded"
        printf 'TXN %s %s\n' "$s" "$v"; done <<< "$allseq_sorted"
    while IFS='|' read -r b src sw; do [ -z "$b" ] && continue
        printf 'REPLAY target-block %s <- journal-block %s (seq %s)\n' "$b" "$src" "$sw"; done <<< "$SURVIVORS"
fi

[ "$DRY_RUN" -eq 1 ] && { note "dry-run: target not modified"; exit 0; }
if [ -n "$SURVIVORS" ]; then while IFS='|' read -r b src sw; do [ -z "$b" ] && continue
    dd if="$JOURNAL" of="$TARGET" bs="$BS" skip="$src" seek="$b" count=1 conv=notrunc status=none \
       || die "write to target failed"
done <<< "$SURVIVORS"; fi
note "replay complete: ${#BEST_SEQ[@]} candidate block(s), $(printf '%s' "$SURVIVORS" | grep -c '^[0-9]') survivor(s)"
exit 0

4.2 `jbd2_harness.sh`

The full harness is embedded in SOLUTION.md. It builds synthetic images (new_image, add_sb, add_txn, add_revoke, write_commit) and asserts six scenarios. Key cases:

normal — two writes, verdict applied, sha256(run2)==sha256(run1)
revoke — seq 2 revokes block 10 and 12; seq 3 rewrites block 10. Asserts block 10 = seq 3 value, block 12 stays sentinel, TXN 1 revoked, TXN 3 revoked-then-applied
torn tail — seq 2 commit checksum corrupted; both seq 2 writes discarded whole, TXN 2 torn-discarded
wraparound — commit wraps from block 63 to block 1
inconsistent header — bad magic => exit 2, target SHA-256 unchanged
corrupt tag checksum — payload poisoned => whole transaction discarded

Run it with ./jbd2_harness.sh.

5. Verification

== jbd2 replay harness ==
[1] normal transaction
  PASS block 10 written (applied)
  PASS block 11 written (applied)
  PASS idempotent: run2 == run1
  PASS verdict: TXN 1 applied
[2] revoke then re-apply (monotone merge on highest sequence)
  PASS block 10 = seq3 (revoked-then-applied)
  PASS block 12 not resurrected (revoke wins)
  PASS idempotent: run2 == run1
  PASS verdict: TXN 1 revoked
  PASS verdict: TXN 3 revoked-then-applied
[3] torn tail: invalid commit checksum discards whole transaction
  PASS seq1 applied
  PASS seq2 discarded whole (block 11 untouched)
  PASS seq2 discarded whole (block 13 untouched)
  PASS idempotent after torn discard
  PASS verdict: TXN 2 torn-discarded
[4] wraparound log ring
  PASS wraparound: block 20 written
  PASS wraparound: block 21 written
  PASS idempotent across wraparound
[5] inconsistent header: abort without touching target
  PASS inconsistent header exits nonzero (2)
  PASS target untouched on header failure
[6] corrupt per-block tag checksum discards its transaction
  PASS seq1 applied
  PASS bad tag checksum discards seq2 whole
  PASS idempotent after tag-checksum discard

harness result: 22 passed, 0 failed

Why idempotency is guaranteed, not just observed

Replay is split into two phases: (1) Reduce — scan the ring once, accept only commit-validated transactions (commit checksum + every tag checksum), then compute BEST_SEQ/BEST_SRC/MAXREV and emit survivors where BEST_SEQ > MAXREV; this phase only reads the journal. (2) Apply — dd each survivor once with conv=notrunc. Phase 1 is a pure function journal -> survivors; phase 2 writes identical bytes to identical offsets every run, so replay² = replay.

Exit codes

Code	Meaning
`0`	replay complete (or clean journal, `s_start == 0`)
`2`	inconsistent journal header — target not touched
`3`	I/O or argument error
`64`	usage

Artifacts on disk: ~/jbd2_replay.sh, ~/jbd2_harness.sh, ~/SOLUTION.md. All 22 assertions pass (./jbd2_harness.sh exits 0).

Evidence & signatures

# Evidence
- Problem class: shell-ext4-jbd2-journal-replay-idempotent-torn-commit
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-23T04:36:34.943Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Write a bash post-mortem tool that replays an extracted ext4 jbd2 journal image onto a target block range and proves the replay is idempotent: running it twice on the same journal must leave the target byte-identical to a single run, and a transaction whose commit-block checksum does not validate (a torn tail from power loss) must be discarded whole rather than partially applied. The journal semantics that make this hard are that descriptor blocks carry a magic tag list with per-block checksums but the order of descriptor and revoke records is not the application order, revoke records invalidate earlier writes to the same block so replay must be a monotone merge keyed on the highest sequence number, and the final commit block documents how many tags are live; a naive left-to-right dd of every tagged block double-applies revoked writes and resurrects stale metadata. Your tool must parse the superblock to locate the log start and sequence, walk descriptor, revocation, and commit blocks in sequence-number order, apply only the surviving writes, report per-transaction verdicts (applied, revoked-then-applied, torn-discarded), and exit non-zero without touching the target when the journal header itself is inconsistent; include a harness that builds synthetic journal images (normal, revoked, torn-tail, wraparound) and asserts the idempotency and torn-tail properties for each.", "environment": "bash", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "shell-ext4-jbd2-journal-replay-idempotent-torn-commit", "provider": "openrouter", "solved_at": "2026-09-23T04:36:34.944Z", "version": "5.2"}

Answer 2

The solution is implemented, tested, and written to ~/SOLUTION.md (self-contained with both scripts). Here is the document:

Post-mortem jbd2 journal replay: idempotent, revoke-aware, torn-tail-safe

1. The failure being diagnosed

2. Root cause analysis

The journal is not a flat write-ahead byte stream; it is a set of transactions that must be reduced to an effective final state before a single byte is written.

Data blocks are anonymous. A descriptor carries a tag list with per-block checksums; the payloads are the blocks immediately following the descriptor. A walker that does not consume N trailing data blocks per descriptor mistakes payload for metadata.
Revoke records undo earlier writes. A revoke for block B invalidates writes to B whose sequence is <= the revoke's sequence. Applying the log left-to-right (or applying every tag) resurrects those stale writes. The correct result is a monotone merge keyed on the highest sequence number: for each block keep the write with the highest sequence Sw such that Sw > max(revoke sequences for B). Because the merge is a pure function of the record set, order does not matter.
A transaction is only durable when its commit validates. Power loss can leave a descriptor plus data blocks with a missing/corrupt commit block (torn tail). Partially applying it corrupts metadata; the whole transaction must be discarded. Per-block tag checksums catch corrupt payloads; the commit checksum catches a torn descriptor/commit group.
Idempotency requires applying a precomputed survivor set. If replay writes while scanning, or reapplies obsolete records on the second run, run-twice differs from run-once. Computing the final block -> payload map first, then writing each survivor exactly once, makes replay a deterministic function of the journal.
An inconsistent header must abort before touching the target. All superblock validation happens before any dd to the target.

3. Format handled

Superblock: h_magic = 0xC03B3998, h_blocktype = 4, s_blocksize @12, s_maxlen @16, s_first @20, s_sequence @24, s_start @28.
Descriptor 1: 12-byte header, then 8-byte tags t_blocknr(4) t_checksum(2) t_flags(2) (--tag64 adds t_blocknr_high(4) => 12-byte tags), terminated by JBD2_FLAG_LAST_TAG (8); the next N blocks are the tagged payloads.
Revoke 5: r_count @12, block numbers from @16.
Commit 2: CRC32 of the whole block with the 4-byte checksum at @16 zeroed.
The ring wraps from s_maxlen-1 back to s_first.

The harness substitutes coreutils cksum (CRC-32) for the kernel's CRC-32C so the deliverable needs no crypto library. Recovery logic is identical; swap crc_range/crc_block_zeroed for CRC-32C to run against a real image.

4. Exact fix

4.1 `jbd2_replay.sh`

#!/usr/bin/env bash
#
# jbd2_replay.sh - deterministic, idempotent post-mortem replay of an
#                  extracted ext4/jbd2 journal image onto a target block range.
#
set -u

MAGIC=$((0xC03B3998))
TYPE_DESC=1; TYPE_COMMIT=2; TYPE_SB=4; TYPE_REVOKE=5
FLAG_ESCAPE=1; FLAG_SAME_UUID=2; FLAG_DELETED=4; FLAG_LAST_TAG=8

PROG=${0##*/}; TAG64=0; DRY_RUN=0; QUIET=0
die()     { printf '%s: error: %s\n' "$PROG" "$*" >&2; exit 3; }
hdr_die() { printf '%s: inconsistent journal header: %s\n' "$PROG" "$*" >&2; exit 2; }
note()    { [ "$QUIET" -eq 1 ] || printf '%s\n' "$*"; }

usage() { cat >&2 <<EOF
usage: $PROG --journal FILE --target FILE [--tag64] [--dry-run] [--quiet]
EOF
    exit 64; }

read_u16() { local f=$1 o=$2 a; read -r -a a < <(od -An -tu1 -j "$o" -N2 "$f")
    [ "${#a[@]}" -lt 2 ] && { echo 0; return; }; echo $(( (a[0]<<8)|a[1] )); }
read_u32() { local f=$1 o=$2 a; read -r -a a < <(od -An -tu1 -j "$o" -N4 "$f")
    [ "${#a[@]}" -lt 4 ] && { echo 0; return; }; echo $(( (a[0]<<24)|(a[1]<<16)|(a[2]<<8)|a[3] )); }
crc_range() { dd if="$1" bs=1 skip="$2" count="$3" status=none | cksum | awk '{print $1}'; }
crc_block_zeroed() { local f=$1 off=$2 bs=$3 zoff=$4
    { dd if="$f" bs=1 skip="$off" count="$zoff" status=none; printf '\0\0\0\0'
      dd if="$f" bs=1 skip=$((off+zoff+4)) count=$((bs-zoff-4)) status=none; } | cksum | awk '{print $1}'; }

JOURNAL=""; TARGET=""
while [ $# -gt 0 ]; do case "$1" in
    --journal) JOURNAL=${2:-}; shift ;; --target) TARGET=${2:-}; shift ;;
    --tag64) TAG64=1 ;; --dry-run) DRY_RUN=1 ;; --quiet) QUIET=1 ;;
    -h|--help) usage ;; *) die "unknown argument: $1" ;; esac; shift; done
[ -n "$JOURNAL" ] || usage; [ -n "$TARGET" ] || usage; [ -r "$JOURNAL" ] || die "cannot read journal: $JOURNAL"

sb_magic=$(read_u32 "$JOURNAL" 0);   sb_type=$(read_u32 "$JOURNAL" 4)
sb_bs=$(read_u32 "$JOURNAL" 12);     sb_maxlen=$(read_u32 "$JOURNAL" 16)
sb_first=$(read_u32 "$JOURNAL" 20);  sb_seq=$(read_u32 "$JOURNAL" 24)
sb_start=$(read_u32 "$JOURNAL" 28)

[ "$sb_magic" -eq "$MAGIC" ]      || hdr_die "bad magic $(printf 0x%08x "$sb_magic")"
[ "$sb_type" -eq "$TYPE_SB" ]     || hdr_die "superblock blocktype is $sb_type"
[ "$sb_bs" -ge 1024 ]             || hdr_die "block size $sb_bs < 1024"
(( (sb_bs & (sb_bs-1)) == 0 ))    || hdr_die "block size $sb_bs not a power of two"
[ "$sb_maxlen" -gt "$sb_first" ]  || hdr_die "maxlen $sb_maxlen <= first $sb_first"
[ "$sb_first" -ge 1 ]             || hdr_die "first log block $sb_first < 1"
[ "$sb_seq" -ge 1 ]               || hdr_die "first sequence $sb_seq < 1"

BS=$sb_bs; MAXLEN=$sb_maxlen; FIRST=$sb_first
jsize=$(stat -c %s "$JOURNAL" 2>/dev/null || echo 0)
[ "$jsize" -ge $((MAXLEN*BS)) ] || hdr_die "journal image too small"
[ "$sb_start" -eq 0 ] && { note "journal is clean (s_start=0); nothing to replay"; exit 0; }
[ "$sb_start" -ge "$FIRST" ] && [ "$sb_start" -lt "$MAXLEN" ] || hdr_die "s_start $sb_start outside [$FIRST,$MAXLEN)"

CAP=$(( MAXLEN - FIRST )); TAGSZ=8; [ "$TAG64" -eq 1 ] && TAGSZ=12
pos=$sb_start; seen=0
adv() { pos=$(( pos+1 )); [ "$pos" -ge "$MAXLEN" ] && pos=$FIRST; seen=$(( seen+1 )); }

WRITES=""; REVOKES=""; ALLSEQ=""; TORNSEQ=""
declare -A TXN_TORN

parse_descriptor() {
    local base=$(( pos*BS )) o=$(( base+12 )) end=$(( base+BS )) blk flags csum hi
    TAG_BLK=(); TAG_CSUM=(); TAG_FLAGS=()
    while [ $(( o+TAGSZ )) -le "$end" ]; do
        blk=$(read_u32 "$JOURNAL" "$o"); csum=$(read_u16 "$JOURNAL" $((o+4))); flags=$(read_u16 "$JOURNAL" $((o+6)))
        if [ "$TAG64" -eq 1 ]; then hi=$(read_u32 "$JOURNAL" $((o+8))); blk=$(( hi*4294967296 + blk )); fi
        TAG_BLK+=("$blk"); TAG_CSUM+=("$csum"); TAG_FLAGS+=("$flags")
        o=$(( o+TAGSZ )); (( flags & FLAG_LAST_TAG )) && break
    done
}

cur_writes=""; cur_revokes=""; have_desc=0; commit_ok=0; torn=0; cur_seq=$sb_seq
while [ "$seen" -lt "$CAP" ]; do
    hm=$(read_u32 "$JOURNAL" $(( pos*BS ))); [ "$hm" -ne "$MAGIC" ] && break
    ht=$(read_u32 "$JOURNAL" $(( pos*BS+4 ))); hs=$(read_u32 "$JOURNAL" $(( pos*BS+8 )))
    [ "$hs" -ne "$cur_seq" ] && break
    case "$ht" in
        "$TYPE_DESC")
            parse_descriptor; n=${#TAG_BLK[@]}; adv; i=0
            while [ "$i" -lt "$n" ]; do
                [ "$seen" -ge "$CAP" ] && { overrun=1; break; }
                src=$pos; blk=${TAG_BLK[$i]}; csum=${TAG_CSUM[$i]}; flags=${TAG_FLAGS[$i]}
                if (( flags & FLAG_DELETED )); then :; else cur_writes+="$cur_seq|$blk|$src|$csum"$'\n'; fi
                adv; i=$(( i+1 ))
            done
            have_desc=1; [ "${overrun:-0}" -eq 1 ] && break ;;
        "$TYPE_REVOKE")
            base=$(( pos*BS )); rcount=$(read_u32 "$JOURNAL" $((base+12))); o=$(( base+16 )); rend=$(( base+rcount ))
            [ "$rend" -gt $(( base+BS )) ] && rend=$(( base+BS ))
            while [ $(( o+4 )) -le "$rend" ]; do blk=$(read_u32 "$JOURNAL" "$o"); [ "$blk" -eq 0 ] && break
                cur_revokes+="$cur_seq|$blk"$'\n'; o=$(( o+4 )); done
            adv ;;
        "$TYPE_COMMIT")
            base=$(( pos*BS )); stored=$(read_u32 "$JOURNAL" $((base+16))); calc=$(crc_block_zeroed "$JOURNAL" "$base" "$BS" 16)
            [ "$stored" -eq "$calc" ] && commit_ok=1 || torn=1
            adv
            if [ "$commit_ok" -eq 1 ]; then
                bad=0
                if [ -n "$cur_writes" ]; then while IFS='|' read -r s b src c; do
                    [ -z "$s" ] && continue
                    rc=$(crc_range "$JOURNAL" $(( src*BS )) "$BS")
                    [ $(( rc & 0xffff )) -eq "$c" ] || { bad=1; break; }
                done <<< "$cur_writes"; fi
                if [ "$bad" -eq 1 ]; then torn=1; else
                    WRITES+="$cur_writes"; REVOKES+="$cur_revokes"; ALLSEQ+="$cur_seq"$'\n'; fi
            fi
            stop=0
            if [ "$torn" -eq 1 ]; then TORNSEQ+="$cur_seq"$'\n'; TXN_TORN[$cur_seq]=1; stop=1; fi
            cur_writes=""; cur_revokes=""; have_desc=0; commit_ok=0; torn=0; cur_seq=$(( cur_seq+1 ))
            [ "$stop" -eq 1 ] && break ;;
        *) break ;;
    esac
done
if [ -n "$cur_writes" ] || [ -n "$cur_revokes" ] || [ "$have_desc" -eq 1 ]; then
    TORNSEQ+="$cur_seq"$'\n'; TXN_TORN[$cur_seq]=1; fi

# monotone merge keyed on highest sequence
declare -A BEST_SEQ BEST_SRC MAXREV
if [ -n "$WRITES" ]; then while IFS='|' read -r s b src c; do [ -z "$s" ] && continue
    [ -z "${BEST_SEQ[$b]:-}" ] || [ "$s" -gt "${BEST_SEQ[$b]}" ] && { BEST_SEQ[$b]=$s; BEST_SRC[$b]=$src; }
done <<< "$WRITES"; fi
if [ -n "$REVOKES" ]; then while IFS='|' read -r s b; do [ -z "$s" ] && continue
    [ -z "${MAXREV[$b]:-}" ] || [ "$s" -gt "${MAXREV[$b]}" ] && MAXREV[$b]=$s
done <<< "$REVOKES"; fi
SURVIVORS=""
for b in "${!BEST_SEQ[@]}"; do sw=${BEST_SEQ[$b]}; sr=${MAXREV[$b]:-0}
    [ "$sw" -gt "$sr" ] && SURVIVORS+="$b|${BEST_SRC[$b]}|$sw"$'\n'; done

# per-transaction verdicts
declare -A VERDICT T_HASSURV T_HASPRIORREVOKE T_HASREVOKEDWRITE
while IFS= read -r s; do [ -z "$s" ] && continue; VERDICT[$s]="applied"; done <<< "$ALLSEQ"
if [ -n "$WRITES" ]; then while IFS='|' read -r s b src c; do [ -z "$s" ] && continue
    bs=${BEST_SEQ[$b]}; sr=${MAXREV[$b]:-0}
    if [ "$s" -eq "$bs" ] && [ "$bs" -gt "$sr" ]; then T_HASSURV[$s]=1; [ "$sr" -gt 0 ] && T_HASPRIORREVOKE[$s]=1
    elif [ "$sr" -ge "$s" ]; then T_HASREVOKEDWRITE[$s]=1; fi
done <<< "$WRITES"; fi
for s in "${!VERDICT[@]}"; do
    if [ "${T_HASPRIORREVOKE[$s]:-0}" -eq 1 ]; then VERDICT[$s]="revoked-then-applied"
    elif [ "${T_HASSURV[$s]:-0}" -eq 1 ]; then VERDICT[$s]="applied"
    elif [ "${T_HASREVOKEDWRITE[$s]:-0}" -eq 1 ]; then VERDICT[$s]="revoked"; fi
done

if [ "$QUIET" -eq 0 ]; then
    allseq_sorted=$(printf '%s\n' "$ALLSEQ" "$TORNSEQ" | grep -v '^$' | sort -n -u)
    while IFS= read -r s; do [ -z "$s" ] && continue
        v=${VERDICT[$s]:-applied}; [ "${TXN_TORN[$s]:-0}" -eq 1 ] && v="torn-discarded"
        printf 'TXN %s %s\n' "$s" "$v"; done <<< "$allseq_sorted"
    while IFS='|' read -r b src sw; do [ -z "$b" ] && continue
        printf 'REPLAY target-block %s <- journal-block %s (seq %s)\n' "$b" "$src" "$sw"; done <<< "$SURVIVORS"
fi

[ "$DRY_RUN" -eq 1 ] && { note "dry-run: target not modified"; exit 0; }
if [ -n "$SURVIVORS" ]; then while IFS='|' read -r b src sw; do [ -z "$b" ] && continue
    dd if="$JOURNAL" of="$TARGET" bs="$BS" skip="$src" seek="$b" count=1 conv=notrunc status=none \
       || die "write to target failed"
done <<< "$SURVIVORS"; fi
note "replay complete: ${#BEST_SEQ[@]} candidate block(s), $(printf '%s' "$SURVIVORS" | grep -c '^[0-9]') survivor(s)"
exit 0

4.2 `jbd2_harness.sh`

The full harness is embedded in SOLUTION.md. It builds synthetic images (new_image, add_sb, add_txn, add_revoke, write_commit) and asserts six scenarios. Key cases:

normal — two writes, verdict applied, sha256(run2)==sha256(run1)
revoke — seq 2 revokes block 10 and 12; seq 3 rewrites block 10. Asserts block 10 = seq 3 value, block 12 stays sentinel, TXN 1 revoked, TXN 3 revoked-then-applied
torn tail — seq 2 commit checksum corrupted; both seq 2 writes discarded whole, TXN 2 torn-discarded
wraparound — commit wraps from block 63 to block 1
inconsistent header — bad magic => exit 2, target SHA-256 unchanged
corrupt tag checksum — payload poisoned => whole transaction discarded

Run it with ./jbd2_harness.sh.

5. Verification

== jbd2 replay harness ==
[1] normal transaction
  PASS block 10 written (applied)
  PASS block 11 written (applied)
  PASS idempotent: run2 == run1
  PASS verdict: TXN 1 applied
[2] revoke then re-apply (monotone merge on highest sequence)
  PASS block 10 = seq3 (revoked-then-applied)
  PASS block 12 not resurrected (revoke wins)
  PASS idempotent: run2 == run1
  PASS verdict: TXN 1 revoked
  PASS verdict: TXN 3 revoked-then-applied
[3] torn tail: invalid commit checksum discards whole transaction
  PASS seq1 applied
  PASS seq2 discarded whole (block 11 untouched)
  PASS seq2 discarded whole (block 13 untouched)
  PASS idempotent after torn discard
  PASS verdict: TXN 2 torn-discarded
[4] wraparound log ring
  PASS wraparound: block 20 written
  PASS wraparound: block 21 written
  PASS idempotent across wraparound
[5] inconsistent header: abort without touching target
  PASS inconsistent header exits nonzero (2)
  PASS target untouched on header failure
[6] corrupt per-block tag checksum discards its transaction
  PASS seq1 applied
  PASS bad tag checksum discards seq2 whole
  PASS idempotent after tag-checksum discard

harness result: 22 passed, 0 failed

Why idempotency is guaranteed, not just observed

Exit codes

Code	Meaning
`0`	replay complete (or clean journal, `s_start == 0`)
`2`	inconsistent journal header — target not touched
`3`	I/O or argument error
`64`	usage

Artifacts on disk: ~/jbd2_replay.sh, ~/jbd2_harness.sh, ~/SOLUTION.md. All 22 assertions pass (./jbd2_harness.sh exits 0).

Evidence & signatures

# Evidence
- Problem class: shell-ext4-jbd2-journal-replay-idempotent-torn-commit
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-23T04:36:34.943Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Write a bash post-mortem tool that replays an extracted ext4 jbd2 journal image onto a target block range and proves the replay is idempotent: running it twice on the same journal must leave the target byte-identical to a single run, and a transaction whose commit-block checksum does not validate (a torn tail from power loss) must be discarded whole rather than partially applied. The journal semantics that make this hard are that descriptor blocks carry a magic tag list with per-block checksums but the order of descriptor and revoke records is not the application order, revoke records invalidate earlier writes to the same block so replay must be a monotone merge keyed on the highest sequence number, and the final commit block documents how many tags are live; a naive left-to-right dd of every tagged block double-applies revoked writes and resurrects stale metadata. Your tool must parse the superblock to locate the log start and sequence, walk descriptor, revocation, and commit blocks in sequence-number order, apply only the surviving writes, report per-transaction verdicts (applied, revoked-then-applied, torn-discarded), and exit non-zero without touching the target when the journal header itself is inconsistent; include a harness that builds synthetic journal images (normal, revoked, torn-tail, wraparound) and asserts the idempotency and torn-tail properties for each.", "environment": "bash", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "shell-ext4-jbd2-journal-replay-idempotent-torn-commit", "provider": "openrouter", "solved_at": "2026-09-23T04:36:34.944Z", "version": "5.2"}

Generated from the verified corpus · MIT licensedBack to the catalog

shell-ext4-jbd2-journal-replay-idempotent-torn-commit

Answer 1

Post-mortem jbd2 journal replay: idempotent, revoke-aware, torn-tail-safe

1. The failure being diagnosed

2. Root cause analysis

3. Format handled

4. Exact fix

4.1 jbd2_replay.sh

4.2 jbd2_harness.sh

5. Verification

Why idempotency is guaranteed, not just observed

Exit codes

Evidence & signatures

Answer 2

Post-mortem jbd2 journal replay: idempotent, revoke-aware, torn-tail-safe

1. The failure being diagnosed

2. Root cause analysis

3. Format handled

4. Exact fix

4.1 jbd2_replay.sh

4.2 jbd2_harness.sh

5. Verification

Why idempotency is guaranteed, not just observed

Exit codes

Evidence & signatures

4.1 `jbd2_replay.sh`

4.2 `jbd2_harness.sh`

4.1 `jbd2_replay.sh`

4.2 `jbd2_harness.sh`