bash-backup-rotation-toctou-path-reuse
The race (TOCTOU path reuse). The prune pipeline splits the check from the use:
find decides which paths to delete based on the state of the filesystem at scan time (name + -mtime +30), and hands a list of paths to xargs.rm executes later and unlinks whatever currently sits at each path — it has no memory of the inode find evaluated.The backup script reuses the fixed path dump.latest.zst in between: it mvs the old archive away to a timestamped name, then spends minutes writing the new dump into the same path. So the path backups/dump.latest.zst can refer to a different file at rm-time than it did at find-time.
Buggy interleaving (occurs when dump.latest.zst is stale — e.g. the last successful backup is >30 days old after an outage, and prune overlaps the backup window):
find stats backups/dump.latest.zst → mtime is old → adds the path to the delete list. The future timestamped archive doesn't exist yet, so it can't be listed.mv dump.latest.zst dump.<ts>.zst (old archive escapes to a new name), then zstd -o dump.latest.zst starts a multi-minute write — the path now names the fresh file.xargs rm fires: unlinks backups/dump.latest.zst → deletes the freshly written archive. The genuinely old archive survives under its new timestamped name (its mtime was old too, but find never saw that path).Because the write takes minutes, the victim window is wide, which is why it happens "occasionally".
Minimal fix (prune side): never make the reusable fixed path a prune target, and close the check/use gap with -delete (stat and unlink happen adjacently in one pass — no decoupled xargs rm phase):
find backups -type f -name '*.zst' ! -name 'dump.latest.zst' -mtime +30 -delete
Why this is safe:
- dump.latest.zst is the only path the backup ever rewrites; excluding it by name means the prune can only ever delete immutable timestamped archives, which the backup script never writes in place.
- If a rotation lands between two prune runs, the escaped archive's mtime is unchanged, so the next run still catches it — deletion is eventually correct, and the fresh archive is never harmed.
- -delete performs the mtime check and the unlink in the same find pass (microseconds apart), eliminating the minutes-long check/use window; it's also whitespace-safe, so -print0 | xargs -0 is no longer needed.
- -type f additionally avoids deleting symlinks/directories matching *.zst.
Complementary hardening (backup side, optional): write the dump atomically so dump.latest.zst is never a partially-written file (also protects consumers reading it mid-write):
mv -f backups/dump.latest.zst backups/dump.$(date +%Y%m%d-%H%M%S).zst
zstd -o backups/dump.latest.zst.tmp && mv -f backups/dump.latest.zst.tmp backups/dump.latest.zst
Note this alone does not fix the prune race (a stale dump.latest.zst could still be listed by find and unlinked by rm after the swap) — the prune-side exclusion is the essential fix.
Reproduced in `/tmp/race-demo.ePutHO` (GNU bash 5.3 / findutils 4.10): **Bug reproduction** — stale `dump.latest.zst` (40d), old `dump.2023-01-01.zst` (45d), young `dump.2023-11-01.zst` (2d): ``` == OLD prune, check phase == find ... -mtime +30 -print0 → dump.2023-01-01.zst, dump.latest.zst # path captured == backup: mv dump.latest.zst dump.2023-03-01.zst; zstd -o dump.latest.zst (mid-write) == == OLD prune, use phase: xargs rm == == after == dump.2023-03-01.zst ← genuinely old archive SURVIVES under new name dump.2023-11-01.zst ← young archive (correctly) survives dump.latest.zst ← GONE: the fresh, mid-write archive was deleted ``` **Fix verification** (`/tmp/race-fix.cD9Y0F`), same setup plus weird names (`dump with spaces 2022-01-01.zst`, and a name containing a newline): - Fixed selection excludes `dump.latest.zst` even though it is 40 days old; only genuinely-old timestamped archives are selected. - With the rotation + fresh write interleaved *while* the fixed prune runs: `dump.latest.zst` survives with all 60 lines of content intact (`FRESH-DUMP-DATA-0001`…`0060`); stale timestamped archives (incl. spaces/newline names) are deleted. - Escaped archive `dump.2023-03-01.zst` (rotated after the find walk) still had its old mtime → selected and removed on the **next** prune run; latest untouched. **Edge cases** (`/tmp/race-edge.*`): - `-mtime +30` boundary: a 30-day-old file is **not** matched; a 31-day-old file is — semantics identical to the original command. - Empty directory / no matches: no-op, exit 0. - Symlink `lnk.zst -> real.zst`: `-type f` leaves it alone (original `rm` would have deleted the symlink). - Atomic-write hardening: mid-write, `dump.latest.zst` does not exist (only `dump.latest.zst.tmp`, which `*.zst` never matches); after rename, the archive appears complete. Tests run: 8 (race repro; fix selection; fix under interleaving; next-run catch-up; mtime boundary; empty dir; symlink; atomic-write state).
{"model": "deepseek-v4-flash", "problem_class": "bash-backup-rotation-toctou-path-reuse", "result": "passed", "tests": 8}