sh-flock-fd-inheritance-hang

Answer

Root cause. flock(2) locks are attached to the open file description, not to the guarding process. A descriptor opened by exec 9>"$LOCK" carries no FD_CLOEXEC, so every &-backgrounded child (and its descendants) inherits fd 9 through fork. The kernel therefore keeps the lock held until every inherited descriptor is closed — long after the guard script exits. The next cron invocation then blocks (or, with flock -n, spuriously reports "already running").

Fix principle. Guarantee that no process outliving the job can hold the lock descriptor. The only holder must be the guard itself, so the lock's lifetime becomes exactly the guard's lifetime. In pure bash this is done by running the job in a subshell that closes the inherited fd before any child can be spawned:

#!/bin/bash
LOCK=/var/lock/cron-job.lock

exec 9>"$LOCK" || exit 1                      # open lock file on fd 9
flock -n 9 || { echo "job already running, skipping"; exec 9>&-; exit 0; }
trap 'exec 9>&-' EXIT                          # belt-and-suspenders release

# Run the protected job in a subshell whose copy of fd 9 is closed first.
# Any background children, daemons, or grandchildren the job spawns are
# forked from that subshell, so they can never inherit the lock descriptor.
(
    exec 9>&-
    run_protected_job                          # may spawn sleep 300 &, daemons, etc.
) &
wait                                           # keep the lock until the job completes

Why it works: the subshell's exec 9>&- closes only its own fd 9 copy (the parent's stays open, so the lock is held throughout). The job then forks children from a process with fd 9 already closed, so the lock is released the moment the guard exits — wait returns, the EXIT trap runs, fd 9 closes, kernel drops the lock. This is also SIGKILL-safe: even a kill -9 of the guard releases the lock, because no other process holds the descriptor.

Alternatives (verified):

• Explicit unlock — flock -u 9 after the job completes also fixes it: since fork duplicates the fd, the child shares the same open file description, and LOCK_UN from the parent unlocks it even while the child still has the fd open. Simpler, but only releases at a single point (e.g., it does not survive kill -9 of the guard mid-job) and depends on util-linux -u.
• flock -c 'job' is not a safe fix on this system: util-linux 2.41.3 execs the command shell without close-on-exec, so background children inherit the lock fd (verified: child held 3 -> /tmp/.../sflockc.test). Avoid it for jobs that background processes, or verify with ls -l /proc/<pid>/fd in your own environment.
• General principle: any helper that opens the lock fd with O_CLOEXEC (Python/Perl fcntl) before execing the job gives the same result; the subshell-close above is the portable bash equivalent.