A cron backup that overlaps the previous run and doubles disk usage

2026-11-09 · 4 min read

A cron backup that overlaps the previous run and doubles disk usage

The dump started at 02:00 and took 71 minutes. The next one started at 03:00 as scheduled, because cron does not care what the last job is still doing. By 04:10 there were two mysqldump processes writing two full copies of the same database to /var/backups, and by 05:00 the volume was at 100% — a size that comfortably held one dump but not two. The database stayed up. The backup script kept "succeeding" right up until the moment it couldn't write, and then it started failing loudly, at 02:00 and 03:00 and 04:00, every hour, forever.

Why the overlap doubles disk usage and hides itself

Each run writes a complete artifact. Two concurrent runs write two. If a run takes longer than the interval, the count grows by one per interval until the schedule and the runtime settle into an accidental equilibrium, or until the disk fills.

The failure is quiet because every individual run looks healthy: exit code 0, log line present, file of roughly the expected size. Nothing in a per-run exit code describes the *number* of concurrent runs, so monitoring that only checks exit codes will report green while the disk drains. Partial files from a killed run are also easy to mistake for complete ones, which matters when retention eventually deletes by age and keeps whatever is newest. [link needed]

How to prevent overlapping cron backup runs

Hold a lock for the duration of the run, and make a second invocation exit immediately instead of queueing. With flock from util-linux, wrapping the whole job is a two-line change to the crontab.

1. Confirm flock is available: command -v flock. It ships with util-linux on most distributions. [link needed]

2. Move the body of the backup into a script, e.g. /usr/local/bin/db-backup.sh, so the lock wraps everything including compression and upload.

3. Change the crontab entry to take the lock non-blockingly and exit if it is held:

0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh

4. Make the "skipped" case visible. flock -n exits 1 when the lock is held, so log a distinguishable message rather than letting it look like a backup failure.

5. Put the lock file on a local filesystem, not on NFS, and ensure the user running the job can create it.

6. Test it: start a slow run by hand, then trigger the cron entry manually and confirm the second invocation exits 1 without writing a file.

Two properties matter. -n (non-blocking) is what prevents a queue from forming — blocking flock without -n would just stack up waiting processes and eventually run every missed backup back to back, which is the same disk problem with extra steps. And the lock must be held by the process that does the work, not by a wrapper that forks and returns.

Verify the fix, don't assume it

Check that a skipped run is logged, that the lock is released when the script exits non-zero or is killed (flock releases on process exit, including on SIGKILL), and that the lock path is unique per job — a shared lock file across backup and restore scripts will block the wrong pairs. Also confirm how long a normal run takes: time /usr/local/bin/db-backup.sh. If the runtime is 70 minutes against an hourly schedule, the lock prevents overlap but you are still running nearly back to back, and a shorter interval is a separate fix. [link needed]

When NOT to do this

Skip the -n lock if the job genuinely must run every interval and the artifacts are independent, cumulative and cheap — for example an append-only log shipper where a missed run loses data. Skipping means a gap, so a non-blocking lock trades coverage for disk safety; a blocking lock or a queueing supervisor such as systemd timers with a persistent queue may suit better. It is also the wrong tool when the real problem is a single run outliving its window: fix the runtime with incremental or snapshot-based backups before adding locks. And if backup and restore share a lock, don't use -n — a restore should wait, not skip.

Checklist

ad slot — paste the network tag here