A backup job that succeeds but restores nothing: adding a restore check
A backup job that succeeds but restores nothing: adding a restore check
The job finished in 94 seconds and exited 0. The dashboard tile went green, the log line said backup complete, and nobody looked again for eleven days. Then a table got dropped in production and someone tried to restore. The archive unpacked cleanly into an empty directory: 1,204 files, all zero bytes. The backup agent had been dutifully archiving a staging volume that a colleague had unmounted during a host migration. The job kept succeeding because tar was never told the directory was supposed to have anything in it.
The cost wasn't the backup storage. It was the incident call, the eleven days of lost writes, and the hour spent establishing that the *other* backup tier didn't cover that database either.
Why exit code 0 proves almost nothing
A backup job's exit code reports whether the *backup process* finished, not whether the resulting artifact contains data. Common ways that gap opens up:
- The source path exists but is empty after a mount change or a container reschedule.
- The dump tool connects to a replica that has replication stopped.
- Disk fills mid-write and the archive is truncated but valid.
- An exclusion rule (
*.tmp,node_modules, a stale.gitignore-derived list) quietly matches more than intended.
Each of these produces a non-empty, well-formed, useless artifact. Without backup restore verification, the only test your backups pass is "the process ran."
What backup restore verification actually checks
Restore verification means restoring an artifact and asserting on its contents, not just its existence. A useful check has four properties: it reads the artifact the same way a real recovery would, it asserts something concrete (row counts, checksums, file sizes, a specific record you planted), it runs on a schedule rather than once, and it alerts a human when the assertion fails.
The assertions matter more than the mechanism. "Restored without error" is weak; "restored and contains the 40 canary rows written in the last hour, with matching checksums" is a test. Freshness assertions also catch the case where the job is copying an old snapshot over and over.
Scope it to a sample. Full restore tests of a multi-terabyte database on every run are rarely affordable; a table-level sample plus an occasional full drill gets most of the coverage for a fraction of the cost.
A minimal restore check you can wire up this week
1. Create a canary dataset. A small table, directory, or object prefix that a scheduled writer appends to hourly, containing values you can assert on. Keep it out of the paths anyone would exclude.
2. Snapshot the most recent backup. Copy the latest artifact to a scratch location so a corrupt read doesn't touch the original.
3. Restore into an isolated target. A separate database schema, namespace, or temp directory. Never the production instance — a restore check that can overwrite prod is its own outage risk.
4. Assert. Check row count, file count and non-zero size, checksum of the canary values, and that the artifact timestamp is within your RPO window.
5. Fail loudly. Exit non-zero, post to the on-call channel, and page if two consecutive runs fail. A verification job that fails silently is the same failure mode you started with.
6. Log the artifact ID and assertion results. When a real restore happens, you want the history of which artifacts were proven good and when.
Run it on the same schedule as the backup, offset by enough time for the job to complete. Once a quarter, do the slow version by hand: full restore, real recovery steps, a clock running.
When NOT to do this
Skip automated restore verification if the data is genuinely regenerable and cheap to lose — build caches, package mirrors, ephemeral scratch volumes. The check costs money in compute and operator attention, and for a cache it usually isn't worth it.
Also hold off if you have no isolated restore target and no way to get one cheaply; restoring into prod on a timer causes more incidents than it prevents. And don't automate the check before you've done one manual restore end-to-end. Writing assertions for a procedure nobody has executed produces assertions that test the wrong thing — verify the runbook by hand first, then codify it.
Checklist
- [ ] Canary dataset exists, outside every exclusion rule, written on a schedule
- [ ] Restore target is isolated from production
- [ ] Assertions cover count, size, checksum, and artifact age
- [ ] Failures alert a human; two consecutive failures page
- [ ] Verification runs on the same cadence as the backup, offset
- [ ] Quarterly manual full-restore drill on the calendar
- [ ] Runbook executed by hand before any of the above was automated
Run the checklist against your current jobs. If you can't tick the second and third boxes, you already know what your next backup incident looks like — the question is only whether you find out from a page or from a restore.