A Backup You Haven't Restored From Is Just a File

A backup job can succeed for weeks while writing a file that's silently corrupted, incomplete, or missing a table that got added after the backup script was last updated. Nothing about a green checkmark in your job scheduler checks any of that, because the job's own definition of success is just "the write completed without an error."

The only test that actually validates a backup is using it to rebuild a working database somewhere else, and comparing what comes out against what you expected to see. Anything short of that is trusting the file, not verifying it.

A recovery time objective, how long you're willing to be down while restoring, only means something once you've actually measured how long a restore takes on your real data volume, not a guess based on how big the file looks. Teams routinely set an RTO of an hour without ever having timed a full restore, and then discover during a real incident that it takes four.

If your availability target is 99.9 percent, you're defending about 8.76 hours of downtime for the entire year¹, and a single restore that runs long can consume a meaningful share of that budget in one incident. Time an actual restore before you commit to a number anyone outside engineering will hold you to.

Pick a recent backup and restore it into an isolated environment on a fixed schedule, monthly is reasonable for most teams. Run a specific verification query against the restored data, row counts on your most important tables, a spot check on recent records, not just "did the restore command exit with a success code."

Write down every step as you go the first time, including the parts that felt obvious in the moment, because the person running this drill during a real incident may not be the person who ran it calmly last month. A runbook written under no time pressure is far more reliable than instructions reconstructed from memory during one.

A monthly restore drill can follow these steps:

1. Pick a recent backup and restore it into an isolated environment that mirrors production closely enough to be a fair test.
2. Time the restore, and compare the result against the recovery time objective you have set.
3. Compare row counts on your most critical tables against a known-recent snapshot.
4. Spot check that a handful of recent records exist and look correct.
5. Record anything that failed, such as a missing credential, network path or tool version, and fix it before the next drill.

The most common discovery isn't that the backup is missing data, it's that the restore process depends on a credential, a network path, or a tool version that quietly changed since the runbook was last tested. A restore script that worked fine six months ago can fail on a database version bump, a rotated credential, or a decommissioned staging environment it was pointed at.

The second most common discovery is time: a restore that takes far longer than anyone assumed, because nobody had measured it against current data volume rather than the volume from when the process was first built. Both findings are only valuable if the drill actually runs; a runbook nobody has executed recently is a document, not a tested procedure.

Restore drills get skipped for the same reason tech debt gets skipped: nothing forces them onto the calendar, and there's always a feature that feels more urgent this week. Put the drill on a recurring calendar invite with a named owner, and treat a skipped drill the same way you'd treat a skipped deploy of a security patch, not the same way you'd treat a deferred nice to have.

Track the last successful drill date somewhere visible, the same dashboard where you'd track uptime, so it's obvious at a glance whether the team's confidence in its backups is current or stale.