Why Your Backups Might Not Actually Restore

Most backup tooling reports success based on the job completing, not based on the resulting file being usable. A database that locked mid-backup, a snapshot taken during an inconsistent write, or a storage target silently truncating a large file can all still produce a job that reports success while leaving you with a backup that won't actually restore.

Treat "backup completed" and "backup is restorable" as two separate claims that each need their own verification, because they fail independently of each other.

Automate restoring the latest backup into an isolated environment on a fixed schedule, weekly for a fast-moving product, monthly at minimum for a stable one, rather than relying on someone remembering to test it manually. A drill that depends on a person's memory eventually stops happening the first time that person is busy.

Measure two things every time: did the restore complete without error, and does the resulting data pass a basic integrity check, like row counts matching or a known record being present and correct. A restore that completes but returns corrupted or incomplete data is arguably worse than one that fails outright, because it looks successful.

A restore drill that holds up looks like this:

1. Restore the latest backup into an isolated environment automatically, so the drill never touches production and never depends on someone remembering to run it.
2. Put the drill on a fixed schedule, weekly for a fast-moving product and at least monthly for a stable one.
3. Record whether the restore completed, and how long it took from start to a usable database.
4. Run a basic integrity check on the restored data, since a restore that finishes with incomplete data can look fine until someone relies on it.

How long a restore actually takes matters as much as whether it works, because that time is what you're spending against your availability commitment during a real incident. A team promising 99.9% uptime has a downtime budget of about 0.365 days a year, roughly nine hours, and if your restore drill shows the process alone takes several hours, that's a meaningful share of your entire annual budget spent on data recovery before the rest of the incident response even starts¹.

Track this restore time drill over drill. A restore that gets slower as your data grows is a warning sign worth acting on well before an actual incident forces the question.

A drill that always restores the most recent, healthy backup doesn't tell you what happens if you need to restore from three backups ago, because the most recent one was itself corrupted or taken after a bad deploy. Periodically restore an older backup specifically to confirm your retention window actually contains a usable point to recover from.

This matters more than it sounds like it should: the scenario where you need an old backup is usually the scenario where something has already gone wrong for a while, which is exactly the situation your drills should be built to survive.

Check the restore against a point in time before whatever caused the problem, not just against the most recent snapshot before the incident. A backup taken an hour before a bad migration ran is still a bad backup, even though it's technically recent.

A restore drill that only one specific engineer knows how to run is a single point of failure hiding inside your disaster recovery plan. Document the exact steps, including any credentials or access needed, somewhere the whole on-call rotation can find and follow it without that one person being awake and available.

Run the drill occasionally with a different engineer executing it, following only the written documentation, to confirm the documentation is actually complete rather than relying on tribal knowledge that lives in one person's head.

That handoff test tends to surface gaps a fluent author never notices: a credential that lives in the original engineer's personal password manager, a step that assumes context nobody wrote down. Finding those gaps during a calm, scheduled drill is far better than finding them during a real incident.