The Backup You've Never Restored Isn't a Backup

The job succeeding means the write completed and the file landed in storage. It says nothing about whether that file is structurally valid, whether the restore script still matches your current schema, or whether whoever needs to run it during an incident knows how. Treat 'backup succeeded' and 'restore verified' as two separate facts you track, not one. Most teams only ever measure the first.

For example, a team can track two separate columns: last successful backup job and last verified restore. The first column is green every day, while the second reads never. That gap is the finding. Closing it takes one scheduled drill that restores a copy into an isolated environment, checks a few known records, and writes the outcome next to the date. Until that drill exists, the honest statement about recoverability is that it hasn't been proven, no matter how many green checkmarks the backup job has collected.

A drill that restores into a throwaway environment, a fresh database instance nobody else is using, lets you test the real mechanics without any risk to what's actually running. Automate the teardown afterward so the drill environment doesn't linger as an unmonitored, unpatched copy of production data sitting around indefinitely. If your data includes anything sensitive, treat the drill environment with the same access controls as production itself, not a lighter version.

The number that matters during a real incident is how long the restore takes, not whether it's technically possible. A restore that works but takes six hours is a very different commitment than one that takes six minutes, especially against a tight downtime budget: a 99.95% uptime target leaves roughly 4.38 hours a year to spend across every incident combined¹, so a single slow restore can burn most of an entire year's allowance. Write the actual timed result down after every drill, not an estimate.

Restoring last night's full backup is the easy case. The harder, more realistic ones are point-in-time restore to a specific minute before a bad migration ran, or restoring a single table without touching everything else. If your only tested path is 'restore everything from last night,' you don't actually know whether the narrower, more common request, get this one table back to ten minutes ago, is even possible with your current tooling. Say a bad migration corrupts a customer's order history at 2 p.m.: the request you'll actually get is 'get this one table back to before 2 p.m. without touching anything else,' not 'restore the whole database from last night.' If that narrower request has never been tested, you won't find out it's unsupported until someone's asking for it during the incident itself.

A restore runbook that references institutional knowledge, the person who set this up left eight months ago, and their shorthand notes assume context nobody else has, is a runbook that fails exactly when it's needed most, during an incident when the original author might be asleep, on vacation, or gone. Have someone who didn't write the runbook run the next drill from it, cold. Every place they get stuck is a gap to fix before it's a real outage.

Quarterly is a reasonable default cadence for most systems, monthly for anything that would be genuinely painful to lose. A tool like ClickUp can hold the drill calendar and the result of each one, timed duration, what broke, what got fixed, so a pattern across drills is visible instead of scattered across whoever happened to run each one. Compliance platforms such as Vanta can often ingest that same drill log as evidence when a SOC 2 auditor asks whether disaster recovery is actually tested, not just documented as a policy; confirm the integration you need in the vendor's current documentation.

A useful restore drill follows these steps:

1. Restore into a throwaway environment that nobody else uses, never toward production.
2. Time the restore from start to finish and write the actual result down, not an estimate.
3. Test the request a real incident would produce, such as point in time recovery or a single table.
4. Have someone who didn't write the runbook run the drill from it, cold.
5. Tear the drill environment down afterward and log what broke and what got fixed.