Running Your First Chaos Drill Without Breaking Prod

A useful first drill tests one concrete hypothesis: what happens if this specific database replica disappears, what happens if this specific downstream API starts timing out instead of failing fast. Vague goals produce vague results and make it hard to know afterward whether the drill actually proved anything. Write the hypothesis down before you start, something like: this system should fail over within thirty seconds if this dependency disappears, and design the drill to test exactly that claim.

The single most important decision in any drill is how much of the system, and how many real users, it can possibly affect if something goes wrong. Start with the smallest scope that can still test the hypothesis: a single instance, a single low traffic region, a single internal service with no customer facing dependency. Only expand scope once smaller drills have run clean and the team trusts the abort mechanism. A drill that could plausibly take down the whole product is not a first drill, no matter how confident the team is going in.

Every drill needs a way to instantly stop the injected failure and return the system to normal, tested separately from the drill itself before it ever runs against anything real. If the kill switch takes longer to work than the failure takes to cause damage, the drill isn't actually safe, it's a real incident with extra steps. Practice pulling the kill switch on a system that isn't currently broken first, so the team knows exactly what it does and how fast it works before relying on it under pressure.

A single chaos drill run once a year tests the system as it existed a year ago, not as it exists today after a dozen deploys changed its behavior. Smaller, more frequent drills, monthly or even weekly for critical paths, catch regressions closer to when they were introduced and keep the team's on call response current. On-demand deploy teams in the fastest DORA cluster, the ones shipping every day, tend to treat drills as a routine part of that cadence, while teams stuck deploying once a month or less often let drills lapse for the same reason they let everything else slow down¹. The drill schedule and the deploy schedule end up reinforcing each other, for better or worse.

The value of a drill isn't the drill itself, it's the gap between what you expected to happen and what actually happened. A drill that confirms the system behaved exactly as designed is a fine outcome, but a drill that surfaces a missing timeout, an alert that never fired, or a fallback path that was quietly broken is the one worth running again to confirm the fix. Document every drill's hypothesis, actual result, and any follow up work, in the same place every time, so patterns across drills are visible instead of scattered across individual engineers' memories.

A first drill runs in this order:

1. Choose one concrete failure to test, such as a specific replica disappearing or one downstream API timing out.
2. Set the smallest scope that can still test the idea: a single instance, a low traffic region or an internal service.
3. Confirm the kill switch stops the injected failure quickly, testing it separately before the drill touches anything real.
4. Run the drill and compare what actually happened against what you expected, noting every surprise.
5. Document the findings, fix gaps such as a missing timeout or a silent alert, and rerun to confirm the fix.

A drill scheduled for a leadership review, with a known outcome and a rehearsed narrative, teaches the team almost nothing, because everyone already knows what's supposed to happen and quietly makes sure it does. The whole value of a drill comes from genuine uncertainty about the outcome. If your team can predict exactly how a drill will go before running it, either the hypothesis is too safe to be useful or the drill has drifted into theater. Keep drills small enough that they're routine, not performances, and resist the urge to polish one into a showcase the first time someone senior wants to watch.