How to Run a Chaos Engineering Drill Without Causing a Real Outage

A chaos drill without a specific hypothesis is just breaking something to see what happens, and it produces interesting stories more often than useful fixes. Start instead with a testable claim: if the primary database connection pool is exhausted, the service should degrade gracefully and shed non-critical requests rather than failing entirely.

The drill then either confirms that claim or, more usefully, disproves it in a contained setting instead of during a real incident where the same discovery costs a lot more than an afternoon of planned testing.

The blast radius, how much of production the failure can actually touch, should be the smallest slice that still tests the hypothesis honestly: a single instance behind a load balancer, a single non-critical service, a small percentage of traffic routed through a feature flag.

Expanding the blast radius should be earned by drills that went cleanly at a smaller scale, not assumed from the start because a bigger test feels more thorough. A contained drill that actually finishes teaches you more than an ambitious one that has to be aborted halfway through.

Document the blast radius in writing before the drill starts, not just in the head of the engineer running the injection. A written boundary, this load balancer, this single instance, this feature flag at a small traffic percentage, gives the person watching dashboards something concrete to check the failure against, and it gives the team something to compare afterward if the drill drifted wider than planned.

Before injecting anything, define the one or two metrics that represent normal operation, error rate, latency at a specific percentile, and confirm they're stable before you start the injection. Without a clear steady state to compare against, you can't tell whether a spike during the drill is the failure you intended or an unrelated problem happening to occur at the same time.

Write that baseline down before the drill begins, not from memory afterward. A remembered baseline has a way of shifting to match whatever the drill produced.

Pick a metric sensitive enough to move when the injected failure actually happens, but stable enough on a normal day that a false alarm doesn't waste the drill. Error rate at the specific endpoint you're testing usually works better than an aggregate, system-wide error rate, which can stay flat even while the one path you're testing degrades badly.

Assign roles before the drill starts: someone injecting the failure, someone watching dashboards, and someone with the explicit authority to call it off immediately if the blast radius creeps wider than planned. The kill switch, whatever mechanism stops the injected failure instantly, should be tested on its own before the drill.

A kill switch you've never actually exercised is not something to trust mid-incident, and finding out it doesn't work cleanly is a much cheaper lesson to learn during a planned drill than during the drill itself going wrong.

A contained drill runs in this order:

1. State a specific, testable hypothesis about how the system should behave when one component fails.
2. Choose the smallest blast radius that still tests that hypothesis honestly.
3. Confirm the steady-state metrics, such as error rate and latency, are stable before you inject anything.
4. Assign three roles: one person injecting the failure, one watching dashboards and one with authority to call it off.
5. Inject the failure, watch for the radius widening, and roll back immediately if it does.

If you're targeting 99.99 percent uptime, you're defending about 52.6 minutes of downtime for the entire year¹, which is exactly why a drill that surfaces a real gap and then doesn't get followed up on is worse than not running the drill at all: the gap is now documented, known, and still there.

Every drill should end with a short write-up and, for anything the hypothesis disproved, a ticket with an owner and a rough timeline, not just a retelling of what happened at the next team meeting.

Not every system needs a recurring drill on the same fixed cadence. A service you're actively rewriting is a poor candidate this month, since the failure mode you'd test is about to change anyway, while a service nobody has touched in two years can be more valuable to test precisely because nobody remembers exactly how it fails.

Revisit the list of hypotheses each quarter rather than running the same rotation indefinitely. A hypothesis the team has confirmed twice in a row is a candidate to retire from the regular schedule and replace with one you haven't tested yet, since repeating an already-confirmed result teaches the team less each time it runs.