A Chaos Drill Walkthrough: From Hypothesis to Fixed Bug

"Let's kill a node and see what happens" is not a hypothesis. "If the primary database connection pool is exhausted, we expect the application tier to shed load gracefully and recover within a set time once the pool frees up" is one, because it can be wrong in a specific, discoverable way. Write the hypothesis down before the drill, including what you expect to happen and what would count as a failure of that expectation, so the outcome isn't reinterpreted after the fact to sound like it went fine.

Run the first version of any new drill in staging, or in production against a small, deliberately chosen slice of traffic with a fast rollback available, not against your full production fleet on the first attempt. The point of starting small is that your hypothesis about the blast radius is itself part of what you're testing; if the failure spreads further than expected even at small scale, you want to discover that with a rollback ready, not while you're mid-incident.

During the drill, watch not just the component you're directly testing but its neighbors: the services that depend on it, the ones it depends on, and anything sharing its infrastructure. A connection pool exhaustion test often reveals that a completely unrelated service sharing the same database instance also degrades, which is exactly the kind of hidden coupling a drill is supposed to surface. If nobody is watching beyond the target component, that discovery gets missed.

Compare the outcome against the hypothesis explicitly: did the application tier shed load gracefully, did it recover within the expected time, and if not, exactly where did the behavior diverge from what was predicted. Resist the pull to write the finding as a success because nothing caught fire; a drill where recovery took three times longer than expected is a valuable finding even if the system technically survived, and writing it up honestly is what makes the next drill worth running.

The most common way chaos engineering stops delivering value is that findings live in a debrief document nobody revisits. File the specific gap the drill uncovered as a ticket with an owner and a deadline, the same as any other bug, and schedule a follow-up drill against the same hypothesis once the fix ships, to confirm the behavior actually changed rather than assuming the fix worked because it looked right in review.

Run each drill in this order:

1. Write a falsifiable hypothesis, including what would count as a failure of your expectation.
2. Run the first version in staging, or against a small slice of production with a fast rollback ready.
3. Inject the failure and watch neighboring and dependent services, not just the component under test.
4. Record the outcome against the hypothesis, including exactly where the system behaved differently than predicted.
5. File each gap as a ticket with an owner and deadline, then rerun the drill once the fix ships.

A single chaos drill tells you about the state of the system on one day. Recurring drills, run on a set schedule against a rotating set of hypotheses, catch regressions that a one-time exercise never will, since a fix that worked when it shipped can quietly stop working after an unrelated change six months later. Treat the drill calendar as part of the system's ongoing maintenance, not a special project that wraps up once the first round is done.

A drill run silently by one engineer without telling the rest of the on-call rotation risks being mistaken for a real incident, or worse, having its findings dismissed because nobody else trusts what happened during an exercise they didn't know was happening. Announce the drill window ahead of time to whoever might reasonably be paged during it, and share the write-up afterward with the full team, not just whoever proposed the hypothesis, so the lesson actually spreads instead of staying with one person.