Running Chaos Drills Without Breaking Production for Real

Pick a failure mode your team has already recovered from before, a single instance restart, a dependency timing out, a cache going cold, rather than inventing a novel scenario for your first drill. The point of an early drill is confirming your known recovery path actually works when triggered on purpose, not discovering an entirely new failure class.

Run it in staging first, even if your eventual goal is production chaos testing. A drill that reveals a scripting mistake in staging is a non-event; the same mistake triggered directly in production is its own incident.

Decide, in writing, exactly what the experiment is allowed to touch: one service, one instance, one percentage of traffic, and nothing beyond that boundary. Have a specific person watching dashboards in real time with authority to abort immediately if the blast radius starts to spread past what was agreed.

An experiment without a defined stop condition isn't a controlled drill, it's just breaking something and hoping for the best.

Write the abort procedure down alongside the blast radius, not just the intention to abort if needed. Knowing exactly which command reverts the failure, and confirming it works before the drill starts, is what actually keeps a contained experiment contained if something doesn't go the way you expected.

Write these items down before starting a drill:

• The exact scope the experiment may touch: one service, one instance, or one slice of traffic, and nothing beyond it.
• The named person watching dashboards in real time, with authority to abort immediately.
• The specific command that reverts the failure, confirmed to work before the drill starts.
• The stop condition that tells the watcher the blast radius has spread past what was agreed.

Chaos drills exist to confirm your system degrades gracefully within the downtime budget you've committed to, not to test whether it can survive anything. A team promising 99.9% availability has roughly 0.365 days, about nine hours, of downtime to spend across an entire year, and a drill's job is proving that a single dependency failure doesn't quietly eat a large slice of that budget in one afternoon¹.

Framing the goal this way also tells you when to stop running bigger drills: once you've confirmed your known failure modes are contained within budget, further escalation should target new, specific risks, not general bravado.

Move to production only after staging drills have run clean multiple times, and only against a small, clearly bounded percentage of real traffic at first. Schedule it for a low-traffic window, with the same real-time monitoring and abort authority as your staging drills, just with higher stakes if something goes wrong.

Tell the on-call team the drill is happening before it starts. A chaos experiment that pages an unsuspecting on-call engineer at 2 a.m. teaches the wrong lesson about what chaos engineering is for.

A chaos drill run once, written up, and never repeated tells you your system handled a specific failure on a specific day. It says nothing about whether a code change three months later reintroduced the same fragility. Recurring drills, even simple ones repeated on a set cadence, catch regressions that a single celebrated success never will.

The goal isn't a growing list of dramatic experiments. It's a short list of realistic failure modes, tested regularly enough that a regression gets caught before a real incident finds it first.

Keep that list short on purpose. Five or six failure modes that map to your system's real dependencies, tested on a repeating schedule, teach you more over a year than a long backlog of exotic scenarios that only ever get run once for a demo.

For example, a team could pick three failure modes that match its real dependencies: a single instance restart, a dependency timing out, and a cold cache. It schedules each on a repeating cadence, records how long recovery took, and compares the result with the previous run. If a code change makes recovery slower, the drill catches it before a real incident does. A useful decision rule is to add a new failure mode only when a real incident, a new dependency, or an architecture change creates a risk the existing list does not cover.