Running Your First Chaos Engineering Drill Without Breaking Production

Every drill should start with a specific, falsifiable claim: "if the payment provider times out, checkout falls back to the retry queue within five seconds." Not "let's see what happens if we kill this service." A hypothesis gives you a clear pass or fail, and a random failure just gives you a story.

Write the hypothesis down before you run anything. If you can't state what you expect to happen, you're not ready to run the experiment yet, you're ready to go read the code for that path first.

Your first drill should be able to fail safely: a single instance, a single non-critical service, during low-traffic hours, with a person watching and a clear stop button. Save the ambitious, wide-blast-radius experiments for after your team has run several small ones successfully.

This isn't caution for its own sake. A drill that goes wrong and causes a real customer-facing outage teaches your organization that chaos engineering is dangerous, and it will be much harder to get buy-in for the next one.

Your allowed downtime for the year is a real budget, and a drill that goes wrong spends from that same budget just like a real outage does¹. Schedule drills with that in mind: don't run an aggressive experiment the same week you're already tight against your quarterly downtime allowance.

Teams that deploy often have an advantage here, because a small, contained failure is closer to their normal operating conditions and easier to recover from quickly than it is for a team that ships rarely and hasn't exercised its rollback path recently.

It's easy to focus entirely on the metric your hypothesis is about and miss that the experiment caused a different, unrelated problem elsewhere. Have someone watching overall system health, not just the specific path under test, and agree on the stop condition before you start, not while you're already mid-experiment and deciding under pressure.

For example, a team tests whether killing one worker instance triggers a clean restart. The hypothesis holds, but a second observer notices that latency for an unrelated job type climbed during the restart. Watching only the restart path would have shown a clean pass. Because someone was watching overall health, the team files a second finding about a shared connection pool. Give that finding an owner like any other, and add the pool behavior to the next drill's hypothesis. A pass on the tested path doesn't prove the experiment was harmless elsewhere.

A drill that reveals a gap and doesn't lead to a fix within a reasonable window teaches your team that the gap is acceptable. Track findings the same way you'd track a bug from a real incident, with an owner and a deadline, or the same failure mode will show up again the next time it's not a drill.

The biggest obstacle to a first drill usually isn't technical, it's convincing a skeptical team or a nervous founder that deliberately triggering a failure is safe. Bring evidence, not enthusiasm: walk through the small blast radius, the stop plan, and the specific hypothesis before asking for the go-ahead.

Running the very first drill on a low-stakes internal tool, something with no customer impact even in the worst case, is an easy way to build that trust before asking to test anything closer to revenue-critical systems.

A drill that produces a finding but no written record is a finding your team will rediscover the hard way in six months, during a real outage that feels oddly familiar. Write up what you tested, what you expected, what actually happened, and what changed as a result, the same format you'd use for a real incident review.

Over time, this record becomes genuinely useful: a growing list of verified assumptions about how your system behaves under specific failures, which is a far more concrete asset than a vague sense that the team has tried a few things.

A safe first drill follows this order:

1. Write a specific, falsifiable hypothesis before running anything, such as how a fallback path should behave when a dependency times out.
2. Pick a single instance or non-critical service during low traffic, with a person watching and a stop button ready.
3. Agree on the stop condition in advance and assign someone to watch overall system health, not only the path under test.
4. Check your remaining downtime budget so an experiment that goes wrong doesn't overspend it.
5. Record what you tested, expected, observed, and changed, then give each finding an owner and a deadline.