Chaos Drills That Actually Test Your Inference Fallback

Terminate a node while it is actively serving traffic and watch what happens to the requests it was handling. The question is not whether some requests fail, some probably will. It is whether your system retries them cleanly against a healthy node, whether the caller gets a clear error instead of hanging indefinitely, and whether losing one node causes a cascading slowdown on the remaining nodes because they suddenly absorb more traffic than they were sized for.

A full outage is the easy case to test because your system either has a fallback or it does not. A provider that responds slowly but does not error is harder, and more common in practice: it can tie up connections, blow through timeouts inconsistently, and degrade your own latency without ever tripping a health check. Simulate this by adding artificial delay to a test endpoint rather than only testing a hard failure, and confirm your timeout and retry logic actually protects the rest of your system from one slow dependency.

A fallback model that has never served real production traffic is a hope, not a plan. Route a meaningful slice of real traffic to it during a drill and check three things: does it handle your actual prompt formats and lengths, is its latency acceptable for the traffic you would be sending it during a real failover, and does whatever calls it downstream handle its slightly different output correctly. Finding a formatting mismatch during a calm drill is a minor fix. Finding it during an actual provider outage is a customer-facing incident on top of the outage you were already having.

If you run inference in multiple regions for redundancy, test an entire region going dark, not just one node. This is the drill most likely to surface a hidden single point of failure: a shared configuration service, a shared secret store, or a DNS setup that assumed the primary region would always be available. These dependencies are easy to miss until the exact failure that exposes them actually happens.

Every drill should end with a written list of what broke, not just a sense of how it went. Assign an owner and a rough timeline to each finding, and schedule the next drill before this one is finished, so fixing what you found does not quietly become the last chaos drill anyone runs. A drill that never gets repeated only tells you about the system as it existed on that one day.

After every drill, work through these steps:

1. Write down what broke during the drill, not only how the exercise felt overall.
2. Assign each finding an owner and a rough timeline for the fix.
3. Schedule the next drill before this one wraps up, so fixing the findings does not quietly end the practice.
4. Raise the difficulty next time by combining two failures, running during busier traffic, or removing a safety net the team relies on.

A drill that always succeeds is a sign the drill has gotten too easy, not a sign the system is finished being tested. Increase the difficulty over time: combine two failures instead of one, run the drill during a busier traffic window instead of a quiet one, or remove a safety net the team has come to rely on during previous drills. The goal is not a passing grade. It is finding the next thing that breaks before a real failure finds it for you.

Include whoever would actually be paged during a real incident, not only the engineers who built the fallback path. A drill is also a rehearsal for the humans, testing whether the on-call rotation knows where the runbook lives, whether the escalation path is current, and whether a decision-maker is reachable if the drill needs a judgment call partway through. Finding that the runbook is stale during a calm drill costs almost nothing. Finding it during a real outage costs the minutes you needed most.