Running a Chaos Drill Against Your RAG Pipeline Without Breaking Production

Start narrow: what happens if the embedding provider times out, if the vector database returns an empty result set, if the reranker returns malformed output, or if the generation model is unavailable. Each of these has a different correct behavior, and testing them together in one drill makes it impossible to tell which failure caused which symptom. Run one at a time, and only combine failures in a drill once you understand each one in isolation.

Simulate an embedding provider timeout by adding latency or a forced error at the network call, not by killing the whole service, since a real provider outage looks like slow or failing responses, not your application crashing. For the vector database, test what happens when a query returns zero results, a query the index genuinely can't answer, not just when the database is unreachable; these are different code paths and often only one of them is handled.

The question isn't only whether the pipeline survives the failure. It's what the user actually sees: a clear error, a stale cached answer clearly marked as such, or a confidently wrong answer generated from an empty or corrupted retrieval? The last one is the failure mode worth the most attention, since a RAG system that fails by generating a fluent, wrong answer from no real retrieved context is worse than one that fails visibly. Check specifically for this during every drill, not just for crashes.

A chaos drill against a staging environment with synthetic traffic tells you less than one against production with a small, bounded share of real traffic, because synthetic queries rarely match the query patterns that actually break things. Start in staging to catch the obvious failures, then graduate to a small share of production traffic with a fast kill switch, and always have someone watching live during the drill, not just reviewing results afterward.

A drill that surfaces a real problem and doesn't lead to a code change or a runbook update is a wasted drill. After each one, write down what broke, what the system should have done instead, and who owns the fix, with a deadline. Re-run the same drill after the fix ships to confirm it actually resolved the failure mode, rather than assuming it did because the code looks right.

Keep a running log of every drill you've run and its outcome, not just the most recent one. Over time this log becomes its own asset: a record of which failure modes you've actually verified the system handles correctly, and which ones you've only assumed are fine because nobody's tested them yet.

The first chaos drill against production traffic tends to cause more anxiety than the drill itself justifies, mostly because people who aren't involved don't know it's happening and assume a real incident is underway. Tell support, on-call, and anyone who might notice degraded responses ahead of time, with a clear start and end window, so a drill doesn't accidentally trigger its own separate incident response.

• One failure mode injected at a time, at the point it would realistically happen
• A clear definition of correct behavior for that failure, written down before the drill runs
• Someone watching live, with a fast way to stop the drill if it goes wrong
• A check for confidently wrong generated answers, not only for crashes or errors
• A follow-up fix with an owner and a re-run to confirm it worked