Designing a Load Test That Finds Where RAG Actually Breaks

Pull a sample of real queries from your logs and replay a realistic distribution, not a single query repeated thousands of times. Include the popularity skew that real traffic has, some queries recur constantly, most are closer to unique, since that skew affects cache hit rates and index access patterns in ways a uniform test won't reveal. A load test built on unrealistic traffic gives you a confident, precise number that doesn't actually predict what will happen in production.

Include a mix of cheap and expensive queries too, short filters against a narrow collection alongside broad, high-top-k requests that trigger reranking, since a test built only from the easy end of that spectrum understates how much load the expensive end actually places on the system.

Increase load steadily rather than jumping straight to a target queries-per-second number, and watch for the point where latency starts climbing disproportionately faster than load, commonly called the knee in the curve. That point, not your test's maximum throughput number, is the more useful planning figure, since it tells you where the system starts to strain well before it actually falls over, giving you an early warning threshold rather than just a breaking point.

A freshly started system with an unwarmed index or a cold cache behaves very differently under load than one that's been serving steady traffic for hours. Since production experiences a cold state after every deploy and every scale-out event, not just at initial launch, test both explicitly and know which condition any given result describes. A load test result that only covers the warm, steady-state case tells you nothing about the minutes right after your next deploy, which is often exactly when load testing would have mattered most.

Production doesn't pause ingestion while serving query traffic, so a load test that exercises only one at a time misses the real contention between them, ingestion write load competing with query read load on the same underlying infrastructure. Run a realistic ingestion workload alongside your query load test to see how they actually interact, since a system that handles each one fine in isolation can still struggle when both are happening at once, which is the normal state in production.

When the system does start failing under load, the important question is how: responses getting slower but still correct is a far safer failure mode than requests erroring out or the service crashing outright. Design toward graceful degradation deliberately, and use your load test specifically to confirm which failure mode you actually get under real pressure, rather than only checking whether the system holds up at your target load and stopping there.

A load test that produces a number nobody acts on isn't worth much more than not running one. Translate the knee-point findings into a concrete plan: at what load do you scale up, what's the lead time to do it, and how does that compare against your availability target, since a downtime budget shrinks fast at higher availability tiers¹ and a capacity shortfall that isn't caught ahead of time eats directly into that budget.

A load test that predicts production runs in this order:

1. Replay a sample of real queries with realistic popularity skew instead of one query repeated thousands of times.
2. Ramp load steadily and note the knee, the point where latency climbs faster than load.
3. Run cold and warm states separately, since every deploy and scale-out starts cold.
4. Run ingestion concurrently with queries to expose contention on shared infrastructure.
5. Record how the system fails, since slower but correct is safer than errors or crashes.
6. Convert the knee point into a capacity trigger with a defined scale-up lead time.

A load test report full of raw percentile tables tends to get skimmed once and forgotten. Summarize the knee point, the failure mode observed, and the resulting capacity trigger in plain language at the top, with the detailed numbers available for whoever wants to dig in. A result that only the engineer who ran the test understands doesn't actually change how the team plans for growth, and a plan nobody outside that one engineer can act on isn't really a plan yet.