Load Testing an Agent System Before It Meets Real Traffic

Simulate full, realistic conversations, not single requests: a mix of your most common task types, with their typical number of turns and tool calls, run concurrently at a target volume. A load test that only hits the first model call in a conversation misses the tool calls and the later turns, which is usually where the real bottlenecks live.

Test well past your current peak, not just up to it, since the point of the exercise is finding out where things break, not confirming that today's traffic is fine. A reasonable target is two to three times your highest recorded peak, adjusted based on how fast usage has actually been growing.

A useful decision rule is to set the target load before the test and tie it to something you can defend. For example, if usage has been growing quickly, a test at only the current peak proves little, so keep pushing and record the point where fallback rate or tool timeouts first climb. Write that concurrency level down as your working ceiling, then compare it against your launch forecast. A common mistake is stopping the ramp as soon as the system looks healthy, which confirms today's traffic but says nothing about future headroom. Keep raising load until something degrades, note which component gave way first, and fix that component before running the test again.

A normal service under stress typically slows down or starts returning errors in a predictable, visible way. An agent under stress can degrade more subtly: a downstream tool starts timing out intermittently, the agent falls back more often, or its reasoning quality drops because it's working with partial tool results, none of which necessarily shows up as a clean error rate spike. Watch fallback rate and tool timeout rate specifically during a stress test, not just response time and error count.

Track these signals during an agent stress test:

• Fallback rate, because a climbing share of fallbacks is often the first sign an agent is degrading while error rates still look healthy.
• Tool timeout rate for each downstream dependency, since intermittent timeouts under concurrency are a common source of quiet degradation.
• Response time and error count, as in any load test, read alongside the two signals above rather than on their own.
• Any drop in answer quality when tools return partial results, which a clean error rate will not reveal.

A downstream dependency that shows any sign of degrading under the test load, not just outright failing, is worth fixing before launch, since real production traffic tends to reveal the same weak point eventually, usually at a worse time. Treat "it got a little slow but didn't error" as a finding worth acting on, not a pass.

Write down a clear go or no-go threshold before running the test, rather than deciding in the moment how concerning a given result looks. It's easier to hold a launch to a standard you agreed to in advance than to argue, under schedule pressure, that a borderline result is actually fine.

Say a team runs a stress test ahead of a public launch, simulating three times their current internal usage. Response times hold up well through most of the test, then, past a specific concurrency threshold, the agent's fallback rate climbs sharply even though nothing has technically failed and error rates stay low. Digging into the traces from that window shows a specific tool, one that enriches a customer record with data from a third-party API, timing out intermittently once enough concurrent calls are in flight, and the agent's fallback behavior for that tool, while functioning as designed, was firing far more often than anyone expected.

Nothing about this would have shown up as an outage in standard monitoring; it would have looked like the agent just getting quietly less helpful during peak hours. Catching it in a stress test, rather than in the first week of a real launch, gave the team time to add a cache in front of that specific tool before it mattered to a real customer, and to fix it on a normal engineering timeline instead of during a launch-week fire drill. The team also added fallback rate specifically to their standard stress-test checklist afterward, since response time and error count alone would have called this test a clean pass. That single addition to the checklist has caught the same class of subtle degradation twice since, on two entirely unrelated tools.