Why Your API Gateway Load Test Doesn't Match Production

A gateway advertising a huge requests per second figure is almost always measuring a bare route with no authentication check, no rate limiting, no request transformation, and no logging, because every one of those adds latency and every one of those is something you'll actually have turned on in production.

Treat the marketing number as an upper bound under ideal conditions, not a prediction of your own performance. The gap between that number and what you'll actually see is usually driven entirely by which middleware you enable, which is exactly the part a synthetic benchmark leaves out.

Start from your real traffic shape: the mix of route types, payload sizes, and middleware your production system actually runs, not a single, simplest possible endpoint. If eighty percent of your traffic hits routes with authentication and rate limiting enabled, benchmark that configuration, not a bare passthrough route that represents almost none of your real requests.

Replay a sample of real production traffic against the gateway if you can, rather than synthetic load, since synthetic traffic generators tend to produce a far more uniform request pattern than real users ever do, and that uniformity can hide latency spikes that only show up under a realistic, uneven load.

Throughput alone hides the number that actually matters to your users: tail latency, the ninety-ninth percentile response time, not the average. A gateway with excellent average latency and a bad tail can still produce a steady trickle of slow requests that show up as real user complaints, even while every dashboard built around averages looks fine.

Measure latency under sustained load over a period long enough to reveal degradation, not a short burst test. Some gateways perform well for the first minute and then degrade as connection pools or internal caches fill up, a pattern a thirty second test will never catch.

The meaningful differences between gateways usually aren't in raw throughput, most modern gateways can handle far more traffic than a small or mid sized company will ever send through one. The meaningful differences are in operational fit: how easy the configuration is to reason about and version control, how the gateway behaves under partial failure of an upstream service, and how observable its own internal behavior is when something goes wrong.

Run your benchmark with an upstream service deliberately failing or slow, not just healthy, since gateway behavior under a struggling backend, does it retry sensibly, does it fail fast, does it isolate the failure, matters more day to day than its peak throughput on a good day.

For example, suppose two gateways both clear your projected peak traffic on a healthy backend. Run the same replay again with one upstream service slowed down and watch what each gateway does: does it retry sensibly, fail fast, or let queued requests pile up and drag unrelated routes down with it? The gateway that isolates the struggling backend is usually the better daily choice, even if its peak throughput is a little lower. Also note how easy each configuration was to read and version control during the test, since that is the part your team will live with long after the benchmark chart is forgotten.

A benchmark is only useful if it changes a decision. Before running one, write down the specific question it needs to answer: can this gateway handle our projected peak traffic with our full middleware stack enabled and stay under our latency target, not a general "which gateway is fastest."

Re-run the benchmark against any gateway you're seriously considering with the same test harness and the same traffic replay, so the comparison is actually apples to apples. A benchmark run once against one candidate, compared against a vendor's marketing number for another, isn't a comparison at all.

Check a benchmark against this list before you trust its result:

• It runs your real middleware stack, including authentication, rate limiting, request transformation, and logging, instead of a bare passthrough route.
• It matches your production mix of route types and payload sizes, ideally by replaying a sample of real traffic.
• It reports tail latency at the ninety-ninth percentile, not just averages or raw throughput.
• It holds sustained load long enough to expose connection pool or cache degradation, and repeats the run with a slow or failing upstream.
• It uses the same harness and traffic replay for every candidate, so the comparison is apples to apples.