How to Actually Compare API Gateways on Latency

A gateway's added latency at the median is only part of the picture; the tail, the 95th or 99th percentile, matters more for user experience, and it often comes from a different set of features than the median does: request transformation, authentication checks, or rate limiting logic. Compare gateways on the specific features your traffic will actually exercise, not just a bare pass-through benchmark that neither option will resemble once it's actually handling your production requests.

A gateway benchmarked against a steady stream of identical, small requests behaves differently under your real mix of request sizes, payload types, and burstiness. Capture a representative sample of your actual traffic, or as close an approximation as you can build, and replay it against each candidate gateway rather than trusting a vendor's published number, which was almost certainly measured under conditions chosen to look favorable.

A gateway's behavior when a backend is slow or unavailable, whether it fails fast, queues requests, or degrades gracefully, matters as much as its best-case latency number, because backends do fail, and the gateway's behavior during that moment is often what a user actually experiences as an outage. Include a deliberate backend-failure scenario in your comparison, not just a happy-path latency test, since this is often where the real difference between two otherwise similar candidates actually shows up.

Include these scenarios in every gateway comparison:

• A replay of a representative sample of your real traffic, including its mix of request sizes, payload types and bursts.
• The specific features your traffic exercises, such as request transformation, authentication checks and rate limiting.
• Tail latency at the 95th and 99th percentile, not just the median.
• A deliberate backend failure or slow backend, to see whether the gateway fails fast, queues requests or degrades gracefully.

The gateway with the best latency number isn't automatically the right choice if it requires meaningfully more operational effort to run, upgrade, and debug than an alternative with slightly worse but still acceptable numbers. Factor in your team's existing familiarity with a given gateway's ecosystem and its debugging tools; a small latency advantage rarely outweighs a large gap in how quickly your team can diagnose a problem with it at 2 a.m., when familiarity matters far more than any single number on a benchmark chart.

To weigh operations against speed, score each candidate on a few practical questions your team can answer from experience. How quickly could someone diagnose a failure with it late at night? Does your team already know its configuration and tooling? How painful are upgrades? For example, if one option is slightly faster but nobody on the team has run it, ask whether the difference will ever matter to users more than a slower diagnosis will. A common mistake is treating the fastest benchmark as the answer. Record the tradeoff explicitly so the choice can be defended later and revisited when the traffic mix changes.

A comparison done when your traffic looked one way can go stale once your traffic mix changes meaningfully, a new client type, a new region, a different payload size distribution. Treat the benchmark as something to redo when the traffic shape changes significantly, not as a one-time decision that holds indefinitely regardless of how the underlying workload evolves.

The first time you build a real traffic-shape benchmark takes real effort: capturing a representative sample, building the replay harness, defining the failure scenario. Document exactly how you did it, so the next evaluation, whether it's a routine re-check or a genuinely new candidate gateway, starts from a working methodology instead of being rebuilt from memory by whoever happens to own the decision next time.

Before running the comparison, write down what result would justify switching gateways versus what result would confirm your current choice is fine. Without that upfront line, it's easy to run a benchmark, get an ambiguous result, and default to whichever option required less migration effort regardless of what the numbers actually showed, which defeats the purpose of running a rigorous comparison in the first place and wastes the real engineering time the benchmark itself cost to build.