Benchmark Your Own Gateway Before You Trust Anyone Else's Numbers

A published p50 latency figure usually comes from a minimal request, a small payload, no authentication overhead, run against infrastructure tuned specifically for that benchmark. Your production traffic carries auth checks, payload transformation, rate limiting logic, and a request size distribution the vendor's number never accounted for. Treat any vendor benchmark as a ceiling on what's possible under ideal conditions, not a prediction of what you'll actually see.

Average latency hides the requests that were genuinely slow behind the much larger number that were fine. p50 tells you about the typical request; p99 tells you about the request your slowest, most frustrated customers are actually experiencing. If you only track one number, track p99, since it's the one most likely to correlate with support tickets and churn, not the one that makes a dashboard look the best.

For example, when comparing two gateways, put the p50 and p99 for each side by side against the same replayed traffic, and note which requests sit in the slow tail. If the slowest requests are the large uploads, the tail belongs to payload handling, not routing, and that calls for a different question to the vendor than a general one about speed. Ask each vendor to explain the tail on your traffic. A vendor that can only point back to its published benchmark hasn't answered the question you are actually deciding on.

Replay a sample of actual production request logs, not a synthetic payload generator, against any gateway you're evaluating, including your current one as the baseline. Match the real distribution of payload sizes, authentication patterns, and endpoint mix, since a gateway that looks fast against small, uniform synthetic requests can behave very differently once it's handling your actual mix of large uploads, small polling requests, and everything in between.

Build a fair gateway benchmark in this order:

1. Sample real production request logs to replay, rather than using a synthetic payload generator.
2. Match the real mix of payload sizes, authentication patterns and endpoints in the replayed traffic.
3. Run the same replay against your current gateway as the baseline for comparison.
4. Turn on every feature you'll use in production together, including auth, rate limiting and transformation, instead of testing each alone.
5. Compare p99 as well as p50, since the slowest requests drive support tickets and churn.
6. Rerun the whole test quarterly at minimum, and after any meaningful change.

Each individual feature, auth token validation, rate limiting, request transformation, response caching, adds a small amount of latency on its own, but they compound, and the combined overhead is rarely the simple sum of each feature's advertised cost in isolation. Benchmark with every feature you'll actually run in production turned on together, not one at a time, since a gateway that's fast with authentication alone and fast with rate limiting alone can still be surprisingly slow with both running at once.

A gateway sitting at the edge of your infrastructure, handling every request before it reaches anything else, is also one of your highest value patching targets: a known exploited vulnerability in gateway software carries a fourteen day remediation clock under federal guidance for CVEs assigned since 2021¹. Benchmarking performance and staying current on security patches are separate workstreams, and it's worth explicitly confirming neither one is quietly starving the other of attention.

Say a synthetic benchmark shows a candidate gateway beating your current one by a wide margin on small, uniform requests. Once you replay real production traffic, including the ten percent of requests carrying large file uploads, the gap narrows sharply or reverses, because the candidate's transformation layer handles large payloads less efficiently than your current setup. That gap only shows up once you test with your actual traffic shape, which is exactly why the synthetic number alone would have led to the wrong decision.

A benchmark run once during initial evaluation goes stale the moment traffic patterns shift, a new endpoint gets added, or the gateway's configuration changes to support a new feature. Treat the replay benchmark as a recurring check, quarterly at minimum, rather than a one-time decision made during procurement. The gateway that won the original comparison can quietly become the slower option eighteen months later if nobody's checked since, and the only way to know is to actually run the test again, not assume the original result still holds.