How to Benchmark an API Gateway Without Fooling Yourself

A generic benchmark hammers a single simple endpoint with identical requests. Real traffic is a mix: some requests hit cache, some don't, payload sizes vary, and some routes do far more work than others. Build your test traffic from a sample of real request logs, not a synthetic script that only exercises the easiest path.

This single change, testing with realistic request shape instead of a generic pattern, is usually the difference between a benchmark that predicts real behavior and one that just produces an impressive-looking number.

An average latency number can look fine while a meaningful share of requests are genuinely slow. Look at your higher percentiles specifically, since those are the requests that actually generate complaints and support tickets, even when the average across everything looks healthy.

A gateway that looks great on average latency but has a long tail of slow outliers is a worse choice in practice than one with a slightly higher average and a much tighter, more predictable tail.

Total requests per second matters less than how many of those requests are happening at the same moment your system has to handle other things too, deploys, background jobs, database maintenance. Run at least one benchmark pass concurrent with a simulated deploy or a heavy background job, since that's closer to a real bad day than a clean, isolated test window.

How often you actually face this concurrency depends partly on your deployment frequency, since a higher deployment frequency means more windows where gateway load and deploy load genuinely overlap in production¹.

A self-hosted gateway can often win a raw latency benchmark while losing badly on the operational cost of running it yourself, patching it, scaling it, and being the one paged when it has a bad night. A managed gateway service can lose the latency benchmark by a small margin while saving significant engineering time better spent elsewhere.

Decide up front which tradeoff matters more for your team's current size and priorities, and weigh the benchmark result against that decision rather than treating raw latency as the only number that counts.

A benchmark run once at launch goes stale as your traffic pattern shifts, new endpoints get added, and request volume grows. Rerun it whenever traffic composition changes meaningfully, not on a fixed calendar schedule disconnected from what's actually happening to your system.

Six months later, nobody remembers exactly what traffic shape or concurrency scenario a benchmark used, which makes the old numbers useless for deciding whether a regression is real or just a different test setup. Record the methodology alongside the results: what sample of requests, what concurrency, what background load was running at the same time.

That record is what turns a one-off benchmark into something you can actually compare against next time, instead of starting the whole design process over from scratch every time a gateway decision comes up again.

A benchmark you can trust follows these steps:

1. Build test traffic from a sample of real request logs so it reflects cache hits, varied payloads, and uneven route costs.
2. Report higher percentiles alongside the average, since slow outliers generate the complaints and support tickets.
3. Run at least one pass concurrently with a simulated deploy or heavy background job.
4. Weigh self-hosted against managed options on operational cost as well as raw latency.
5. Record the methodology with the results and rerun the benchmark when traffic composition changes.

A vendor's published benchmark numbers are usually run against traffic and hardware chosen to make that vendor look good, and there's nothing wrong with that, it's marketing, not deception. The mistake is treating those numbers as a substitute for testing against your own request shape and your own concurrency pattern.

Use a vendor's own numbers as a rough first filter to narrow your options, then always run your own benchmark before committing to a migration, since the gap between a marketing number and your real workload's number can be substantial.

For example, a vendor publishes a strong throughput number measured on a single simple route with identical requests. Your traffic mixes cached reads, large uploads, and a few expensive routes. Use the published figure only to shortlist candidates, then replay a sample of your own request logs against each one. You may find that the vendor with the lower headline number has tighter tail latency on your mix, which is the number your users feel. Keep both results in the write-up so the next comparison starts from a known baseline.