Building a Throughput Benchmark You Can Actually Trust

Before you run a single test, write down what you're actually simulating: a steady, even stream of requests, a sudden spike from a marketing push, or a slow ramp over the course of a day. These produce very different results against the same system, and a benchmark run under one pattern tells you little about how the system handles another.

Pull this pattern from your own traffic logs where you can, rather than a generic load-testing template, so the test actually resembles a day you might really have.

The point where your system technically stops accepting more load and the point where it's still healthy, with normal latency and no dropped requests, are usually different numbers, sometimes very different ones. Record both explicitly. Reporting only the ceiling gives leadership a number that sounds reassuring but isn't the number you'd actually want to be operating anywhere near.

A reasonable rule of thumb is to treat your comfortable operating point as the real capacity for planning purposes, and the ceiling as an emergency margin you never plan to use intentionally. Sharing both numbers, instead of just the more impressive one, also makes it much easier to explain to the rest of the company why a marketing push that doubles traffic still needs a heads-up in advance.

Keep a consistent record for every run so results are comparable over time:

• The exact load pattern and duration used
• Requests per second at the comfortable operating point and at the ceiling
• Latency percentiles at each of those points, not just at the ceiling
• Which resource hit its limit first (CPU, memory, a downstream dependency, a connection pool)
• What changed in the system since the last run

Without that last item, it's hard to tell whether a change in results came from your architecture or from something else that shifted in the meantime.

A synthetic test that always sends the same request, or requests for the same narrow slice of data, tends to benefit unrealistically from caching and connection reuse in ways real, varied traffic wouldn't. It also usually skips the messy parts of real traffic: retries, malformed requests, and uneven request sizes.

Vary your synthetic requests enough to avoid these artificial advantages, and treat a benchmark result that looks unusually good as a reason to check your test's realism before you trust it.

Say a service handles normal traffic fine but degrades sharply during a specific type of spike. Running the worksheet above on a ramping load pattern that mirrors that spike, rather than a steady one, is what surfaces the real limit: often a downstream dependency or a connection pool hitting its ceiling well before the application server itself does. A steady-load benchmark alone would have missed this entirely, since it never recreates the specific pattern that actually causes the problem.

A benchmark result has a shelf life. A change to how a service caches data, a new downstream dependency, or a database migration can shift your real capacity meaningfully without anyone noticing until traffic tests it for you. Rerun the benchmark after any change significant enough that you wouldn't be surprised if capacity moved, not on a fixed calendar schedule that might miss the change entirely. Keep a short log of results over time next to your architecture decisions, so a future regression is easy to trace back to the change that actually caused it instead of turning into a fresh investigation from scratch.