Finding Your Real Latency Bottleneck Before Customers Do

Averages hide the problem. A p50 of 120 milliseconds can sit next to a p99 of four seconds on the same endpoint, and the average will look fine while a slice of your users are furious. Pick p95 and p99 per endpoint, not a single blended number across your whole API, since a slow admin report endpoint will otherwise drown out a fast, high-traffic one in the average.

Measure under realistic concurrency, not a single request at a time. A benchmark run with one client hitting an idle server will miss lock contention and connection pool exhaustion entirely, and those are usually where the real regressions live.

The same logic teams already apply to uptime works for latency. A 99.9% availability target allows a little under 8.76 hours of downtime a year¹, and a latency budget works the same way: you decide upfront how much slowness is acceptable before it counts as a regression, instead of arguing about it after a customer complains.

Write the budget per endpoint, in milliseconds, and treat crossing it the same way you would treat an uptime breach: an alert, an owner, and a fix window, not a Slack message that scrolls away.

Before blaming the database, check the boring stuff first:

• N+1 queries hiding behind an ORM that looks fine in code review
• Cold starts on serverless functions that only show up on the first request after a quiet period
• Third-party API calls made synchronously inside the request path instead of queued
• Payload serialization on large responses that never gets profiled because it works fine locally
• Connection pool limits that quietly queue requests once traffic passes a threshold nobody tested

Each of these shows up cleanly in a distributed trace and almost never shows up in a load test that only measures total request time.

Run load tests against a staging environment seeded with production-sized data and a traffic shape that matches your real usage pattern, not a flat rate of identical requests. A synthetic test that hits the same cached endpoint a thousand times a second tells you about your cache, not your application.

When a regression shows up in production, correlate the latency graph against your deploy timeline first. A comparison of observability platforms is worth reading if your current tooling cannot answer that question in under five minutes, because that five minutes is usually the difference between a quick rollback and a multi-hour investigation.

Before reaching for a bigger database instance, look at what you are sending over the wire and how often you are asking for it twice. Caching hot reads that do not change every request, trimming response payloads down to what the client actually renders, and reusing connections instead of opening new ones per request tend to close most of the gap between a slow endpoint and an acceptable one.

Only after those are done does it make sense to look at horizontal scaling or a faster data store, since scaling a query that should not have run twice just makes the waste more expensive.

For example, suppose a report endpoint shows a p99 of several seconds while its database queries finish quickly. A trace might show the time going into serializing a very large response that the client mostly ignores. Trimming the payload to the fields the page actually renders fixes that endpoint without touching the database at all. The common mistake is to skip this check and upgrade the database instance first, which costs more and leaves the slow endpoint slow. Before any infrastructure change, compare the time spent in the query, the serialization and the network transfer for one slow request. The largest piece is your first target.