Building a Latency Budget Before You Chase Microsecond Fixes

Pick the single slowest user-facing flow you've had a complaint about: a search, a checkout step, a dashboard load. Draw every hop it takes: browser to load balancer, load balancer to API gateway, API gateway to each backend service it calls, each service to its database or cache, and back. Most teams have never drawn this diagram for their own product, which is why the first optimization attempt usually targets the wrong hop.

For each hop, write down what you'd guess its current latency is. You'll be wrong about several of them, and that's the point: the map tells you where to actually measure.

This exercise usually takes less than an hour and surfaces hops nobody remembered existed, like an internal auth check that calls out to a third service, or a feature flag lookup that hits a separate cache on every request. Those forgotten hops are disproportionately likely to be where your budget is actually being spent, precisely because nobody has looked at them since they were added.

Add tracing spans (OpenTelemetry is the common choice now) around each hop from your map, not around every function in your codebase. You're looking for the one or two hops that eat most of the budget, not a complete performance profile of everything. For example, a team running a typical three-tier web app often finds that a single unindexed database query or an uncached call to a third-party API accounts for more of the total latency than every application-code path combined.

Run the instrumented flow under realistic load, not a single request from your laptop. A query that takes 5 milliseconds when the database is idle can take 200 milliseconds when ten other requests are competing for the same connection pool.

Once you know each hop's real latency, allocate your total budget across them with headroom: for example, keep the sum of your hop budgets to about 70 percent of the total, so a slow day in one hop doesn't blow the whole page. A hop that's already over its budget is your actual bottleneck; everything else is noise until that one is fixed.

This is also where deployment cadence matters: teams that deploy small changes often can isolate which specific change moved a hop's latency, while teams that batch weeks of changes into one release have to guess. Teams in DORA's highest-performing cluster practice on-demand deployment, often shipping several times in a single day, which is partly why they catch latency regressions within hours instead of discovering them a sprint later¹.

Resist fixing more than one hop at a time. If you add a cache layer, switch a query index and upsize a database instance in the same week, you won't know which change actually mattered when the next regression shows up. Ship one change, re-run the same instrumented flow under the same load, and compare against your budget.

A common mistake here is declaring victory when the average latency improves but the tail doesn't. If your p50 drops from 300ms to 150ms but your p99 is still 2 seconds, a meaningful share of real users are still having the slow experience your budget was supposed to prevent.

A latency budget that only gets checked during an incident stops paying for itself. Add a lightweight alert on the one or two hops you identified as tightest, set, for example, at roughly 80 percent of their budget, so you get warned before a hop breaches rather than after users complain. Revisit the whole map when you add a new service to the request path, not on a fixed calendar, since that's when budgets actually drift.

Write the budget down somewhere the whole team can see it, next to the architecture diagram if you have one, rather than in a document only the engineer who ran the original exercise remembers exists. A budget nobody else knows about doesn't change how new code gets written, and new code is where the next regression will come from.

Run the whole process in this order:

1. Pick the slowest user-facing flow and draw every hop it takes, from the browser through each service and database and back.
2. Add tracing spans around each hop on that map, not around every function in the codebase.
3. Allocate the total budget across the hops with headroom, and treat any hop already over its share as the bottleneck.
4. Change one hop at a time, then re-run the same instrumented flow under the same load and compare against the budget.
5. Alert on the tightest hops before they breach, and redraw the map whenever a new service joins the request path.