Getting Real Value Out of OpenTelemetry Instead of Just Installing It

OpenTelemetry's auto-instrumentation libraries make it genuinely easy to get spans flowing from most common frameworks and libraries without writing custom code. That's real progress over the manual instrumentation era, but it also means teams stop there, with traces that show HTTP calls and database queries but nothing about what actually happened inside your own business logic. The trace shows you made a call to another service and it took three seconds; it doesn't show you why, unless you've added spans around the specific logic that matters.

The traces that actually help during an incident have custom spans around the parts of your code where a decision gets made or where variable-cost work happens: a retry loop, a cache lookup that might miss, a batch operation whose size depends on the input. Auto-instrumentation gives you the network boundary for free; it can't know which internal function is your team's actual bottleneck. Pick the handful of services most often involved in past incidents and add targeted spans there first, rather than trying to instrument everything evenly.

A distributed trace only stays connected across service boundaries if trace context gets propagated on every call, HTTP headers, queue message metadata, background job payloads. It's common for teams to get this right for synchronous HTTP calls and miss it for asynchronous work: a message published to a queue that doesn't carry the trace ID forward breaks the chain right at the point where debugging usually gets hardest, an async handoff. Audit every place your architecture crosses a synchronous-to-asynchronous boundary and confirm trace context survives it.

Tracing every single request at high fidelity gets expensive fast, so most setups sample, keeping only a percentage of traces. The problem is that a flat percentage sample, say ten percent, is likely to miss the rare, slow, or erroring requests that matter most for debugging, precisely because they're rare. Tail-based sampling, which decides whether to keep a trace after seeing how it completed, keeping all the slow or erroring ones and a smaller sample of the normal ones, gets you a much more useful dataset for the same storage cost. It's more complex to set up than head-based sampling, but it's usually worth the tradeoff once tracing volume becomes a real cost line.

A trace that's only ever pulled up after the fact, during a postmortem, is missing half its value. Link your tracing tool directly from your alerting and dashboards, so when a latency alert fires, the on-call engineer is one click from a relevant trace instead of starting a manual search through a tracing UI under pressure. Teams that ship every day especially benefit from this, since a fast deploy cadence means more candidate changes to check against when something goes wrong, and a trace that's hard to find slows that check down exactly when speed matters¹.

Checks that turn installed tracing into usable tracing:

• Add custom spans around retry loops, cache lookups that might miss, and batch operations whose size depends on the input.
• Propagate trace context on every hop, including queue message metadata and background job payloads, not only synchronous HTTP headers.
• Consider tail-based sampling so rare slow or erroring requests are kept, since a flat percentage sample tends to miss exactly those.
• Link alerts and dashboards directly to a relevant trace so the on-call engineer is one click away from it during an incident.

Say a checkout request that should take two hundred milliseconds is taking eight seconds for a subset of users, and the trace shows five spans: the API gateway, an auth check, an inventory service call, a pricing service call, and the database write, with most of the time sitting inside the pricing service span and nothing more specific than that. Without a custom span inside the pricing service around its actual logic, discount lookup, tax calculation, currency conversion, you're left guessing which of those three did the damage. Add spans around each one, and the next occurrence points straight at the actual bottleneck instead of just the service boundary that contains it.