Rolling Out OpenTelemetry Without Drowning in Spans

Instrument service boundaries first, every inbound and outbound call between services, before instrumenting inside a single service's internal logic. This gets you the request-level trace, which service called which and how long each hop took, which is the view that actually answers why a request was slow for most incidents.

Deep internal instrumentation inside a single service is worth adding later, specifically for the services that show up as the bottleneck in that request-level view, not everywhere up front.

Tracing every single request at high traffic volume gets expensive fast, both in storage and in the overhead added to every request. Head-based sampling, deciding whether to trace a request before it starts, is simple but risks dropping exactly the slow or erroring requests you most want visibility into.

Tail-based sampling, deciding after the request completes based on whether it was slow or errored, keeps the traces that actually matter at the cost of buffering every request's spans until a sampling decision can be made. Most production setups land on tail-based sampling for anything above a modest traffic volume, specifically because it biases toward keeping the traces you'd actually want to look at.

A span named after a dynamic value, like an individual user ID or order ID baked into the span name itself, makes every trace look unique to your tracing backend and destroys any ability to aggregate or search across similar requests.

Name spans after the operation, not the instance, a create-order call, not a specific order's path. Put the dynamic values in span attributes instead, where they're still searchable but don't fragment your span cardinality into an unnavigable mess.

• Instrumenting everything at once instead of starting with the two or three services in your slowest, most-reported-on request path.
• No context propagation across an async boundary, like a message queue, which silently breaks the trace into two disconnected pieces at exactly the point where debugging usually gets hardest.
• Treating trace data as a debugging tool only, instead of also using span duration data to catch a slow creep in latency before it becomes an incident.
• Skipping sampling strategy entirely until a storage bill or performance overhead forces the conversation, instead of deciding on one before rollout.

Say a checkout request takes three seconds end to end, and without tracing, the working theory rotates between the payment provider, the database, and the recommendations service, none of it based on real evidence. A request-level trace across the checkout path shows the actual breakdown: 200 milliseconds in the payment call, 150 in the database write, and 2.6 seconds in a synchronous call to the recommendations service that the checkout flow never actually needed a response from.

That's the entire value proposition of tracing in one example: turning a guessing exercise across three plausible suspects into a direct look at where the time actually went, without adding logging to any of the three services individually first.

Sampling rate isn't the only lever on tracing cost. Span attribute cardinality, how many distinct values a given attribute can take, drives storage cost independently of how many traces you keep, since a high-cardinality attribute like a raw request body fragment attached to every span multiplies storage far faster than the trace volume alone would suggest. Review which attributes are actually queried during an investigation versus which were added speculatively, and drop the ones nobody's used in the last few months of incidents.