The OpenTelemetry Rollout Order That Keeps the Trace Bill Sane

The instinct to instrument every service simultaneously produces a flood of trace data before anyone has built the dashboards or alerts to use it, and the rollout stalls under its own volume. Start with the services on your single most important request path, the one most likely to be the target of the next incident review, get end-to-end traces flowing and useful there, then expand service by service. A working trace across three services beats a broken, half-instrumented trace across thirty.

Head-based sampling, deciding whether to keep a trace at the very first service, is simple but throws away traces before you know if they were interesting, meaning a rare, expensive slow request has the same chance of being dropped as a fast, boring one. Tail-based sampling, deciding after the full trace completes, lets you deliberately keep every error trace and every genuinely slow trace while sampling normal traffic down aggressively, which is almost always the right tradeoff, but it requires buffering full traces somewhere before the sampling decision, which has its own memory and infrastructure cost to plan for.

A trace breaks, silently becoming several disconnected traces instead of one, whenever context propagation fails: a message queue that doesn't carry trace headers, an async job that starts a new context instead of inheriting the parent's, or a language boundary where two different OpenTelemetry SDKs handle propagation slightly differently. Test context propagation explicitly across every async boundary and message queue in your system rather than assuming it works because it works for synchronous HTTP calls, which is usually the only path anyone actually tests.

Cardinality is the usual culprit: attaching a high-cardinality attribute, a full user ID or a raw request body, to every span multiplies your storage cost in a way that's hard to predict from a dashboard until the invoice arrives. Set clear guidelines for what belongs as a span attribute versus what belongs in a linked log, and review cardinality on your highest-volume services specifically, since that's where an unbounded attribute does the most damage.

Picture a team that adds the raw request body as a span attribute on their busiest endpoint to help debug a tricky class of bug, ships it, and moves on. Weeks later the trace storage bill has grown well past what traffic growth alone would explain, and nobody connects it to that one debugging attribute because it looked harmless in a local test with a handful of requests. At production volume, a single high-cardinality attribute on a high-traffic span multiplies stored data far faster than intuition suggests, which is exactly why cardinality review belongs in the pull request process, not in a retrospective after the invoice arrives.

Tracing infrastructure with no dashboards, alerts, or runbook links tied to it becomes a tool three people know how to use and nobody else touches during an incident. Build a small number of high-value views, the request-path latency breakdown for your top three services, an error-rate-by-service view, before expanding to more, and link directly to a trace search filtered to the relevant service from your existing alerting tool so an on-call engineer doesn't need to learn a new interface mid-incident. Whichever observability backend you land on for storing and querying traces, that link-from-alert step is the one teams skip most often.

Roll out tracing in this order:

1. Pick the single most important request path and instrument only the services on it, until end-to-end traces work there.
2. Build a small set of dashboards, such as a request-path latency breakdown and an error-rate-by-service view, before adding more services.
3. Test context propagation across every queue, async job, and language boundary so traces don't split into disconnected fragments.
4. Move from head-based to tail-based sampling so every error trace and unusually slow trace is kept while routine traffic is sampled down.
5. Set span attribute guidelines that keep high-cardinality values, like full user IDs and raw request bodies, out of spans, and review your busiest services regularly.