Getting Started With OpenTelemetry on a Small Team

Four ideas cover most of what you need:

• Signals: traces (the path of one request across services), metrics (numbers over time) and logs (event records). Traces are the best starting point because they show where time goes.
• SDK and instrumentation: libraries in your app that create spans. Auto-instrumentation covers common frameworks and clients with little code, and you add manual spans for your own business operations.
• Context propagation: a trace ID passed between services in a header so spans join into one trace. If a service drops the header, the trace breaks.
• Collector: a process that receives telemetry, batches and filters it, and forwards it to a backend. It keeps backend credentials and routing out of your application code.

The standard protocol between them is OTLP, which is why you can switch backends without re-instrumenting.

Pick the request path customers care about most, such as sign-in or checkout, and follow these steps:

1. Choose one service on that path and enable the language's auto-instrumentation using environment variables where possible.
2. Set a clear service name and an environment attribute, since every dashboard and query depends on them.
3. Run a collector next to the service, or as a small shared deployment, and point the SDK at it.
4. Send data to a backend, even a free tier or a local viewer at first.
5. Trigger the request and confirm you see spans for the incoming call, database queries and outgoing HTTP calls.
6. Repeat for the next service on the path and verify propagation by checking that both appear in one trace.

Resist instrumenting everything at once. One complete, trustworthy trace beats ten partial ones.

Because the format is open, backend choice is reversible. Datadog and New Relic both accept OpenTelemetry data, and you can also use open-source stores you operate yourself. For a small team, the deciding factors are usually who will run it, how you want to search and alert, and what the bill looks like as volume grows. The comparison is in Datadog vs New Relic vs Dynatrace.

Whichever you pick, keep exporter settings in the collector, not scattered through application code. That way, switching or adding a second destination is a config change. Confirm current ingest pricing and limits in a demo, since plans change.

Telemetry volume tends to grow faster than expected. Manage it deliberately:

• Sample traces. Keep all errors and slow requests, and a fraction of normal ones. Tail-based sampling in the collector can make that decision after seeing the whole trace.
• Drop noisy spans such as health checks and static asset requests.
• Watch attribute cardinality. Putting a user ID or full URL with parameters into a metric label can explode series counts. Keep those in traces, not in metric dimensions.
• Set retention to what you use. Few people query traces older than a couple of weeks.
• Set a monthly volume alert with your backend so a bad deploy doesn't produce a surprise bill.

Also keep sensitive data out of telemetry. Don't record request bodies, tokens or personal data in span attributes unless you've reviewed the privacy implications with your counsel.

Turn the data into decisions. Build one dashboard per critical path with request rate, error rate and latency percentiles, and add a deploy marker so changes are visible against the graphs. Add alerts tied to customer symptoms and link them to your paging setup; tie them to service level objectives if you have them. Then extend to metrics and logs, using trace IDs in log lines to jump between them.

Keep an eye on delivery health too. DORA's 2024 report puts change failure rate at 5 percent in its highest-performing cluster and 40 percent in its lowest¹, and traces make it faster to see whether a release caused a regression. Related reading includes distributed tracing with OpenTelemetry for DevSecOps and the developer platform version.