What to Instrument First in a Distributed System

Before adding another dashboard, make sure a single request can be traced across every service it touches. That means a request ID generated at the edge and propagated through every downstream call, logged consistently so you can pull every line related to one failing request in one query instead of grepping five services by hand.

Teams with plenty of logs but no correlation ID still end up guessing which service log to check first during an incident, which is the exact problem observability is supposed to solve.

Propagating the ID is the hard part in practice, not generating it. Every internal client, queue message, and background job needs to carry it forward, which usually means adding it to a shared library once rather than hoping every team remembers on their own.

Say CPU on one box runs hot: that isn't an incident if nothing downstream is affected. An error rate climbing on a customer-facing endpoint is. Page on symptoms tied to user impact, an SLO burning down faster than expected, elevated error rates, requests timing out, and route internal resource signals to dashboards instead of pages.

Tie your alert thresholds to the availability target you've actually committed to. A service promising 99.9 percent has roughly 8.76 hours of downtime to spend across the whole year¹. An alert that doesn't fire until you've already burned a meaningful chunk of that budget in one incident is set too late.

Say customers occasionally report a failed checkout, but it never reproduces when an engineer tries it manually. Without correlation, this stays a mystery. With a request ID threaded through logs, the pattern becomes visible: the failures cluster around moments when a downstream inventory service's connection pool is exhausted, which only happens under concurrent load an engineer testing alone can't recreate.

The fix here isn't better error messages, it's a metric on connection pool saturation that would have shown the exhaustion happening well before customers noticed anything.

• A dashboard built for one incident and never updated as the system changed
• Metrics with no owner, so nobody notices when they stop reporting data at all
• Alerts tuned so loose they never fire, or so tight the on-call engineer mutes them out of habit
• Logs with no consistent structure, so search means guessing at field names service by service

A dashboard that isn't tied to an action someone takes when it changes isn't observability, it's wallpaper.

A log line that reads well in a terminal is usually hard to query at scale. Structured logging, consistent field names for request ID, service name, status, and duration across every service, turns your logs into something you can actually filter and aggregate instead of just read.

This matters more as the number of services grows. Two services with inconsistent log formats is an annoyance. Twenty services with inconsistent formats means every incident starts with reverse-engineering that day's logging conventions before you can even start debugging.

An alert that pages someone with no next step attached turns every incident into a puzzle the on-call engineer has to solve from scratch, usually at the worst possible hour. Attach a short runbook to every alert: what this usually means, where to look first, and who to escalate to if it's outside your area.

The runbook doesn't need to be exhaustive. A few lines pointing at the right dashboard and the service's known failure modes turns a 2am page from a guessing game into a checklist, which matters most for whoever is newest on the rotation.