A 30-Minute Check for Blind Spots in Your Observability Setup

Open your alerting tool and filter to the last 90 days. Sort by which alerts have fired the most, and separately, which have never fired at all. An alert that's never fired is either genuinely tracking something that's never gone wrong, or it's misconfigured and silently useless; you often can't tell which without checking the underlying query directly.

Cross-reference against your incident history: for each incident in the last quarter, was there an alert that should have caught it earlier? If the answer is no for more than one or two incidents, that's your biggest gap, and it's worth fixing before anything else on this list.

Pick a critical user-facing flow, like checkout or login, and trace a single request through every service it touches. At each hop, ask: is there a metric for latency and error rate here, and would someone get paged if this hop started failing. It's common to find that the entry point and the database are well instrumented but a middle service, often one added later or owned by a different team, has no visibility at all.

Pay particular attention to anything asynchronous: a queue consumer, a background job, a webhook handler. These tend to get instrumented last because they don't sit directly in the request-response path a developer is staring at while building the feature, but a stuck queue consumer can silently back up for hours before anyone notices, precisely because nobody's watching it the way they'd watch a slow API response.

For example, imagine a checkout flow that passes through an API, a payment service, and a queue consumer that sends receipts. During the trace you find latency and error metrics on the first two but nothing on the consumer. The fix is small: add a queue-depth metric and an alert on messages waiting longer than a set age, then confirm the on-call rotation actually receives it. A useful decision rule: if a hop can fail silently for an hour without anyone being paged, instrument it first, ahead of polishing dashboards that already work.

A dashboard with no clear owner tends to drift out of date the moment the system it monitors changes shape, a service gets split in two, a queue gets renamed, and nobody updates the panel. Skim your top five dashboards and confirm each one has a name attached, not just a team. If a panel shows a metric that no longer exists or a service that's been decommissioned, that's a signal the whole dashboard hasn't been reviewed in a while.

The same check applies to who actually looks at each dashboard, not just who's nominally responsible for it. A beautifully built panel that nobody opens outside of an incident retrospective isn't really observability, it's decoration. If you can't name the last time someone checked a given dashboard outside of an incident, fold its most important metric into an alert instead of trusting a human to remember to look.

Your availability target implies a specific downtime budget: a service promising 99.9% uptime can absorb roughly 8.76 hours of downtime across a year, while 99.99% only allows around 52.6 minutes¹. Check whether your alert thresholds are actually tuned to that budget. It's common to find alerts set to fire on any error at all, which trains the on-call rotation to ignore them, when the real target should be an error-budget-burn-rate alert tied to how fast you're eating into that annual allowance.

If you can't say off the top of your head how much of this year's downtime budget you've already used, that's itself a finding worth writing down before you move on.

Keep this checklist handy for your next run:

1. Minutes 0 to 10: filter 90 days of alert history, compare alerts that fire most with those that never fire, and match recent incidents to alerts that should have caught them.
2. Minutes 10 to 20: trace one critical flow, such as checkout or login, through every service and confirm each hop has latency and error metrics plus a page.
3. Minutes 20 to 25: confirm each top dashboard has a named owner, and that someone actually looks at it outside of incidents.
4. Minutes 25 to 30: compare alert thresholds to your downtime budget, and move noisy any-error alerts to error-budget burn-rate alerts.