Building Synthetic Probes That Catch an Outage Before Customers Do

A health check that just confirms a process is running and responding tests the shallowest possible layer of your system. It misses a degraded database connection pool, a downstream dependency that's timing out, or a message queue that's backing up silently while the API layer itself looks perfectly healthy. None of those show up in a check that only verifies the process is alive.

The gap matters most for real time pipelines specifically, where a health check on the ingestion endpoint can stay green for a long stretch while events pile up unprocessed behind it, and nobody notices until a downstream consumer starts complaining about stale data.

A synthetic probe should walk the same path a real user or a real event takes: submit test data through the actual ingestion endpoint, confirm it's processed and shows up on the other side within an expected window, then clean up after itself. This catches the failures that matter, a stuck queue, a broken transformation step, a downstream write that's silently failing, not just whether a server is accepting connections.

Run probes from outside your own infrastructure when possible, not just from inside the same network the service runs on. A DNS or network issue that only affects external traffic will pass every internal check while real customers can't reach you at all.

A single failed probe run is often noise, a transient network blip or a slow but ultimately successful request. Alert on a small number of consecutive failures instead of a single one, and set the threshold based on your actual latency distribution rather than an arbitrary round number. If your real time pipeline's normal processing time varies between two and eight seconds, a threshold set at three seconds will page someone constantly for behavior that's completely normal.

Review alert history monthly and retire or retune any probe that's fired more than a couple of times without a corresponding real incident. A noisy probe trains engineers to ignore pages, which defeats the entire purpose of having one.

Probes that write test data and never clean it up eventually pollute production tables or dashboards, which is its own kind of incident. Probes running on a schedule too infrequent to catch a short outage, once every fifteen minutes, can miss brief but customer visible failures entirely.

Probes that don't cover the paths customers actually use most, testing an endpoint nobody calls while the one everybody relies on has no coverage at all, give a false sense of security. Periodically audit your probe coverage against your actual traffic patterns, not just against what was easiest to instrument when the probes were first built.

Watch for these common probe mistakes:

• Probes that write test data and never clean it up, which slowly pollutes production tables and dashboards.
• Running probes too rarely, such as once every fifteen minutes, so a short but customer visible outage passes between runs.
• Covering endpoints few people use while skipping the paths customers rely on most.
• Alerting on a single failed run instead of a few consecutive failures, which turns ordinary network noise into false pages.
• Leaving a probe without an owner, so nobody notices when the transaction it tests changes shape.

A probe with no clear owner tends to rot: it keeps running, but nobody notices when it stops testing what it was originally built to test, because the underlying transaction changed shape and the probe wasn't updated alongside it. Assign each critical probe to the team that owns the transaction it tests, not to a central monitoring team that doesn't have context on what changed in that code.

Treat probe code with the same review standard as production code, since a bug in the probe itself, one that makes it always pass regardless of the real system's state, is worse than having no probe at all. A green dashboard that's actually broken is more dangerous than an honest gap in coverage, because it actively tells engineers not to worry.