The Retry Logic That Makes Outages Worse, Not Better

Picture a downstream service that briefly returns errors due to a deploy or a momentary resource spike. Every caller's naive retry logic fires at roughly the same time, often within milliseconds of each other since they were all triggered by the same failure. That synchronized wave of retries hits the already-struggling service exactly when it's least able to handle extra load, which can turn a two-second blip into a multi-minute outage as the retries themselves become the dominant source of traffic.

This is exactly the failure mode that makes an incident hard to diagnose in the moment. The on-call engineer sees a downstream dependency's error rate climbing and reasonably assumes the dependency itself is the sole problem, when a meaningful share of that load is actually their own system's retries piling on top of an already-recovering service.

Instead of every caller retrying at a fixed interval, add randomized jitter to the retry delay so requests spread out over a window rather than arriving in a synchronized burst. Combined with exponential backoff, doubling the wait time on each successive attempt, this gives a struggling downstream service breathing room to recover instead of facing a wall of retries every time a fixed interval elapses.

Unlimited retries turn a downstream outage into an upstream one: a caller stuck retrying forever holds a connection, a thread, or a queue slot the whole time, and enough stuck callers exhaust your own service's capacity even though the actual problem is somewhere else entirely. Set a hard cap on retry attempts and a clear, fast failure past that cap, so a struggling dependency degrades your service gracefully instead of taking it down entirely alongside it.

A circuit breaker tracks failure rate to a specific dependency and, once it crosses a threshold, stops sending new requests to that dependency for a cooldown period, failing fast instead. This protects both sides: your own service stops wasting resources on calls likely to fail, and the struggling downstream dependency gets a reprieve from load while it recovers, rather than facing a continued stream of retries throughout its entire outage.

• Add jitter to retry delays so requests don't arrive in a synchronized wave
• Use exponential backoff, not a fixed interval, between successive attempts
• Cap total retry attempts and fail fast, clearly, once the cap is hit
• Add a circuit breaker for dependencies where repeated failures are common enough to warrant one

Review these four items together whenever a new external dependency is added, not just after an incident forces the conversation. It's far cheaper to design retry behavior correctly the first time a call site is written than to retrofit jitter, backoff and a circuit breaker into a service after a retry storm has already caused real customer-facing downtime.

When a retry ultimately fails, what does the calling service actually do? A generic error passed straight to the end user is honest but often not the best available option. For some flows, a cached or default value is a better fallback than a hard failure, showing a slightly stale price instead of no price at all, for example. Decide this per call site rather than defaulting to the same generic error handling everywhere; the right fallback for a checkout flow is rarely the right fallback for a non-critical recommendation widget.