Designing Retry and Fallback Logic That Doesn't Make Outages Worse

An immediate retry against a service that's failing because it's overloaded adds more load at exactly the wrong moment. Exponential backoff, waiting longer between each successive retry, gives a struggling dependency room to recover instead of compounding the problem. Add jitter, a small random variation in the wait time, so a fleet of callers doesn't retry in synchronized waves that create their own load spikes.

And always cap the total number of retries. Unlimited retries on a permanently failing request just delays the inevitable failure while consuming resources the whole time, which helps nobody.

A circuit breaker tracks recent failure rates to a dependency and, once failures cross a threshold, stops sending requests entirely for a cooldown period instead of continuing to retry against something that's clearly not going to succeed. This protects the struggling dependency from being kept down by retry traffic, and it protects your own system's resources from being tied up waiting on calls that are very unlikely to succeed.

After the cooldown, let a small number of test requests through to check whether the dependency has recovered before resuming full traffic, rather than an abrupt full resumption that could immediately overwhelm it again.

Retrying assumes the request will eventually succeed. Sometimes the better answer is a fallback: cached data that's slightly stale but good enough, a degraded response that skips a non-essential enrichment step, a clear error to the user instead of a spinner that eventually times out anyway. Decide, per dependency, whether retry or fallback is the right first response, since treating everything as retryable extends outages that a good fallback would have quietly absorbed.

A 99.95% target leaves you about 4.38 hours of downtime a year to spend¹, and a retry storm that keeps hammering an already struggling dependency can burn through a meaningful slice of that in one bad afternoon that a well-designed fallback would have avoided entirely.

Retrying a request that isn't idempotent, one that has a side effect like charging a card or sending an email, risks duplicating that side effect every time a retry fires after a response was lost but the underlying action actually succeeded. Before adding retry logic to any write path, confirm it's genuinely safe to execute more than once, or add an idempotency key so a retry returns the original result instead of repeating the action.

This is a common source of embarrassing duplicate-charge incidents, and it's entirely avoidable by treating idempotency as a precondition for retryability, not an afterthought.

When each layer of a call chain retries independently, a client retrying three times against a service that itself retries three times against its own dependency can multiply into nine actual attempts at the bottom of the chain from a single original request. Coordinate retry behavior across layers, often by having only the outermost layer retry and having inner layers fail fast, so the effective load multiplication is understood and intentional rather than an accident of independently reasonable-looking decisions.

Trace a request through your actual call chain and count how many total attempts a single failure could generate; the number is often larger and more surprising than anyone expected going in.

A simple fix that works well in practice: pass a retry budget or attempt count along with the request itself, so an inner layer can see it's already been retried upstream and skip its own retry rather than compounding it. That one small piece of shared context prevents most of the accidental multiplication without requiring every team to coordinate their retry configuration manually.

Check each dependency call against this list:

• Retry with exponential backoff and jitter, and always cap the number of attempts.
• Add a circuit breaker so a dependency that is clearly down stops receiving requests during a cooldown.
• Decide per dependency whether retry or a fallback, such as slightly stale cached data, is the first response.
• Confirm a write path is idempotent, or add an idempotency key, before adding retries to it.
• Let only the outermost layer retry so attempts do not multiply down the call chain.