Retry, Circuit Break, or Dead-Letter: Handling a Failing Consumer

A brief network blip or a downstream service's momentary slowdown is exactly what retries are for: wait a short, increasing interval and try again, and most transient failures resolve within a few attempts. Without a cap, a retry loop against a genuinely down dependency just hammers it harder while burning through your consumer's capacity on messages that were never going to succeed this cycle.

Use exponential backoff with a maximum retry count, and make sure retries are visible in your monitoring; a consumer quietly retrying the same message hundreds of times looks like healthy processing on a naive lag dashboard while actually making no real progress.

A circuit breaker trips after enough consecutive failures against a specific dependency and stops sending new requests to it for a cooldown period, checking back periodically rather than continuously. This protects a struggling downstream service from a consumer that would otherwise keep retrying against it at full volume, and it protects the consumer from burning all its capacity on calls that are very likely to fail anyway.

Set the failure threshold and cooldown based on the dependency's actual recovery pattern, not a generic default. A dependency that typically recovers within thirty seconds needs a much shorter cooldown than one that tends to stay down for several minutes.

Once retries are exhausted, a message goes to a dead-letter queue instead of being dropped silently or blocking the whole partition behind it. This keeps the rest of the topic moving while preserving the failed message for investigation, which matters both for debugging and for reprocessing once the underlying issue is fixed.

A dead-letter queue that's never monitored is just a place where data quietly disappears with extra steps. Alert on its growth rate, and review what's actually landing there on a regular cadence, not only when someone happens to notice it's grown large.

In practice: retry a few times with backoff for transient failures, let a circuit breaker trip if a dependency is consistently failing so you're not retrying against a wall, and send anything that still fails to a dead-letter queue for later handling. The failure mode to watch for is a circuit breaker set so aggressively that everything ends up dead-lettered within seconds of a minor blip, which erases the distinction between a genuine outage and a normal transient hiccup.

Apply the three layers in this order:

1. Retry a few times with exponential backoff and a maximum count for transient failures, and make retries visible in monitoring.
2. Let a circuit breaker trip after consecutive failures against a dependency, then check back after a cooldown.
3. Send anything that still fails to a dead-letter queue instead of dropping it or blocking the partition.
4. Monitor the dead-letter queue and reprocess its messages once the underlying issue is fixed.
5. After an incident, note which layer caught the failure and tune settings to match real failure patterns.

At 99.9 percent availability a system has roughly 8.76 hours of allowed downtime a year, and at 99.99 percent that shrinks to about 52.6 minutes¹; your retry and circuit breaker settings are effectively spending or saving that budget every time a dependency has a bad few minutes. A cooldown tuned too long burns budget waiting to retry something that recovered already; one tuned too short burns it hammering something that hasn't.

After an incident, ask specifically which of the three layers handled it: did retries quietly absorb a brief blip, did the circuit breaker trip and protect the dependency, or did messages end up dead-lettered because all three layers ran out of options. That answer tells you whether your current tuning matches the failure patterns you're actually seeing, rather than the ones you assumed when you first configured retries and circuit breakers.

A pattern of dead-lettered messages from the same dependency, incident after incident, usually means the circuit breaker's threshold or cooldown needs adjusting, not that the dead-letter queue is doing anything wrong. Fix the layer that's actually undertuned instead of just clearing the queue and moving on.