The Circuit Breaker Checklist Most Teams Skip Half Of

A circuit breaker that trips on the first failure treats a single transient blip the same as a genuine outage, and will flap open and closed constantly against any dependency with even modest normal error noise, training the team to distrust or disable it. One that requires an unrealistically high failure count before tripping delays protection long enough that the cascading failure it exists to prevent has already started elsewhere in the system. Base the threshold on your dependency's actual observed error rate under normal conditions, not a default copied from a library's documentation example. Revisit that threshold periodically too, since a dependency's normal error rate drifts as its own traffic and infrastructure change over time.

After a breaker trips and its cooldown period passes, it should move to a half-open state that allows a small number of test requests through to check whether the dependency has recovered, rather than either staying fully closed to all traffic or snapping immediately back to fully open. The most common mistake is letting all traffic flood back the instant the breaker reopens, which can immediately overwhelm a dependency that had only just started to recover, tripping the breaker again in a loop that never actually resolves. Limit half-open traffic to a small, fixed number of probe requests, and only fully reopen once a clear majority of those probes succeed.

A circuit breaker protects against a single failing dependency; a bulkhead protects against that failing dependency consuming so many resources, threads, connections, memory, that it starves every other part of the system too, even parts with no relationship to the failure at all. Without bulkhead isolation, a slow or hanging call to one downstream service can exhaust your entire connection pool, and a customer trying to do something completely unrelated to the failing dependency gets a timeout anyway. Give each external dependency its own dedicated resource pool, sized deliberately, rather than sharing one pool across everything.

For example, a checkout service calls a tax provider, an address validator, and an email service through one shared pool of connections. When the email service hangs, requests pile up waiting on it and the pool empties, so tax calculation times out even though the tax provider is healthy. Splitting the pool into one per dependency, each sized to that dependency's importance, contains the damage: email calls back up in their own pool while checkout continues. Pair each pool with its own timeout so a slow call releases its slot quickly instead of holding it until the caller gives up.

• Return cached or default data when the failing dependency is non-critical to the immediate request, a recommendation widget, a secondary data enrichment call, rather than failing the whole request over one optional piece.
• Queue for later when the call is a write that can tolerate delay, rather than blocking the user-facing request on it succeeding synchronously.
• Fail the request clearly when there's genuinely no safe fallback, a payment authorization, for instance, rather than silently proceeding with incomplete or stale data that could mislead a downstream decision.

Deciding this per dependency in advance, not improvising during an incident, is what actually makes a circuit breaker rollout pay off.

A circuit breaker with no dashboard or alert tied to its state changes protects the system silently, which sounds fine until a dependency has been degraded for hours and nobody on the team noticed because nothing ever paged. Emit a metric on every state transition, closed to open, open to half-open, half-open back to closed or back to open, and alert specifically on a breaker staying open past a reasonable window, since that's a strong signal of a real, ongoing outage worth a human's attention rather than a transient blip the breaker absorbed on its own.