Circuit Breakers and Bulkheads: Configuring Them So They Help

A breaker trips based on an error rate measured over a rolling window, not a single failed request, and moves into an open state where it fails fast instead of calling the downstream dependency at all. After a cooldown period it moves to a half-open state, letting a small number of probe requests through to check whether the dependency has recovered before fully closing again.

The rolling window size matters as much as the error rate threshold. Too short a window and normal noise trips it constantly; too long a window and it's slow to react to a dependency that's actually degrading.

Pick a window long enough to smooth over a single bad second of traffic but short enough that a genuinely failing dependency trips the breaker before its errors pile up into a bigger incident. There's no universal number here; it depends on your request volume against that specific dependency.

A circuit breaker stops you from calling a failing dependency, but if that dependency shares a connection pool or thread pool with everything else your service does, a slow dependency can exhaust the shared pool before the breaker's error-rate threshold even trips, since slow isn't the same as failing.

Separate pools per downstream dependency, the bulkhead pattern, keep one slow dependency from starving requests that have nothing to do with it. Size each pool to what that specific dependency actually needs, not to a single shared number that assumes every dependency behaves the same way.

If you're holding yourself to three nines of availability, work backward from that downtime budget when deciding how long a breaker stays open before it probes again¹.

A cooldown period that's too long keeps failing fast on a dependency that already recovered, quietly extending an outage past when it needed to end. A cooldown that's too short sends real traffic back at a dependency that hasn't actually stabilized, and can trip the breaker right back open.

Tripping a breaker doesn't make the underlying problem disappear; it moves the failure one hop up the call chain wearing a different name. If the caller has no fallback behavior for an open breaker, it just times out or errors instead, and whatever called it faces the exact same choice.

Design an explicit fallback for every breaker you add, whether that's cached data, a degraded response, or a clear error the caller can act on, rather than assuming an open breaker alone counts as handling the failure.

A call to a non-critical enrichment service, one that makes a response nicer but not wrong if it's missing, tolerates a much more aggressive breaker than a call to your payments processor, where you'd rather wait a little longer for a real answer than fail fast into a degraded checkout flow.

Configure breakers per dependency based on what failing fast actually costs you there, not a single default copied across every integration in the codebase.

A breaker that's never actually tripped in a rehearsed test is an untested piece of failure handling, no different from any other code path nobody has exercised. Fault injection, deliberately forcing a downstream call to fail or time out in a controlled environment, confirms the breaker trips at the threshold you configured and that the fallback you wrote actually runs.

Run this check whenever the threshold or cooldown changes, not only when the breaker is first added. A configuration change that looks harmless on paper can shift when it trips in ways that only show up once you force it.

A simple test routine covers both the breaker and its fallback:

1. Use fault injection in a controlled environment to force a downstream call to fail or time out.
2. Confirm the breaker trips at the error-rate threshold you configured, not sooner or later.
3. Check that the fallback you wrote actually runs while the breaker is open, whether that is cached data or a degraded response.
4. Watch the half-open probes after the cooldown and confirm the breaker closes once the dependency has recovered.