Circuit Breakers and Bulkheads: Stopping One Outage Becoming Three

Without a breaker, every caller to a slow dependency waits out its own timeout before giving up, and while it waits, it's holding a connection, a thread, or a resource from whatever pool it borrowed from. Enough concurrent callers stuck waiting exhausts that pool, and now requests that have nothing to do with the failing dependency start failing too, because there's nothing left in the pool to serve them.

A tight uptime target leaves very little room for that kind of cascade to play out before it's a real incident, since the downtime budget behind a high availability target is measured in minutes a year, not hours¹. A circuit breaker interrupts the pattern by failing fast once a dependency is clearly unhealthy, instead of letting every caller queue up to discover that themselves.

A breaker has three states: closed, calls go through normally while the dependency looks healthy; open, calls fail immediately without even attempting the dependency, once failures cross a threshold; and half-open, a trial state that lets a small number of calls through to test whether the dependency has recovered.

Half-open is the state teams most often misconfigure. Let too much traffic through during the test and a dependency that's only partially recovered gets hit with a full load again, tripping the breaker straight back to open before it had a real chance to stabilize. Keep the trial volume small and the threshold to fully close deliberate, not a single lucky successful call.

A bulkhead gives each downstream dependency its own connection or thread pool instead of sharing one pool across all of them. When one dependency slows down, it exhausts its own allocation and its own alone, leaving the resources the rest of your application needs to talk to healthy dependencies untouched.

Without that isolation, a single struggling dependency can starve calls to completely unrelated, perfectly healthy services simply because they were all drawing from the same shared pool. Bulkheads and circuit breakers solve related but different problems, and most resilient systems need both: the breaker decides whether to call a dependency at all, the bulkhead limits the blast radius when something goes wrong anyway.

A retry policy without a sane timeout underneath it just multiplies load on a dependency that's already struggling, each failed attempt retried adds more requests to something that couldn't handle the load it already had. Set a timeout that's meaningfully shorter than whatever is calling you is willing to wait, and use exponential backoff with jitter between retry attempts so retries from many callers don't all land at once and create a new spike.

These settings interact with each other and with the breaker's own thresholds, so tune them as a set, not independently: a generous timeout paired with aggressive retries can keep a breaker from ever tripping even while the dependency is clearly unhealthy, because from the breaker's view, calls are eventually succeeding, just very slowly.

The only way to know a breaker actually works is to watch it trip. In a lower environment, deliberately fail a dependency, block its port, return errors from a test double, kill the process, and confirm the breaker opens, calls fail fast instead of hanging, and the application degrades gracefully rather than cascading.

Watch what "degrades gracefully" actually looks like for your users during that test, a fallback response, a cached value, a clearly disabled feature, rather than assuming the breaker tripping automatically means the user experience is acceptable. Those are two different things, and only one of them is guaranteed by the breaker configuration alone.

A simple drill to prove the breaker works:

1. In a lower environment, deliberately fail a dependency by blocking its port, returning errors from a test double, or killing the process.
2. Confirm the breaker opens once failures cross its threshold, so calls fail immediately instead of hanging.
3. Check that half-open lets only a small number of trial calls through, and that the breaker closes again only when they succeed.
4. Watch what graceful degradation looks like for your application, and confirm calls to healthy dependencies keep working through their own bulkhead pools.