Circuit Breakers and Bulkheads: Stopping a Failure From Spreading

A circuit breaker watches the failure rate of calls to a specific dependency, and once that rate crosses a threshold, it stops sending new calls entirely for a cooldown period, failing fast instead of letting every request queue up waiting on a dependency that's already struggling.

After the cooldown, it allows a small number of test requests through; if those succeed, it closes again and traffic resumes normally, and if they fail, it stays open and waits longer. The point isn't preventing the failure, it's preventing the failure from cascading into every other part of the system that's waiting on a response.

A bulkhead limits how much of a shared resource, a thread pool, a connection pool, concurrent request slots, any single dependency or feature can consume, so a slow or failing one can't starve every other feature of the same resource.

Without bulkheads, a circuit breaker on one dependency can still fail to help if every thread in your shared pool is already blocked waiting on that same slow dependency by the time the breaker trips. Partition your resource pools per dependency, or at least per criticality tier, so a checkout-blocking failure and a recommendations-widget failure aren't drawing from the same limited pool of threads.

A circuit breaker's whole job is making sure one flaky dependency doesn't spend your entire uptime budget at once, and at 99.95 percent availability that budget is only 4.38 hours a year¹.

A dependency failure that's contained by a breaker and a bulkhead costs you a degraded feature for its cooldown period; the same failure without containment can cost you the whole service for as long as the dependency stays down, which is a very different draw against the same annual budget.

• A failure threshold set so low that normal, brief error-rate blips trip the breaker constantly, training engineers to ignore its alerts or disable it outright.
• A cooldown period so short that the breaker flaps open and closed rapidly against a dependency that's genuinely struggling, adding overhead without actually protecting anything.
• Sharing one thread pool across every dependency despite having circuit breakers configured per dependency, which defeats the isolation a bulkhead is supposed to provide.
• No fallback behavior defined for when a breaker is open, so users get a hard error instead of a degraded but functional experience, like cached data or a simplified response.

Say a third-party shipping-rate API starts timing out under its own load. Without a circuit breaker, every checkout request waits the full timeout duration for a response that never comes, threads pile up waiting, and the connection pool serving that call exhausts, which starts affecting unrelated requests sharing the same pool. Checkout, and possibly the whole service, goes down with a dependency that was never actually part of its core function.

With a circuit breaker and a bulkhead isolating the shipping-rate call to its own resource pool, the breaker trips after a handful of timeouts, checkout falls back to a flat-rate shipping estimate for the cooldown period, and every other part of the service, including checkout's core purchase flow, keeps working normally. The dependency failure cost a slightly less accurate shipping quote for a few minutes instead of a full outage.

A circuit breaker's configuration shouldn't be a one-time setup task. Every time a breaker actually trips in production, that's real data about how that dependency fails, how long it typically takes to recover, and whether the current threshold and cooldown period matched what actually happened. Review the trip afterward the same way you'd review any incident: did the breaker trip too early, too late, or about right, and does the fallback behavior that activated actually serve users well, or just technically avoid an error.