Resilience Patterns: Circuit Breakers, Retries, Bulkheads

What you need to know

• Retries with exponential backoff and jitter handle transient failures. Circuit breakers handle sustained failures. Both are needed for robust distributed system behavior.
• Without circuit breakers, a failing downstream service drains threads, exhausts connection pools, and cascades failures to the calling service. The calling service goes down because of a dependency it does not own.
• Bulkheads limit the blast radius of a dependency failure to the specific functionality that depends on that service.
• Timeouts on every external call are the prerequisite for all other resilience patterns. Without timeouts, retries never give up and circuit breakers never open.
• Use a library for resilience patterns. Manual implementations miss edge cases that battle-tested libraries handle correctly.

The core argument

A web service that calls five external dependencies inherits the failure probability of all five. If each dependency has 99.9% availability (one 9-minute outage per week), the calling service's availability without resilience patterns is 0.999^5 = 99.5%, or about 21 hours of downtime per year attributable entirely to dependency failures. With resilience patterns, the calling service can stay up during most dependency failures by degrading gracefully: returning cached data when the cache is unavailable, showing a fallback message when a non-critical service is down, and queuing operations that require the failed service for replay when it recovers.

The implementation order matters. Start with timeouts on every external call. A service that calls a database without a timeout will hang indefinitely on a connection that will never complete, blocking that thread forever. Timeouts are the foundation on which retries and circuit breakers operate. Add retries with exponential backoff for idempotent operations (GET requests, reads). Add circuit breakers for high-traffic paths to dependencies that experience periodic degradation. Add bulkheads when a single dependency's failure is consuming disproportionate resources and affecting unrelated functionality.

The pattern combination that matters for SaaS applications is: short timeouts (500ms to 2 seconds depending on the operation), two to three retries with exponential backoff and jitter for transient failures, circuit breakers that open after 50% failure rate over 10 seconds and half-open after 30 seconds to test recovery, and bulkheads that limit concurrent calls to any single external dependency to a configured maximum. This combination handles the common failure modes that real distributed systems encounter.

Common mistakes

1. Not setting timeouts on database connections and external API calls. Calls without timeouts block indefinitely on network failures, connection timeouts, and hanging servers. Every external call needs a timeout configured at the client level. The default timeout for most HTTP clients is either very long or infinite; always set explicit timeouts.

1. Retrying non-idempotent operations. A POST request that creates a new record should not be retried automatically if the first attempt succeeds but the response is lost due to a network error. Retrying creates a duplicate record. Only idempotent operations (GET, DELETE with idempotency keys, POST with idempotency keys) should be retried automatically. Non-idempotent operations should fail fast and let the caller decide whether to retry.

1. Not adding jitter to exponential backoff. Retries without jitter from many concurrent clients create synchronized retry waves that hit the recovering service simultaneously. Add random jitter: wait = baseDelay * 2^attempt + random(0, 100ms). This spreads retries across the backoff window and reduces load on a recovering service.

1. Opening circuit breakers on percentage-based thresholds with too small a sample window. A circuit breaker that opens when 50% of requests fail in a 5-second window will open immediately if 2 of the first 4 requests fail in a new deployment. Set minimum request count thresholds (do not open the circuit breaker until at least 20 requests have been evaluated in the window) to prevent premature circuit opening from small samples.

1. Not implementing fallback behavior when the circuit is open. An open circuit breaker that returns a 500 error is better than an open circuit that blocks indefinitely, but it is worse than an open circuit that returns a cached result, a degraded response, or a graceful "service temporarily unavailable" message. Define the fallback behavior for each circuit breaker when implementing it.

Where to start

1. Audit every external dependency call for timeout configuration. Check every HTTP client, database connection pool, and cache client configuration for explicit timeout settings. Add timeout configuration to any call that does not have one. For API calls: 500ms to 2 seconds. For database queries: 5 to 30 seconds depending on operation type. For long-running operations: longer timeouts with progress tracking.

1. Install a resilience library and add retry logic to the highest-impact external calls. For Node.js: cockatiel. For Python: tenacity. Configure retry for GET requests and other idempotent operations on the two or three external services that experience the most transient failures. Add exponential backoff with jitter to the retry configuration.

1. Add circuit breakers to the calls that, when they fail, would cascade the failure to user-facing functionality. The payment processor integration, the primary database connection, and any external API called on every user request are the highest-priority circuit breaker candidates. Start with conservative circuit breaker thresholds (70% failure rate over 10 seconds) and tighten based on observed behavior.

Related reading

• Rate Limiting Algorithms Compared: Token Bucket, Leaky Bucket, Fixed Window
• Observability in 2026: Metrics, Logs, Traces
• Job Failure Recovery: How Good SaaS Companies Sleep at Night
• Pager Fatigue and How to Prevent It