# Cascading Failures A cascading failure is a failure that grows over time through positive feedback: one part of a system fails, and that raises the probability that other parts fail too. Google's [[Site Reliability Engineering (SRE)|Site Reliability Engineering]] book uses an example with two clusters. When cluster B fails, its traffic moves to cluster A, which goes from 1,000 to 1,200 requests per second. A can't handle that, runs out of resources, crashes, and pushes the load onto whatever is left. In [[Release It! (Book)|Release It!]], [[Michael Nygard]] separates two kinds of spread. A chain reaction moves sideways, between servers of the same layer, as each one that dies adds load to the others. A cascading failure moves upward: a crack in one layer breaks the layer that calls it. ## How failures spread - **Overload**: the most common cause according to Google. Capacity lost anywhere turns into extra load everywhere else - **Resource exhaustion**: threads, connections, memory. Nygard's examples are connection pools with no timeout and threads blocked on a slow integration point. When a server crashes or fails its health checks, the load balancer sends its share to the remaining servers - **[[Retry pattern|Retries]]**: naive retries add traffic exactly when a service is already overloaded, and retries at several layers multiply each other In both books' examples the original fault is small (one cluster, one slow dependency). The outage grows through what the rest of the system shares: resources, load and retry logic. Isolation ([[Loose Coupling]], bulkheads) limits how far it travels. ## How to stop them - **[[Timeouts (software)|Timeouts]]** on every remote call, a [[Circuit Breaker]] around unreliable dependencies, and [[Bulkheads]] that partition thread and connection pools so one failing part can't drain the rest (Release It!) - **[[Load Shedding|Load shedding]] and degraded results**: drop part of the traffic (e.g., return HTTP 503) or serve cheaper results as a server nears overload (Google SRE) - **[[Backpressure]]**: an overloaded component pushes back on its callers instead of queuing without limit - **Retry budgets**: cap retries per request and per process (Google's example: 60 retries per minute), back off exponentially with jitter ([[Exponential Backoff]]), and retry only [[Idempotency|idempotent]] operations - **Testing**: load-test to find the breaking point and how the system behaves past it, and use [[Chaos Engineering]] to check that the defenses work in production ## References - Mike Ulrich, "Addressing Cascading Failures", in Site Reliability Engineering (Google, O'Reilly, 2016), chapter 22: https://sre.google/sre-book/addressing-cascading-failures/ - Michael Nygard, Release It! Second Edition, Pragmatic Bookshelf, 2018 (Stability Antipatterns chapter): https://pragprog.com/titles/mnee2/release-it-second-edition/ - Martin Fowler, "CircuitBreaker" (2014): https://martinfowler.com/bliki/CircuitBreaker.html - Wikipedia, Cascading failure: https://en.wikipedia.org/wiki/Cascading_failure - https://concepts.dsebastien.net/concept/cascading-failures/ ## Related - [[Circuit Breaker]] - [[Backpressure]] - [[Chaos Engineering]] - [[Idempotency]] - [[Loose Coupling]] - [[Feedback Loop]] - [[Systems thinking]] - [[Release It! (Book)]] - [[Michael Nygard]] - [[Software Architecture]] - [[Timeouts (software)]] - [[Bulkheads]] - [[Load Shedding]] - [[Exponential Backoff]] - [[Fault Tolerance]] - [[Site Reliability Engineering (SRE)]] - [[Rate Limiting]] - [[Retry pattern]]