# Load Shedding Load shedding means that a server close to overload rejects part of the incoming work, early and cheaply, so it can keep completing the rest. An overloaded server that accepts every request ends up answering most of them too late to be useful. ## Why it matters David Yanacek (AWS) separates *throughput*, all the requests sent to a server, from *goodput*, the requests answered without error and fast enough for the client to use the response. Past a certain load, throughput keeps rising while goodput collapses. Overload also feeds itself: clients time out, the server keeps working on requests nobody is waiting for anymore, the clients retry, and the extra load causes more timeouts. In a deep call graph where each layer retries, the load multiplies at every level. That positive feedback loop is a classic path to [[Cascading Failures]]. A server that sheds load keeps its goodput close to its capacity even as offered load keeps growing. Shedding isn't free, though: past some point, the cost of rejecting requests dominates and goodput drops anyway. ## How to shed load - **Reject early and cheaply.** Return an error such as HTTP 503 before doing expensive work, ideally at the first service that receives the request rather than deep in the stack - **Drop work that has already expired.** Requests can wait in TCP buffers or queues until the client has given up. Amazon propagates the remaining time ("deadline") between hops and checks it after taking a request off a queue. See [[Timeouts (software)]] - **Shed the least important work first.** Google tags requests with a criticality (from CRITICAL_PLUS down to SHEDDABLE) and drops the sheddable traffic before anything critical - **Let the limit adapt.** Netflix's concurrency-limits library borrows ideas from TCP congestion control to estimate how many concurrent requests a service can handle, instead of a fixed requests-per-second limit picked from a load test. The relationship behind it is [[Little's Law]]: concurrency = arrival rate × latency - **Throttle on the client too.** In Google's adaptive throttling, each client tracks how many of its requests the backend accepted over the last two minutes and starts failing requests locally when the backend rejects too many [[Michael Nygard]] lists the same idea as "Shed Load" among the stability patterns in [[Release It! (Book)|Release It!]]. ## Load shedding, backpressure and rate limiting - **[[Backpressure]]** is cooperative: the consumer asks the producer to slow down, and the producer listens - **Load shedding** is self-defense: the server protects itself based on its own state, whether callers cooperate or not - **[[Rate Limiting|Rate limiting]]** (throttling) enforces a quota per client for fairness, regardless of how loaded the server is. Amazon uses both: throttling for fairness between clients, load shedding for overload ## Limits Rejected requests tend to come back as [[Retry pattern|retries]]. Shedding only helps if clients back off with jitter and respect a retry budget ([[Exponential Backoff]]). It also has to be tested: Yanacek's advice is to load-test a service to the point where it breaks, and far beyond, to see whether shedding actually holds goodput up. ## References - David Yanacek, "Using load shedding to avoid overload", Amazon Builders' Library (2019): https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/ - Alejandro Forero Cuervo, "Handling Overload", in Site Reliability Engineering (Google, O'Reilly, 2016), chapter 21: https://sre.google/sre-book/handling-overload/ - Mike Ulrich, "Addressing Cascading Failures", in Site Reliability Engineering, chapter 22: https://sre.google/sre-book/addressing-cascading-failures/ - Netflix concurrency-limits: https://github.com/Netflix/concurrency-limits - Michael Nygard, Release It! Second Edition, Pragmatic Bookshelf, 2018 (Stability Patterns chapter): https://pragprog.com/titles/mnee2/release-it-second-edition/ ## Related - [[Cascading Failures]] - [[Backpressure]] - [[Exponential Backoff]] - [[Timeouts (software)]] - [[Bulkheads]] - [[Little's Law]] - [[Site Reliability Engineering (SRE)]] - [[Fault Tolerance]] - [[Release It! (Book)]] - [[Michael Nygard]] - [[Circuit Breaker]] - [[Rate Limiting]] - [[Retry pattern]]