# Retry pattern The retry pattern repeats a failed operation in the hope that the failure was transient: a dropped connection, a brief overload, a timeout. It's a design decision with three parts: *whether* an error is worth retrying, *how many times* (and within what overall deadline), and *where* in the call chain the retries happen. How long to wait between attempts is a separate question, covered in [[Exponential Backoff]]. ## Which errors to retry - **Usually retryable**: timeouts, connection resets, HTTP 503 Service Unavailable, HTTP 429 Too Many Requests (after the delay in `Retry-After`, see [[Rate Limiting]]), throttling errors - **Not retryable**: errors that will fail the same way next time, such as 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found or a validation error. Retrying them only adds load and delays the real error Microsoft's Azure guidance lists three responses to a failure: cancel when the fault isn't transient, retry immediately for rare and unusual faults (a corrupted packet), and retry after a delay for faults like busy services. It also recommends logging every failure that causes a retry, so the underlying problem stays visible. ## How many times - **Cap the attempts.** The AWS SDKs default to 3 attempts (one request plus two retries) in standard mode. Google's SRE book describes a per-request budget of three attempts: a request that failed three times probably hit an overloaded datacenter, and a fourth try won't help - **Respect an overall deadline.** The retries must fit within the time the caller is still willing to wait; past that, they waste work nobody uses. See [[Timeouts (software)]] - **Use a retry budget.** Limit retries across all requests, not only per request. Google's clients retry only while retries stay below 10% of their requests. The AWS SDKs use a token bucket (the "retry quota"): each retry costs tokens, successes refill them, and when the bucket is empty the SDK fails fast instead of retrying ## Retry at one layer Retries multiply across layers. Marc Brooker's example: a call goes five services deep and each layer makes three attempts, so a failing database at the bottom receives up to 3^5 = 243 times the load. Amazon's practice is to retry at a single point in the stack and let the other layers fail fast. Retrying only at the very top has its own cost: it repeats the work every lower call already did. ## Retry storms When a dependency slows down, every caller starts retrying at once. The extra traffic slows it further, which causes more timeouts and more retries. This positive feedback loop is a well-documented path to [[Cascading Failures]], and the reason retries need jitter, budgets and a [[Circuit Breaker]] that stops them entirely when the dependency is down. ## Retries need idempotency A timeout doesn't mean the operation failed. The server may have done the work and lost the response. Retrying a non-idempotent call (charge a card, send an email) can then do it twice. The fix is [[Idempotency]]: design operations so that repeating them has no extra effect, or send an *idempotency key*. Stripe's API accepts an `Idempotency-Key` header (up to 255 characters, ideally a V4 UUID); a retry with the same key returns the original result instead of charging again. [[Resend]] works the same way for emails. ## Asynchronous retries With queues, retries move out of the request path. Delivery is usually *at-least-once*: a message may be processed more than once, so consumers must be idempotent. After a maximum number of attempts, the message goes to a *dead-letter queue* for inspection instead of blocking the queue forever. Amazon SQS calls that limit `maxReceiveCount`; [[Cloudflare Queues]] has per-message retry counts and a dead-letter queue too. Durable workflow engines such as [[Cloudflare Workflows]] retry each step on its own, with a configurable limit and backoff. ## How it fits with the other patterns - [[Circuit Breaker]]: once the breaker is open, stop retrying - [[Load Shedding]]: rejected requests come back as retries, so shedding only works when clients back off and respect budgets - [[Rate Limiting]]: read `Retry-After` before retrying a 429 - [[Bulkheads]]: retries hold threads and connections too, so keep them inside the dependency's partition - [[Backpressure]]: answering a "slow down" signal with immediate retries defeats it ## Implementations Polly (.NET), Resilience4j (Java), tenacity (Python), and the retry modes built into the AWS SDKs (standard, adaptive, legacy). These libraries provide error filters, attempt caps, backoff and jitter as configuration. ## References - Microsoft Azure Architecture Center, Retry pattern: https://learn.microsoft.com/en-us/azure/architecture/patterns/retry - Marc Brooker, "Timeouts, retries, and backoff with jitter", Amazon Builders' Library (2019): https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ - Alejandro Forero Cuervo, "Handling Overload", Site Reliability Engineering (Google, 2016), chapter 21 (retry budgets): https://sre.google/sre-book/handling-overload/ - Mike Ulrich, "Addressing Cascading Failures", Site Reliability Engineering, chapter 22: https://sre.google/sre-book/addressing-cascading-failures/ - AWS SDKs and Tools Reference Guide, Retry behavior: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html - Stripe API reference, Idempotent requests: https://docs.stripe.com/api/idempotent_requests - Brandur Leach, "Designing robust and predictable APIs with idempotency", Stripe (22 February 2017): https://stripe.com/blog/idempotency - Amazon SQS Developer Guide, Using dead-letter queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html - Polly, Retry resilience strategy: https://www.pollydocs.org/strategies/retry.html - Resilience4j, Retry: https://resilience4j.readme.io/docs/retry - tenacity: https://tenacity.readthedocs.io/en/latest/ ## Related - [[Exponential Backoff]] - [[Idempotency]] - [[Timeouts (software)]] - [[Circuit Breaker]] - [[Cascading Failures]] - [[Load Shedding]] - [[Rate Limiting]] - [[Bulkheads]] - [[Backpressure]] - [[Fault Tolerance]] - [[Site Reliability Engineering (SRE)]] - [[Cloudflare Queues]] - [[Cloudflare Workflows]]