# Rate Limiting Rate limiting caps how many requests a client (identified by user, API key, IP address or tenant) can make in a given time window. The limit applies whatever the state of the server: a client that sends its 51st request in a minute against a 50-per-minute quota gets rejected even if the server is idle. That's the difference with [[Load Shedding]], which reacts to the server's own load, and with [[Backpressure]], where a consumer asks its producer to slow down and the producer cooperates. ## Why it matters - **Abuse and attack mitigation**: brute-force logins, credential stuffing, scraping, application-layer floods - **Fairness between tenants**: one customer's runaway script shouldn't eat the capacity everyone shares. Stripe's request rate limiter rejected millions of requests in a single month, many from test-mode scripts gone wrong - **Cost control**: on metered APIs, a limit is also a spending cap - **Protecting dependencies**: a limit in front of a fragile downstream service keeps it from being overwhelmed, one of the defenses against [[Cascading Failures]] ## Algorithms | Algorithm | How it works | Trade-off | |---|---|---| | **Token bucket** | A bucket holds up to *b* tokens and refills at rate *r*; each request takes a token, an empty bucket means rejection | Allows bursts up to *b* while enforcing the average rate. Used by Stripe, AWS API Gateway and the Anthropic API | | **Leaky bucket** | Requests enter a bucket that drains at a constant rate; overflow is rejected | Smooths output to a steady rate; bursts get queued or dropped | | **Fixed window** | Count requests per calendar window (e.g., per minute) and reset at the boundary | Cheap, one counter per client, but a client can send twice the limit around a boundary (end of one window, start of the next) | | **Sliding window log** | Keep a timestamp per request and count those in the last *N* seconds | Exact, but memory grows with the number of requests | | **Sliding window counter** | Weight the previous window's count by how much of it still overlaps the current window | Close to exact with two counters. Cloudflare measured 0.003% of requests wrongly allowed or limited over 400 million requests | ## Where it's enforced At the edge or API gateway (Cloudflare, AWS API Gateway, [[Kong AI Gateway]], [[Cloudflare AI Gateway]]), in the application itself, or both. With several servers, the counters have to be shared, usually in a fast store like [[Redis]] (Stripe's rate limiters run on Redis) or a per-key coordinator such as [[Cloudflare Durable Objects]]. Keys can be anything that identifies the caller: IP, user, API key, organization, endpoint, or a combination. For LLM APIs, the unit that matters is often tokens rather than requests. The [[Claude API]], for example, limits requests per minute, input tokens per minute and output tokens per minute separately for each model class. ## Telling the client - **HTTP 429 Too Many Requests** (RFC 6585, 2012) is the standard status code. HTTP 503 Service Unavailable fits overload of the whole server better - **Retry-After** (RFC 9110) tells the client how long to wait, in seconds or as a date - **RateLimit and RateLimit-Policy header fields** would let servers advertise quotas and remaining capacity up front. As of October 2026 they are still an IETF Internet-Draft (draft-ietf-httpapi-ratelimit-headers), not an RFC. Many APIs send their own headers instead (e.g., `anthropic-ratelimit-requests-remaining`) ## The client's side A well-behaved client reads Retry-After and waits that long. Without it, it [[Retry pattern|retries]] with [[Exponential Backoff]] and jitter, within a retry budget. Retrying a 429 immediately, or from many workers at once, uses the quota up again as soon as it refills. ## Limits - Choosing the limit is a guess: too strict and legitimate bursts fail, too loose and it doesn't protect anything - A rate limit doesn't protect the server from overload when many clients are each within their quota. That still needs load shedding, [[Bulkheads]] and [[Timeouts (software)|timeouts]] - Limits keyed on IP addresses penalize users behind shared NAT and are easy to evade with many addresses ## References - RFC 6585, Additional HTTP Status Codes (429 Too Many Requests), April 2012: https://www.rfc-editor.org/rfc/rfc6585 - RFC 9110, HTTP Semantics, section 10.2.3 Retry-After: https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after - IETF draft, RateLimit header fields for HTTP: https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/ - Paul Tarjan, "Scaling your API with rate limiters", Stripe (30 March 2017): https://stripe.com/blog/rate-limiters - Julien Desgats, "How we built rate limiting capable of scaling to millions of domains", Cloudflare (7 June 2017): https://blog.cloudflare.com/counting-things-a-lot-of-different-things/ - AWS API Gateway, Throttle requests to your REST APIs: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html - Cloudflare WAF, Rate limiting rules: https://developers.cloudflare.com/waf/rate-limiting-rules/ - Anthropic, Rate limits: https://docs.anthropic.com/en/api/rate-limits - Wikipedia, Token bucket: https://en.wikipedia.org/wiki/Token_bucket - Wikipedia, Leaky bucket: https://en.wikipedia.org/wiki/Leaky_bucket ## Related - [[Load Shedding]] - [[Backpressure]] - [[Exponential Backoff]] - [[Cascading Failures]] - [[Bulkheads]] - [[Circuit Breaker]] - [[Fault Tolerance]] - [[Timeouts (software)]] - [[Redis]] - [[Cloudflare WAF]] - [[Cloudflare AI Gateway]] - [[AI Gateway]] - [[Retry pattern]]