Rate limiting

  • Lesson 25 of 40
  • 10 min
  • Intermediate
  • 9 steps

One client gone wild can leave everyone else without service. You will see how to share capacity with per-client limits and what to do when even that is not enough.

Buggy script
10 req/s
Users served
100%
API CPU
40%
Rejected at the gate
0 req/s
usersbuggy scriptGatewayrate limiterGatewayGatewayrate limiterGatewaycountersAPIAPI98%serving almost only one client4,000 req/s40%limit: 10 req/s per client429 Too Many Requests4,000 req/s40%10 tokens, +10 per seconduser: 8 of 10 tokensscript: 0 tokens40%limit: 10 req/s per client429 Too Many RequestsExample limit: 60 per minutefixed windowback to zero every minute0:59 and 1:0060 + 60 = 120 in 2 ssliding windowlooks at the last 60 stoken bucketburst of up to 60, then 1/sbursts yes, average capped40%INCRlimit: 10 req/s per client429 Too Many Requests40%INCRRetry-After: 30now at 10 req/swithin the limit78%INCRtenant A: max 1,500 req/sover quota → 429within the limit90%INCRdropping background workbackground: dropped firstwithin the limit40%INCRlimit: 10 req/s per clientwithin the limit
  • request
  • slow request
  • waiting in queue
  • failed request

Step 1 of 9

Problem

One hasty script, everyone waiting

A customer deploys a script with a bug: it calls your API inside a loop with no pause, 4,000 requests per second. Almost all of the API's CPU goes into serving it.

Real users' requests wait behind that flood and many end in a timeout. Nobody is attacking you: a single client is taking the capacity that belonged to everyone.

Step 2 of 9

Solution

A doorman at the entrance

Put a rate limiter in a gateway in front of the API. Each client is identified (by API key, user id or IP) and gets a budget: in this example, 10 requests per second.

Whatever fits in the budget goes through; the rest is turned away on the spot with 429 Too Many Requests, without touching the API. The script loses 3,990 of its 4,000 requests per second.

What changed

  • Gateway with a rate limiter
  • Script limited to 10 req/s
  • API CPU: 98% → 40%
  • Users served: 58% → 100%

Step 3 of 9

Technique

How it counts: the token bucket

The most widely used algorithm is the token bucket. Each client has a bucket of up to 10 tokens that refills at 10 per second. Every request spends one token; with the bucket empty, the answer is 429.

A normal user hardly ever empties it: right now they have 8 of 10 left. The script drains it the moment new tokens arrive, so almost everything it sends is turned away.

What changed

  • One bucket per client

Step 4 of 9

Technique

Windows versus buckets

A fixed window counts requests per clock minute and goes back to zero on every exact minute. It is simple, but it has an edge: with a limit of 60 per minute, a client can send 60 at 0:59 and another 60 at 1:00, 120 in about 2 seconds.

A sliding window always looks at the last 60 seconds and removes that edge. The token bucket allows controlled bursts. That is why gateways usually offer one of these two.

Step 5 of 9

Problem

Two gateways, one budget

One gateway is no longer enough, so you put two behind a load balancer. If each counts on its own, a client that spreads its requests over both gets twice its limit; with 10 gateways, ten times.

The way out is to keep the counters in a fast shared store, such as Redis: each request does an atomic increment (INCR) and compares the total with the limit.

What changed

  • Second gateway
  • Shared counter store
  • Each client's counters live in the store

Step 6 of 9

Technique

Tell the client when to come back

A good 429 does not just say no: it says when to come back with the Retry-After: 30 header (in seconds) and usually reports the limit and what is left with headers such as X-RateLimit-Remaining, widely used though not standard.

A well-built client waits that long before retrying. The script's owner fixes the loop and it now uses 10 req/s, its fair share, without a single rejection.

What changed

  • Retry-After and limit headers
  • The script drops to 10 req/s
  • Rejected: 3,990/s → 0

Step 7 of 9

Problem

Case: one company with 8,000 users

You sell your API to companies. One of them runs a nightly import with 8,000 users; none goes over their personal limit, but together they add up to 2,400 req/s, 60% of a 4,000 capacity (example values). Every night, every other customer gets slow.

A per-user limit cannot see the problem. Add a per-tenant quota (the company account) based on its plan, on top of the per-user limit: with a 1,500 req/s quota, the 900 extra get a 429.

What changed

  • Per-tenant quota based on the plan
  • The import stays at 1,500 req/s

Step 8 of 9

Technique

When everyone arrives at once: shed load

Limits stop one client from hogging. But on launch day everyone arrives at once, each within their limit, and the total exceeds what you can serve: 6,000 req/s for a capacity of 4,000 (example values).

That is where load shedding comes in: the gateway drops background work first (prefetching, recommendations, analytics), 2,400 req/s. That leaves 3,600, 90% of capacity, and logins and payments keep working.

What changed

  • Peak of real users
  • The gateway drops low-priority work
  • Logins and payments: 100% served

Step 9 of 9

Summary

Summary

Give each client a budget and turn the excess away early, with a 429.

The token bucket tolerates short bursts and caps the average; the sliding window avoids the fixed window's edge.

With several gateways, keep the counters in a shared store.

Stack limits per user, per tenant and per endpoint, and use Retry-After to say when to come back.

If the whole system is saturated, drop low-priority work first.

You will learn

  • Why a single client can leave everyone else without service
  • How the token bucket and windows count
  • How to share one limit across several gateways
  • 429 responses, retries and per-tenant quotas
  • Shedding load when everyone arrives at once