Rate limiting is the art of refusing work cheaply so you can keep doing work at all. Below, a live token bucket you can starve, feed, and flood — and watch exactly when it starts saying no.
while(true) loop with no sleep, sending one HTTP request per iteration. How many requests per second is that from a single laptop?Whatever you guessed, it's more. A tight loop over a keep-alive connection pushes 2,000–10,000 requests a second. From one machine. On hotel Wi-Fi.
Maya runs checkout for a mid-size shop: three pods, comfortable at 220 requests a second, Friday peak around 310. At 18:04 a partner's integration script hits a bug — its error handler retries instantly, forever. By 18:05 her service is fielding 4,100 requests a second from one API key. Latency climbs, health checks fail, pods restart, and every restart makes the survivors' load worse.
Nothing was hacked. Nobody meant harm. The system did exactly what it was built to do: try to serve everyone — which is precisely the bug.
The instinctive answer is capacity: add pods, the load balancer spreads the pain. But watch what actually happens when demand passes supply and every failed request comes back. Press play.
Every timeout produced a retry. Every retry added load. Every added load produced more timeouts. Scaling out buys minutes; the feedback loop eats them. The fleet didn't lose to 4,100 rps — it lost to its own refusal to refuse.
Here's the move Maya's team shipped that weekend, and it's almost embarrassingly small: count each client's requests, and past a line, refuse instantly. That's rate limiting. The refusal is the feature — a rejected request costs microseconds and no database work, so saying no stays cheap even at 10,000 rps.
The counting scheme they used — the one most of the industry uses — is a token bucket. Give each client a bucket holding at most C tokens. Tokens drip in at r per second. Each request takes one token to pass. Empty bucket, no entry: HTTP 429 Too Many Requests.
The entire mechanism, no framework, no queue:
tokens = 10 # capacity C — the burst allowance
refill = 5 # r tokens per second — the sustained rate
last = now()
def allow():
global tokens, last
tokens = min(10, tokens + (now() - last) * refill)
last = now()
if tokens >= 1:
tokens -= 1; return True # 200: do the work
return False # 429: refuse in ~0 msNow run it. A burst of twelve requests lands on a full bucket. Step through what the counter does:
Twelve arrived; eleven got served within half a second; two were told — instantly, politely, with a machine-readable status — to come back later. Total cost of the two refusals: effectively zero. Compare that to chapter 02, where refusal-by-collapse cost everything.
Before you touch the knob, read the bucket's diary. Full bucket, a burst of 8 spends it in one gulp; idle time pays it back at 5 tokens a second; then a sustained 8 rps — 3 more than the refill — drains it for good:
Two promises, two numbers. Capacity (10) is the savings account — how big a spike you'll absorb without complaint. Refill (5/s) is the salary — the rate you'll sustain forever. Clients that spend in bursts live off capacity; clients that grind steadily live off refill. Mixing the two up is the most common rate-limit misconfiguration in production.
This is the whole chapter: one slider. It sets incoming traffic λ against the same bucket — capacity 10, refill 5/s — running live right now. Push λ past 5 and watch which number saturates and which one climbs.
Below 5 rps, the bucket is a bystander — everything passes. At exactly 5, you're spending your salary to the cent. Past 5, allowed throughput flatlines at the refill rate no matter how hard you push, and the 429 percentage absorbs the entire difference. At λ=30 the service still does its 5 rps of real work, calmly, while refusing 83% of the flood in microseconds.
Rate limiting does not solve overload. It chooses who feels it — and every choice of C and r is a real trade-off. Same bucket, three configurations:
And the parts the bucket can't fix, stated plainly:
Distributed counters drift. Ten gateway nodes each holding "the" bucket will collectively let through more than one bucket's worth, unless they share state (Redis, usually) — which adds a network hop to the thing you built to be free.
Clients retry. A 429 with no Retry-After header and no client-side jitter just reruns chapter 02 with a different status code.
Fairness is per-key, not global. Limit by API key and one customer's ten keys beat another's one. Limit by IP and a whole office NAT looks like one abuser. The counter is easy; deciding what to count is the design work.
Capacity is what you forgive.
Refill is what you promise.
The 429 is you keeping that promise out loud.