Design a distributed rate limiter that caps how many requests a client (API key / IP) can make per time window.
How to approach it
Scale by following the request path and fixing the first real bottleneck. Reads usually dominate, so caching and replication come first; writes and storage are handled by sharding on a high-cardinality key. Push slow, retryable work onto queues. Add infrastructure only where the numbers justify it.
Scaling decisions
| Area | Choice |
|---|---|
| Store | Centralized Redis with atomic INCR + TTL (or a Lua script) |
| Latency | Local token bucket with periodic sync to Redis |
| Hot keys | Shard counters / add jitter for very hot clients |
| Failure | Fail-open for availability or fail-closed for protection (policy) |
Why
- Store: shared state so all gateway nodes agree on the count
- Latency: avoids a network hop per request at the cost of slight inaccuracy
- Hot keys: prevents a single Redis key from becoming a bottleneck
- Failure: decide what happens when the counter store is unreachable