Distributed systems fail partially, so design for it. Bound every dependency call with a timeout, retry transient failures with capped backoff, trip a circuit breaker when a dependency is down, and degrade gracefully instead of cascading.
Key ideas
- Always set timeouts — one hung dependency must never hang the whole request.
- Retry with exponential backoff and jitter, with a hard cap — and make the operation idempotent first.
- Use circuit breakers and fallbacks so one slow dependency degrades a feature instead of taking down the system.
Warning
Retries without idempotency and caps turn a small outage into a self-inflicted DDoS.