Designing for Partial Failure

Assume something is always broken

With dozens of services, some dependency is always slow or failing. Without protection, failures cascade: callers wait, threads and connections run out, and the caller fails too. The core defences: timeouts on every remote call, shorter than the caller's own deadline; retries only for idempotent operations, with exponential backoff, jitter and a cap; circuit breakers that stop calling a failing dependency and fail fast; bulkheads that give each dependency its own connection pool or concurrency limit; and graceful degradation, serving a reduced response (cached data, a default, an omitted widget) instead of an error. Prefer asynchronous communication for non-critical work so a slow consumer never blocks a user request. Avoid long synchronous call chains: each extra hop adds latency and failure probability. The resilience patterns course covers implementation details.

A client call with timeout, retry and fallback

Recommendations are optional, so failure returns an empty list rather than an error.

import httpx

client = httpx.AsyncClient(timeout=httpx.Timeout(0.3))   # 300 ms budget for this dependency

async def recommendations_for(user_id):
    for attempt in range(2):
        try:
            r = await client.get(f"http://recs.shop.svc.cluster.local/users/{user_id}")
            r.raise_for_status()
            return r.json()["items"]
        except (httpx.TimeoutException, httpx.TransportError):
            if attempt == 1:
                break
            await asyncio.sleep(0.05)
        except httpx.HTTPStatusError:
            break
    return []          # graceful degradation: page renders without recommendations

A restaurant with one slow supplier

If the dessert supplier is late, a well-run restaurant still serves the main course and apologises about dessert. A badly run one stops seating new customers until the desserts arrive.

Quick check: A product page calls an optional recommendations service that is timing out. What is the best behaviour?

• Return HTTP 500 for the whole page
• Render the page without recommendations after a short timeout
• Wait indefinitely for recommendations
• Retry forever