Lesson 17 / 25

Designing for Partial Failure

Protect services from slow and failing dependencies.

By · 15 years in web development

Assume something is always broken

With dozens of services, some dependency is always slow or failing. Without protection, failures cascade: callers wait, threads and connections run out, and the caller fails too. The core defences: timeouts on every remote call, shorter than the caller's own deadline; retries only for idempotent operations, with exponential backoff, jitter and a cap; circuit breakers that stop calling a failing dependency and fail fast; bulkheads that give each dependency its own connection pool or concurrency limit; and graceful degradation, serving a reduced response (cached data, a default, an omitted widget) instead of an error. Prefer asynchronous communication for non-critical work so a slow consumer never blocks a user request. Avoid long synchronous call chains: each extra hop adds latency and failure probability. The resilience patterns course covers implementation details.

A client call with timeout, retry and fallback

Recommendations are optional, so failure returns an empty list rather than an error.

import httpx

client = httpx.AsyncClient(timeout=httpx.Timeout(0.3))   # 300 ms budget for this dependency

async def recommendations_for(user_id):
    for attempt in range(2):
        try:
            r = await client.get(f"http://recs.shop.svc.cluster.local/users/{user_id}")
            r.raise_for_status()
            return r.json()["items"]
        except (httpx.TimeoutException, httpx.TransportError):
            if attempt == 1:
                break
            await asyncio.sleep(0.05)
        except httpx.HTTPStatusError:
            break
    return []          # graceful degradation: page renders without recommendations

A restaurant with one slow supplier

If the dessert supplier is late, a well-run restaurant still serves the main course and apologises about dessert. A badly run one stops seating new customers until the desserts arrive.

Quick check: A product page calls an optional recommendations service that is timing out. What is the best behaviour?

  • Return HTTP 500 for the whole page
  • Render the page without recommendations after a short timeout
  • Wait indefinitely for recommendations
  • Retry forever
Answer

Render the page without recommendations after a short timeout — Optional dependencies should degrade gracefully so they cannot take down the main flow.