Lesson 17 / 25
Designing for Partial Failure
Protect services from slow and failing dependencies.
By Rahul Agarwal · 15 years in web development
Assume something is always broken
With dozens of services, some dependency is always slow or failing. Without protection, failures cascade: callers wait, threads and connections run out, and the caller fails too. The core defences: timeouts on every remote call, shorter than the caller's own deadline; retries only for idempotent operations, with exponential backoff, jitter and a cap; circuit breakers that stop calling a failing dependency and fail fast; bulkheads that give each dependency its own connection pool or concurrency limit; and graceful degradation, serving a reduced response (cached data, a default, an omitted widget) instead of an error. Prefer asynchronous communication for non-critical work so a slow consumer never blocks a user request. Avoid long synchronous call chains: each extra hop adds latency and failure probability. The resilience patterns course covers implementation details.
A client call with timeout, retry and fallback
Recommendations are optional, so failure returns an empty list rather than an error.
import httpx
client = httpx.AsyncClient(timeout=httpx.Timeout(0.3)) # 300 ms budget for this dependency
async def recommendations_for(user_id):
for attempt in range(2):
try:
r = await client.get(f"http://recs.shop.svc.cluster.local/users/{user_id}")
r.raise_for_status()
return r.json()["items"]
except (httpx.TimeoutException, httpx.TransportError):
if attempt == 1:
break
await asyncio.sleep(0.05)
except httpx.HTTPStatusError:
break
return [] # graceful degradation: page renders without recommendationsA restaurant with one slow supplier
If the dessert supplier is late, a well-run restaurant still serves the main course and apologises about dessert. A badly run one stops seating new customers until the desserts arrive.
Quick check: A product page calls an optional recommendations service that is timing out. What is the best behaviour?
- Return HTTP 500 for the whole page
- Render the page without recommendations after a short timeout
- Wait indefinitely for recommendations
- Retry forever
Answer
Render the page without recommendations after a short timeout — Optional dependencies should degrade gracefully so they cannot take down the main flow.