Case Study: Hardening a Checkout Flow

Patterns working together

A checkout service calls cart, pricing, inventory, payments, fraud scoring and notifications. The hardened design: the gateway rate-limits per user and per IP, sheds low-priority traffic under overload, and propagates a 2-second deadline. Checkout calls pricing and inventory with 300 ms timeouts, one retry with jitter (both idempotent reads) and per-dependency bulkheads. Fraud scoring is optional under load: a circuit breaker with a fallback to rules-based scoring for low-value orders, and a kill switch. Payments uses an 800 ms per-attempt timeout, no automatic retry of the charge itself, an idempotency key derived from the order ID so the client or a later reconciliation can safely retry, and a breaker that returns "payment pending" with the order queued. Notifications are sent asynchronously through a queue. Dashboards show breaker states and fallback rates, and monthly game days inject latency into payments and fraud scoring to verify the behaviour.

Checkout resilience plan

Each dependency has a timeout, a retry rule, isolation and a failure behaviour.

dependency      timeout  retries          isolation       on failure
--------------  -------  ---------------  --------------  -------------------------------------
cart            300 ms   1, jitter        bulkhead 50     error: cannot check out without cart
pricing         300 ms   1, jitter        bulkhead 50     error (prices must be current)
inventory       300 ms   1, jitter        bulkhead 50     accept order, confirm stock async
fraud scoring   400 ms   0                breaker + bh 20 rules-based score for low-value orders
payments        800 ms   0 (idem. key)    breaker + bh 40 "payment pending", queue + reconcile
notifications   async    queue retries    separate queue  delayed email, never blocks checkout

edge: per-user + per-IP rate limits, priority shedding, 2 s deadline propagated

Money paths get idempotency, not blind retries

Never let a generic retry policy wrap a charge call. Use an idempotency key so any retry, by a client, a queue or a person, can only ever charge once.

Quick check: In the case study, why is the payments charge call not retried automatically by a generic policy?

• A blind retry could double-charge; idempotency keys and reconciliation make retries safe instead
• Payments never fail
• Retries are forbidden by HTTP
• The breaker handles all payment errors