Retry Amplification, Retry Budgets and Hedging

Retries multiply across layers

If a mobile app retries 3 times, the gateway retries 3 times and a service retries 3 times, one user action can turn into 27 calls to the deepest dependency, exactly when it is failing. This retry amplification turns brief blips into outages. Defences: retry at one layer only, usually the one closest to the failing dependency, and make other layers fail fast; use a retry budget, which limits retries to a percentage of normal traffic (for example, retries may add at most 10% extra load) so they stop automatically during widespread failure; and combine retries with circuit breakers, which stop retries once a dependency is clearly down. Hedged requests are a related tool for tail latency: if a response has not arrived by roughly the p95 time, send a second request to another replica and use whichever answers first. Hedge only idempotent reads, and cap the extra load.

A simple retry budget

Retries are allowed only while they stay below 10% of recent requests.

import time
from collections import deque

class RetryBudget:
    def __init__(self, ratio=0.1, window_s=10, min_per_sec=10):
        self.ratio, self.window, self.min_per_sec = ratio, window_s, min_per_sec
        self.requests, self.retries = deque(), deque()

    def _trim(self, q, now):
        while q and now - q[0] > self.window:
            q.popleft()

    def record_request(self):
        self.requests.append(time.monotonic())

    def can_retry(self):
        now = time.monotonic()
        self._trim(self.requests, now); self._trim(self.retries, now)
        allowed = self.ratio * len(self.requests) + self.min_per_sec * self.window
        if len(self.retries) < allowed:
            self.retries.append(now)
            return True
        return False

Count total attempts in traces

Add the attempt number to spans and logs. During an incident, a graph of attempts per request quickly shows whether retries are amplifying load.

Quick check: Three layers each retry 3 times. How many calls can one user request cause at the deepest dependency in the worst case?

• 27
• 3
• 9
• 81