Client-Side Throttling and Cooperative Clients

Clients that help the server recover

Even rejected requests cost the server something: connections, parsing, authentication. If clients keep sending at full speed while being rejected, the server spends its capacity saying no. Client-side throttling lets clients reduce their own traffic when they see many rejections. Google's SRE book describes adaptive throttling: each client tracks recent requests and accepts, and rejects a new request locally with probability max(0, (requests − K × accepts) / (requests + 1)), with K typically 2. When the backend accepts everything, almost nothing is dropped locally; as rejections grow, the client sheds more traffic itself. Other cooperative behaviours: honour Retry-After; use exponential backoff with jitter on reconnects so recovering services are not hit by a synchronised herd; respect circuit breakers; and add startup jitter so thousands of devices or pods do not all connect at the same second after an outage.

Adaptive client-side throttling

Clients drop requests locally in proportion to how many the backend has been rejecting.

import random
from collections import deque
import time

class AdaptiveThrottle:
    def __init__(self, k=2.0, window_s=120):
        self.k, self.window = k, window_s
        self.events = deque()            # (timestamp, accepted: bool)

    def _counts(self):
        now = time.monotonic()
        while self.events and now - self.events[0][0] > self.window:
            self.events.popleft()
        requests = len(self.events)
        accepts = sum(1 for _, ok in self.events if ok)
        return requests, accepts

    def should_send(self):
        requests, accepts = self._counts()
        p_reject = max(0.0, (requests - self.k * accepts) / (requests + 1))
        return random.random() >= p_reject

    def record(self, accepted):
        self.events.append((time.monotonic(), accepted))

Add jitter to reconnect storms

After an outage, every mobile app and pod tries to reconnect. Randomised delays spread the reconnects over a minute instead of one second, which can be the difference between recovery and a second outage.

Quick check: In adaptive client-side throttling, what happens when the backend accepts nearly all requests?

• The local rejection probability stays near zero
• The client drops half its traffic anyway
• The client stops sending entirely
• The client doubles its traffic