Fallbacks, Timeouts and Circuit Breakers

Graceful degradation

Model providers have outages, rate limits and slow periods. Give every AI call a timeout and a fallback: a simpler non-AI experience, a cached answer, or a smaller backup model. A circuit breaker stops calling a failing provider after repeated errors, serving the fallback immediately for a cooldown period instead of making every user wait for timeouts, then tries again. The product should stay usable with the AI part switched off.

A circuit breaker around an LLM call, run

I ran this with Python 3 (scipy 1.18.1 where imported) on example numbers, not data from a real product. After three consecutive timeouts the breaker opens: at 10 seconds the provider is healthy again, but the breaker still serves the fallback during its 30-second cooldown. At 40 seconds it tries again and the AI summary returns.

import time
class Breaker:
    def __init__(self, threshold=3, cooldown=30):
        self.fail, self.threshold, self.cooldown, self.opened = 0, threshold, cooldown, None
    def call(self, fn, fallback, now):
        if self.opened is not None and now - self.opened < self.cooldown:
            return fallback(), "fallback (breaker open)"
        try:
            out = fn(); self.fail = 0; self.opened = None; return out, "llm"
        except Exception as e:
            self.fail += 1
            if self.fail >= self.threshold: self.opened = now
            return fallback(), f"fallback ({type(e).__name__})"
outcomes = [True, False, False, False, True, True, True]      # provider ok / timeout per request
def make_llm(ok):
    def llm():
        if not ok: raise TimeoutError
        return "AI summary"
    return llm
b = Breaker()
for t, ok in zip([0, 1, 2, 3, 10, 40, 41], outcomes):
    result, path = b.call(make_llm(ok), lambda: "plain list of recent events", now=t)
    print(f"t={t:>2}s provider {'ok     ' if ok else 'timeout'} -> {path:<26} {result}")

Output:

t= 0s provider ok      -> llm                        AI summary
t= 1s provider timeout -> fallback (TimeoutError)    plain list of recent events
t= 2s provider timeout -> fallback (TimeoutError)    plain list of recent events
t= 3s provider timeout -> fallback (TimeoutError)    plain list of recent events
t=10s provider ok      -> fallback (breaker open)    plain list of recent events
t=40s provider ok      -> llm                        AI summary
t=41s provider ok      -> llm                        AI summary

Design the fallback first

Decide what users see when AI is unavailable before building the AI path; it is also your kill-switch experience.

त्वरित जाँच: Why use a circuit breaker for model calls?

• To stop sending requests to a failing provider and serve a fallback quickly
• To make the model smarter
• To increase costs
• To skip timeouts entirely