# Fallbacks, Timeouts and Circuit Breakers — Safe Rollout Plans for AI Features

Source: https://www.skillbyai.com/en/ai-feature-rollouts/r-fallback

> Keep working when the model does not.

## Graceful degradation

Model providers have outages, rate limits and slow periods. Give every AI call a **timeout** and a **fallback**: a simpler non-AI experience, a cached answer, or a smaller backup model. A **circuit breaker** stops calling a failing provider after repeated errors, serving the fallback immediately for a cooldown period instead of making every user wait for timeouts, then tries again. The product should stay usable with the AI part switched off.

## A circuit breaker around an LLM call, run

I ran this with Python 3 (scipy 1.18.1 where imported) on example numbers, not data from a real product. After three consecutive timeouts the breaker opens: at 10 seconds the provider is healthy again, but the breaker still serves the fallback during its 30-second cooldown. At 40 seconds it tries again and the AI summary returns.

```python
import time
class Breaker:
    def __init__(self, threshold=3, cooldown=30):
        self.fail, self.threshold, self.cooldown, self.opened = 0, threshold, cooldown, None
    def call(self, fn, fallback, now):
        if self.opened is not None and now - self.opened < self.cooldown:
            return fallback(), "fallback (breaker open)"
        try:
            out = fn(); self.fail = 0; self.opened = None; return out, "llm"
        except Exception as e:
            self.fail += 1
            if self.fail >= self.threshold: self.opened = now
            return fallback(), f"fallback ({type(e).__name__})"
outcomes = [True, False, False, False, True, True, True]      # provider ok / timeout per request
def make_llm(ok):
    def llm():
        if not ok: raise TimeoutError
        return "AI summary"
    return llm
b = Breaker()
for t, ok in zip([0, 1, 2, 3, 10, 40, 41], outcomes):
    result, path = b.call(make_llm(ok), lambda: "plain list of recent events", now=t)
    print(f"t={t:>2}s provider {'ok     ' if ok else 'timeout'} -> {path:<26} {result}")
```

Output:

```
t= 0s provider ok      -> llm                        AI summary
t= 1s provider timeout -> fallback (TimeoutError)    plain list of recent events
t= 2s provider timeout -> fallback (TimeoutError)    plain list of recent events
t= 3s provider timeout -> fallback (TimeoutError)    plain list of recent events
t=10s provider ok      -> fallback (breaker open)    plain list of recent events
t=40s provider ok      -> llm                        AI summary
t=41s provider ok      -> llm                        AI summary
```

## Design the fallback first

Decide what users see when AI is unavailable before building the AI path; it is also your kill-switch experience.

**Quiz:** Why use a circuit breaker for model calls?

- [x] To stop sending requests to a failing provider and serve a fallback quickly
- [ ] To make the model smarter
- [ ] To increase costs
- [ ] To skip timeouts entirely

*Answer:* To stop sending requests to a failing provider and serve a fallback quickly. Fail fast, recover automatically.