# Testing Resilience with Fault Injection — Rate Limiting, Circuit Breakers & Resilience Patterns

Source: https://www.skillbyai.com/en/resilience-patterns/p-test

> Verify resilience patterns with fault injection and game days.

## Prove it before production does

Resilience code is rarely exercised in normal operation, so it hides bugs: a timeout set in the wrong unit, a fallback that throws, a breaker that never opens. **Fault injection** tests it deliberately. In **unit tests**, use fake clients that time out, raise errors or return 503 and assert the expected retries, fallbacks and breaker transitions. In **integration tests**, put a proxy such as **Toxiproxy** (by Shopify) between your service and a real dependency to add latency, cut connections or limit bandwidth. In **Kubernetes with Istio**, `VirtualService` fault injection adds delays or aborts for a percentage of requests. **Chaos engineering** experiments in staging or carefully in production (kill pods, slow a dependency, fail a zone) check whole-system behaviour against a hypothesis, with a small blast radius and an abort switch. **Game days** let teams practise incidents and verify runbooks and kill switches.

## Injecting faults between services

A fault-injecting proxy adds delay or errors so you can watch the caller's patterns respond.

![Two service boxes with a small box between them containing a lightning symbol and an hourglass.](assets/figures/resilience-patterns/section-8-map.svg) — Figure 8.1 — Fault injection between a service and its dependency.

## Injecting latency and errors with Istio

10% of requests get a 3-second delay and 5% get HTTP 503, for one test header only.

```yaml
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: recommendations-fault
spec:
  hosts: [recommendations]
  http:
    - match:
        - headers:
            x-chaos-test: { exact: "true" }
      fault:
        delay:
          percentage: { value: 10 }
          fixedDelay: 3s
        abort:
          percentage: { value: 5 }
          httpStatus: 503
      route:
        - destination: { host: recommendations }
    - route:
        - destination: { host: recommendations }
```

## Start with a hypothesis

"If recommendations add 3 s latency, product pages stay under 800 ms p99 and show popular items instead" is a testable hypothesis. "Let's break stuff and see" is not an experiment.

**Quiz:** What is Toxiproxy used for?

- [ ] Encrypting service traffic
- [ ] Load balancing production traffic
- [x] Simulating network faults such as latency and dropped connections between a service and its dependency
- [ ] Storing idempotency keys

*Answer:* Simulating network faults such as latency and dropped connections between a service and its dependency. Toxiproxy injects network conditions so resilience behaviour can be tested.