Testing Resilience with Fault Injection

Prove it before production does

Resilience code is rarely exercised in normal operation, so it hides bugs: a timeout set in the wrong unit, a fallback that throws, a breaker that never opens. Fault injection tests it deliberately. In unit tests, use fake clients that time out, raise errors or return 503 and assert the expected retries, fallbacks and breaker transitions. In integration tests, put a proxy such as Toxiproxy (by Shopify) between your service and a real dependency to add latency, cut connections or limit bandwidth. In Kubernetes with Istio, VirtualService fault injection adds delays or aborts for a percentage of requests. Chaos engineering experiments in staging or carefully in production (kill pods, slow a dependency, fail a zone) check whole-system behaviour against a hypothesis, with a small blast radius and an abort switch. Game days let teams practise incidents and verify runbooks and kill switches.

Injecting faults between services

A fault-injecting proxy adds delay or errors so you can watch the caller's patterns respond.

Injecting latency and errors with Istio

10% of requests get a 3-second delay and 5% get HTTP 503, for one test header only.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: recommendations-fault
spec:
  hosts: [recommendations]
  http:
    - match:
        - headers:
            x-chaos-test: { exact: "true" }
      fault:
        delay:
          percentage: { value: 10 }
          fixedDelay: 3s
        abort:
          percentage: { value: 5 }
          httpStatus: 503
      route:
        - destination: { host: recommendations }
    - route:
        - destination: { host: recommendations }

Start with a hypothesis

"If recommendations add 3 s latency, product pages stay under 800 ms p99 and show popular items instead" is a testable hypothesis. "Let's break stuff and see" is not an experiment.

Quick check: What is Toxiproxy used for?

• Encrypting service traffic
• Load balancing production traffic
• Simulating network faults such as latency and dropped connections between a service and its dependency
• Storing idempotency keys