# Testing and Verifying Consensus Systems — Distributed Systems & Consensus (Raft / Paxos)

Source: https://www.skillbyai.com/en/distributed-consensus/dc-testing

> Choose between deterministic simulation, fault injection with Jepsen and formal specification with TLA+.

## Bugs live in rare interleavings

Consensus bugs hide in rare orderings of crashes, delays and message loss that unit tests never hit. Three complementary techniques are used. **Formal specification**: write the protocol in a language such as **TLA+** and let a model checker (TLC) explore every interleaving of a small configuration; the Raft paper's authors published a TLA+ spec, and many teams use TLA+ for design review. **Deterministic simulation testing**: run the whole cluster in one process with a simulated network, disk and clock driven by a seeded random generator, inject faults aggressively, and when an invariant fails, replay the exact seed to reproduce the bug; FoundationDB popularised this approach. **Black-box fault injection**: tools like **Jepsen** run a real cluster, partition networks, kill processes and skew clocks while clients run operations, then check the recorded history for linearizability violations. Jepsen analyses have found consistency bugs in many well-known databases.

## Determinism makes failures reproducible

A seeded generator decides which messages are dropped. Each seed is a different but exactly repeatable universe of failures.

```python
import random

def run(seed, drop_rate=0.3, rounds=20):
    rng = random.Random(seed)            # the seed fixes every "random" event
    delivered = 0
    for _ in range(rounds):
        if rng.random() >= drop_rate:
            delivered += 1
    return delivered

print([run(seed) for seed in (1, 2, 3)])
print("replay seed 2:", run(2), "(identical every time)")
```

Output:

```
[13, 16, 14]
replay seed 2: 16 (identical every time)
```

## Test the invariants, not the happy path

Assert properties that must always hold: at most one leader per term, committed entries never change, every replica applies the same sequence. Checking these after every simulated step finds bugs far faster than checking final outputs.

**Quiz:** What is the main advantage of deterministic simulation testing over running a real cluster with random faults?

- [ ] It needs no test assertions
- [x] Any failure can be reproduced exactly by replaying the same seed
- [ ] It proves the protocol correct for all cluster sizes
- [ ] It measures production latency accurately

*Answer:* Any failure can be reproduced exactly by replaying the same seed. Because every random choice comes from the seed, a failing run replays identically, which turns rare heisenbugs into debuggable ones.