# Production Pitfalls: Split Brain, Pre-Vote and Disruptive Nodes — Distributed Systems & Consensus (Raft / Paxos)

Source: https://www.skillbyai.com/en/distributed-consensus/dc-pitfalls

> Recognise split brain, disruptive rejoining servers and slow-disk elections, and apply pre-vote and CheckQuorum.

## Ways real clusters misbehave

**Split brain** means two nodes both acting as leader and accepting writes. Raft allows at most one leader per term, and a leader cut off from the majority cannot *commit* anything, but an old leader from an earlier term, isolated by a partition, still believes it leads and can serve stale reads or accept writes that will later fail. **CheckQuorum** makes a leader step down if it has not heard from a majority within an election timeout. A second problem is the **disruptive server**: a node partitioned away keeps timing out and incrementing its term; when it rejoins, its high term forces the healthy leader to step down, causing a needless election. **Pre-Vote** fixes this: before incrementing its term, a would-be candidate asks peers whether they *would* vote for it, and they refuse if they have heard from a current leader recently. Other classic causes of instability are election timeouts shorter than disk fsync or GC pauses, which trigger elections under load, and asymmetric network failures where a node can send but not receive.

## The returning traveller

A colleague returns from a month in the mountains with no signal and insists on re-electing the team lead because they have not heard from anyone. Pre-vote is checking with a few teammates first: 'Is our lead still around?' If yes, there is no need for an election.

## Tune timeouts to your environment

The heartbeat interval should be well below the election timeout, and the election timeout should be comfortably above the worst-case round trip and fsync time. etcd's tuning guide suggests a heartbeat interval around the average round-trip time between members and an election timeout of at least ten times that round-trip time (defaults: 100 ms heartbeat, 1000 ms election timeout).

**Quiz:** What problem does Raft's Pre-Vote extension solve?

- [x] A rejoining partitioned node with an inflated term forcing an unnecessary election
- [ ] Slow disks delaying fsync
- [ ] Logs growing without bound
- [ ] Clients reading stale data from followers

*Answer:* A rejoining partitioned node with an inflated term forcing an unnecessary election. Pre-vote lets a node check whether it could win before incrementing its term, so an isolated node cannot disrupt a healthy leader when it reconnects.