Production Pitfalls: Split Brain, Pre-Vote and Disruptive Nodes

Ways real clusters misbehave

Split brain means two nodes both acting as leader and accepting writes. Raft allows at most one leader per term, and a leader cut off from the majority cannot commit anything, but an old leader from an earlier term, isolated by a partition, still believes it leads and can serve stale reads or accept writes that will later fail. CheckQuorum makes a leader step down if it has not heard from a majority within an election timeout. A second problem is the disruptive server: a node partitioned away keeps timing out and incrementing its term; when it rejoins, its high term forces the healthy leader to step down, causing a needless election. Pre-Vote fixes this: before incrementing its term, a would-be candidate asks peers whether they would vote for it, and they refuse if they have heard from a current leader recently. Other classic causes of instability are election timeouts shorter than disk fsync or GC pauses, which trigger elections under load, and asymmetric network failures where a node can send but not receive.

The returning traveller

A colleague returns from a month in the mountains with no signal and insists on re-electing the team lead because they have not heard from anyone. Pre-vote is checking with a few teammates first: 'Is our lead still around?' If yes, there is no need for an election.

Tune timeouts to your environment

The heartbeat interval should be well below the election timeout, and the election timeout should be comfortably above the worst-case round trip and fsync time. etcd's tuning guide suggests a heartbeat interval around the average round-trip time between members and an election timeout of at least ten times that round-trip time (defaults: 100 ms heartbeat, 1000 ms election timeout).

Quick check: What problem does Raft's Pre-Vote extension solve?

• A rejoining partitioned node with an inflated term forcing an unnecessary election
• Slow disks delaying fsync
• Logs growing without bound
• Clients reading stale data from followers