Lesson 21 / 33
Cluster Membership Changes
Explain why switching configurations atomically is unsafe and how joint consensus and single-server changes avoid two leaders.
By Rahul Agarwal · 15 years in web development
Changing who votes, safely
Replacing a failed machine or growing from 3 to 5 nodes changes the configuration: the set of servers whose votes count. Switching every server from the old configuration to the new one at the same instant is impossible, and during the transition a dangerous moment can occur where a majority of the old configuration and a disjoint majority of the new one each elect a leader in the same term. Raft offers two safe approaches. Joint consensus goes through an intermediate configuration C(old,new) in which elections and commits need separate majorities of both old and new; once that entry commits, the leader moves to C(new). The simpler, widely used alternative is single-server changes: add or remove one server at a time. Any majority of the old set and any majority of the new set then always overlap, so two leaders cannot coexist. Configuration entries are stored in the log, and a server uses the latest configuration in its log, even before it is committed.
Raft in production
Running Raft for real adds membership changes, log compaction with snapshots, linearizable reads, and the client-facing service on top.
Replacing a member in etcd
Removing a dead member and adding its replacement one at a time with etcdctl. The new node starts with --initial-cluster-state existing. Member IDs and URLs are placeholders.
# 1. Find the failed member's ID
etcdctl member list -w table
# 2. Remove it (one change at a time)
etcdctl member remove 8e9e05c52164694d
# 3. Register the replacement, optionally as a non-voting learner first
etcdctl member add node4 --peer-urls=https://10.0.0.14:2380 --learner
# 4. Once it has caught up, promote it to a voting member
etcdctl member promote <new-member-id>Add catch-up learners first
A brand-new server with an empty log counts towards the quorum immediately if added as a voter, and it may take a long time to catch up. Adding it as a non-voting learner first (as etcd supports) avoids reducing availability during the catch-up.
Quick check: Why does Raft recommend adding or removing only one server at a time?
- Any majority of the old configuration overlaps any majority of the new one, so two leaders cannot be elected
- It reduces the size of snapshots
- It avoids the need for leader election entirely
- It lets followers skip the consistency check
Answer
Any majority of the old configuration overlaps any majority of the new one, so two leaders cannot be elected — With a single-server change, old and new majorities always share a server, which prevents disjoint majorities from electing separate leaders.