# Cluster Membership Changes — Distributed Systems & Consensus (Raft / Paxos)

Source: https://www.skillbyai.com/en/distributed-consensus/dc-raft-membership

> Explain why switching configurations atomically is unsafe and how joint consensus and single-server changes avoid two leaders.

## Changing who votes, safely

Replacing a failed machine or growing from 3 to 5 nodes changes the **configuration**: the set of servers whose votes count. Switching every server from the old configuration to the new one at the same instant is impossible, and during the transition a dangerous moment can occur where a majority of the old configuration and a disjoint majority of the new one each elect a leader in the same term. Raft offers two safe approaches. **Joint consensus** goes through an intermediate configuration C(old,new) in which elections and commits need separate majorities of **both** old and new; once that entry commits, the leader moves to C(new). The simpler, widely used alternative is **single-server changes**: add or remove one server at a time. Any majority of the old set and any majority of the new set then always overlap, so two leaders cannot coexist. Configuration entries are stored in the log, and a server uses the latest configuration in its log, even before it is committed.

## Raft in production

Running Raft for real adds membership changes, log compaction with snapshots, linearizable reads, and the client-facing service on top.

![Four stages: membership changes, snapshots and compaction, linearizable reads, and the replicated service.](assets/figures/distributed-consensus/section-8-map.svg) — Figure 8.1: membership changes -> snapshots -> linearizable reads -> replicated service

## Replacing a member in etcd

Removing a dead member and adding its replacement one at a time with etcdctl. The new node starts with `--initial-cluster-state existing`. Member IDs and URLs are placeholders.

```bash
# 1. Find the failed member's ID
etcdctl member list -w table

# 2. Remove it (one change at a time)
etcdctl member remove 8e9e05c52164694d

# 3. Register the replacement, optionally as a non-voting learner first
etcdctl member add node4 --peer-urls=https://10.0.0.14:2380 --learner

# 4. Once it has caught up, promote it to a voting member
etcdctl member promote <new-member-id>
```

## Add catch-up learners first

A brand-new server with an empty log counts towards the quorum immediately if added as a voter, and it may take a long time to catch up. Adding it as a **non-voting learner** first (as etcd supports) avoids reducing availability during the catch-up.

**Quiz:** Why does Raft recommend adding or removing only one server at a time?

- [x] Any majority of the old configuration overlaps any majority of the new one, so two leaders cannot be elected
- [ ] It reduces the size of snapshots
- [ ] It avoids the need for leader election entirely
- [ ] It lets followers skip the consistency check

*Answer:* Any majority of the old configuration overlaps any majority of the new one, so two leaders cannot be elected. With a single-server change, old and new majorities always share a server, which prevents disjoint majorities from electing separate leaders.