Snapshots and Log Compaction

The log cannot grow forever

A replicated log grows with every command, consuming disk and making restarts slow because a restarted server must replay everything. Snapshotting fixes this: each server periodically writes its current state machine state to stable storage along with the lastIncludedIndex and lastIncludedTerm of the last entry it covers, and then discards all log entries up to that index. Servers snapshot independently, based on log size, because the snapshot only covers committed, applied entries, so no coordination is needed. Occasionally a follower falls so far behind that the entries it needs have already been discarded on the leader. The leader then sends an InstallSnapshot RPC transferring its snapshot in chunks; the follower replaces its state and keeps any log entries after the snapshot that still match. Snapshots must be taken without stopping the world for long: implementations use copy-on-write data structures or forked processes so that the state machine keeps serving while the snapshot is written.

The bank passbook

Rather than keep every transaction since you opened the account, the bank prints a balance carried forward on a new page. A new branch helping you can start from that balance instead of replaying ten years of entries.

Snapshot tuning

Snapshot too often and you waste I/O; too rarely and the log and restart time grow. Systems such as etcd expose a threshold of applied entries between snapshots and also need periodic defragmentation of the backing store to give disk space back after old key revisions are compacted (etcdctl compact), which is separate from Raft log snapshots.

Quick check: When does a Raft leader send InstallSnapshot to a follower?

• After every committed entry
• When the follower needs entries the leader has already discarded during compaction
• Whenever a new leader is elected
• When the follower's log is longer than the leader's