Performance: Latency, Batching and Multi-Raft

Where the time goes

A consensus write costs at least one round trip from the leader to a majority plus a durable disk write (fsync) on each of those servers, plus the client's own round trip to the leader. So latency is governed by the follower whose acknowledgement completes the majority (in a 5-node cluster, the second-fastest of the four followers), not the slowest one, and by disk sync speed. Placement therefore matters: a 5-node cluster spread across far-apart regions pays cross-region latency on every write. Throughput is improved by batching (many commands per AppendEntries and per fsync), pipelining (sending the next AppendEntries before the previous is acknowledged) and doing disk writes in parallel with network sends. A single Raft group is still limited by one leader, so large databases such as CockroachDB, TiKV and YugabyteDB use Multi-Raft: data is split into ranges or shards, each with its own Raft group and leader, spread across machines. Cross-shard transactions then need an extra protocol, typically a form of 2PC layered over the Raft groups.

Running consensus at scale

Measure commit latency, batch and pipeline writes, shard into many Raft groups, and harden against pitfalls with pre-vote, CheckQuorum and testing.

Commit latency is set by the majority, not the slowest node

Given each follower's round-trip time from the leader, the commit waits for enough followers to form a majority with the leader. One very slow peer does not slow a 5-node cluster; spreading every node across regions does.

def commit_latency(follower_rtts_ms):
    """Leader needs acks from a majority (itself included)."""
    n = len(follower_rtts_ms) + 1
    acks_needed = n // 2               # followers needed besides the leader
    return sorted(follower_rtts_ms)[acks_needed - 1]

print("3 nodes, one region   :", commit_latency([1, 2]), "ms")
print("5 nodes, 5 regions    :", commit_latency([30, 40, 45, 80]), "ms")
print("5 nodes, one slow peer:", commit_latency([1, 2, 3, 900]), "ms")

Output:

3 nodes, one region   : 1 ms
5 nodes, 5 regions    : 40 ms
5 nodes, one slow peer: 2 ms

More nodes means slower writes

Going from 3 to 5 voters improves fault tolerance but makes each commit wait for one more acknowledgement and adds replication traffic on the leader. If you need more read capacity, add non-voting learners or followers serving ReadIndex reads, not more voters.

त्वरित जाँच: A 5-node cluster's leader has follower RTTs of 2, 3, 70 and 90 ms. Roughly what replication latency does each commit add?

• 90 ms
• 2 ms
• 3 ms
• 70 ms