The Distributed Systems Tax

What you pay for the network

Splitting a process into services turns function calls into network calls, and the network is unreliable. The classic fallacies of distributed computing (attributed to L. Peter Deutsch and others at Sun Microsystems) are false assumptions: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology does not change; there is one administrator; transport cost is zero; the network is homogeneous. Concrete costs follow. Latency adds up across call chains. Partial failures need timeouts, retries and circuit breakers. Data consistency across services needs sagas and eventual consistency instead of transactions. Testing and debugging span many services, requiring contract tests and distributed tracing. Operations multiply: many pipelines, deployments, dashboards, alerts and on-call rotations. Infrastructure costs more: more instances, a gateway, a mesh, brokers and observability tools.

Latency adds up in a call chain

Five sequential calls at a modest p99 add up quickly.

chain_p50_ms = [3, 4, 5, 3, 6]          # each hop, typical
chain_p99_ms = [25, 40, 60, 30, 80]     # each hop, slow case

print("typical", sum(chain_p50_ms), "ms")    # same work in-process: well under 1 ms of call overhead
print("worst-ish", sum(chain_p99_ms), "ms")

# chance that at least one of five hops is at its p99 or slower
print(round(1 - 0.99 ** 5, 3))            # about 5% of requests

Phone calls instead of shouting across the room

In a small office you shout a question across the room and get an answer instantly. Move everyone to different buildings and every question becomes a phone call: sometimes busy, sometimes dropped, always slower, and someone has to pay the phone bill.

त्वरित जाँच: Which is one of the fallacies of distributed computing?

• Databases are always relational
• The network is reliable
• Monoliths cannot use caching
• Containers are secure by default