Lesson 4 / 25

Vertical vs Horizontal Scaling and Statelessness

Choose between scaling up and scaling out, and make services stateless so they can scale out.

Bigger machine or more machines

Vertical scaling (scale up) gives one machine more CPU, memory or faster disks. It is simple, needs no code change and often the cheapest first step, but it has a ceiling, larger machines cost disproportionately more, and one machine remains a single point of failure. Horizontal scaling (scale out) adds more machines behind a load balancer. It has no hard ceiling and improves availability, but requires the application to be stateless: any instance must be able to serve any request. Move state out of the process: sessions into a shared store such as Redis or into signed tokens, uploaded files into object storage, caches into a distributed cache, scheduled jobs into a queue or a single scheduler. Then instances become interchangeable and can be added, removed or replaced at any time.

Scale up versus scale out

One machine grows taller, or many identical machines stand side by side.

Figure 2.1 — Vertical scaling compared with horizontal scaling.

Where state hides in a web app

Each line is a reason two instances would behave differently.

state in the process                     move it to
---------------------------------------  -------------------------------------
login sessions in memory                 Redis / database / signed JWT cookies
uploaded files on local disk             object storage (S3, GCS, Azure Blob)
in-memory cache per instance             shared cache, or accept per-instance cache
cron job running on every instance       one scheduler + queue, or leader election
WebSocket connections                    sticky routing + pub/sub to fan out
rate-limit counters in memory            Redis or the API gateway

Scale up first, but design for out

A bigger database server buys months of growth for an afternoon of work. Do that when it is cheap, but keep application servers stateless from day one so scaling out is never a rewrite.

Quick check: Why must a service usually be stateless to scale horizontally?

Stateless code runs faster on one CPU
Load balancers reject stateful services
So any instance can serve any request, and instances can be added or removed freely
Stateless services do not need databases

Answer

So any instance can serve any request, and instances can be added or removed freely — If state lives in one instance, requests depend on reaching that particular instance.