Hardware Basics: RAM, VRAM and Bandwidth

Memory decides what fits, bandwidth decides speed

To run a model, its weights must sit in memory: VRAM on a discrete GPU, ordinary RAM for the CPU, or unified memory on Apple Silicon, which the CPU and GPU share. A rough rule: weight size in bytes is parameters times bits per weight divided by eight, so an 8-billion-parameter model at 16 bits needs about 16 GB, while a 4-bit version needs about 5 GB. Add room for the KV cache and runtime overhead. Speed is a different question. During decode, every new token requires reading essentially all the weights once, so memory bandwidth (GB per second) sets an upper bound on tokens per second. That is why a GPU with fast VRAM beats a CPU on the same model, and why a smaller or more heavily quantized model generates faster: there are fewer bytes to read per token. If a model does not fit in VRAM, runtimes can split layers between GPU and CPU, at a speed cost.

A back-of-the-envelope speed ceiling

Divide bandwidth by model size to get the most tokens per second you could hope for. Real numbers come in lower.

# Decode speed upper bound: every new token reads all weights once
def max_tokens_per_sec(model_gb, bandwidth_gb_s):
    return bandwidth_gb_s / model_gb

for device, bw in [("laptop DDR5 (~80 GB/s)", 80), ("mid GPU (~450 GB/s)", 450)]:
    print(f"{device}: ~{max_tokens_per_sec(4.9, bw):.0f} tok/s ceiling for a 4.9 GB model")

Output:

laptop DDR5 (~80 GB/s): ~16 tok/s ceiling for a 4.9 GB model
mid GPU (~450 GB/s): ~92 tok/s ceiling for a 4.9 GB model

Start small on a student laptop

On an 8 GB RAM laptop with no GPU, start with a 1B to 3B parameter model at 4-bit. On 16 GB, 7B to 8B models at 4-bit are comfortable. Close the browser tabs first: the model competes with everything else for RAM, and once the OS starts swapping to disk, generation slows to a crawl.

Quick check: Why does a 4-bit quantized model usually generate tokens faster than the same model at 16-bit on the same hardware?

• Fewer bytes must be read from memory for each generated token
• 4-bit models have fewer layers
• 4-bit models skip the tokenizer
• The GPU clock speed increases for smaller files