# Hardware Basics: RAM, VRAM and Bandwidth — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-hardware-basics

> Estimate whether a model will fit on your machine and predict roughly how fast it will run.

## Memory decides what fits, bandwidth decides speed

To run a model, its weights must sit in memory: **VRAM** on a discrete GPU, ordinary **RAM** for the CPU, or **unified memory** on Apple Silicon, which the CPU and GPU share. A rough rule: weight size in bytes is parameters times bits per weight divided by eight, so an 8-billion-parameter model at 16 bits needs about 16 GB, while a 4-bit version needs about 5 GB. Add room for the KV cache and runtime overhead. Speed is a different question. During decode, every new token requires reading essentially all the weights once, so **memory bandwidth** (GB per second) sets an upper bound on tokens per second. That is why a GPU with fast VRAM beats a CPU on the same model, and why a smaller or more heavily quantized model generates faster: there are fewer bytes to read per token. If a model does not fit in VRAM, runtimes can split layers between GPU and CPU, at a speed cost.

## A back-of-the-envelope speed ceiling

Divide bandwidth by model size to get the most tokens per second you could hope for. Real numbers come in lower.

```python
# Decode speed upper bound: every new token reads all weights once
def max_tokens_per_sec(model_gb, bandwidth_gb_s):
    return bandwidth_gb_s / model_gb

for device, bw in [("laptop DDR5 (~80 GB/s)", 80), ("mid GPU (~450 GB/s)", 450)]:
    print(f"{device}: ~{max_tokens_per_sec(4.9, bw):.0f} tok/s ceiling for a 4.9 GB model")
```

Output:

```
laptop DDR5 (~80 GB/s): ~16 tok/s ceiling for a 4.9 GB model
mid GPU (~450 GB/s): ~92 tok/s ceiling for a 4.9 GB model
```

## Start small on a student laptop

On an 8 GB RAM laptop with no GPU, start with a 1B to 3B parameter model at 4-bit. On 16 GB, 7B to 8B models at 4-bit are comfortable. Close the browser tabs first: the model competes with everything else for RAM, and once the OS starts swapping to disk, generation slows to a crawl.

**Quiz:** Why does a 4-bit quantized model usually generate tokens faster than the same model at 16-bit on the same hardware?

- [x] Fewer bytes must be read from memory for each generated token
- [ ] 4-bit models have fewer layers
- [ ] 4-bit models skip the tokenizer
- [ ] The GPU clock speed increases for smaller files

*Answer:* Fewer bytes must be read from memory for each generated token. Decode is mostly memory-bandwidth bound. A 4-bit model has the same architecture but roughly a quarter of the bytes to stream per token.