# Quantization Levels and Trade-offs — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-quantization

> Choose a quantization level such as Q4_K_M or Q8_0 by weighing size, speed and quality.

## Fewer bits per weight

Models are trained in 16-bit (or higher) floating point. **Quantization** stores each weight in fewer bits, typically 8, 6, 5, 4 or even 2 to 3, plus small per-block **scale factors** so values can be approximately reconstructed during computation. llama.cpp names describe this: `Q8_0` is about 8 bits per weight and is close to lossless; `Q4_K_M` uses the **K-quant** scheme at about 4 to 5 bits average, with the *M* (medium) mix keeping some sensitive tensors at higher precision; `S` and `L` variants are smaller and larger mixes. Lower bits mean a smaller file, less memory and faster decode, but more **quality loss**, measured as higher perplexity or worse task scores. The loss is modest down to around 4 bits for most models and grows quickly below that. Larger models tolerate quantization better than small ones, so a 4-bit 8B model often beats an 8-bit 3B model in a similar memory budget.

## Size of an 8B model at different levels

Bits per weight here are approximate averages, including block scales.

```python
def weights_gb(params_billion, bits_per_weight):
    return params_billion * 1e9 * bits_per_weight / 8 / 1e9

for name, bits in [("F16", 16), ("Q8_0", 8.5), ("Q4_K_M", 4.85)]:
    print(f"8B model at {name:7}: {weights_gb(8, bits):5.1f} GB")
```

Output:

```
8B model at F16    :  16.0 GB
8B model at Q8_0   :   8.5 GB
8B model at Q4_K_M :   4.8 GB
```

## Compressing a photo for WhatsApp

A 12 MB camera photo sent over WhatsApp arrives as a much smaller file that still looks fine on a phone. Zoom in and you notice artefacts. Quantization is the same deal for weights: Q8 is the nearly perfect copy, Q4 the good-enough share, and Q2 the blurry forward from a family group.

## A sensible default

Q4_K_M is the common starting point: good quality per GB. Move up to Q5_K_M, Q6_K or Q8_0 if you have spare memory and notice errors; move down only when the model will not otherwise fit. **Importance-matrix** (imatrix) quants use calibration data to choose what to preserve and help most at very low bit widths.

**Quiz:** What does the "_M" in Q4_K_M indicate?

- [ ] The model was trained on medical data
- [ ] It only runs on Mac
- [x] A medium mix that keeps some sensitive tensors at higher precision
- [ ] It uses mixed CPU and GPU automatically

*Answer:* A medium mix that keeps some sensitive tensors at higher precision. K-quant variants come in S, M and L mixes. M is a medium mix where certain tensors get more bits, balancing size and quality.