Quantization: Fitting Big Models on Small GPUs

Fewer bits per weight

A model's weights are numbers stored in a data type. float32 uses 4 bytes per parameter, float16 and bfloat16 use 2, int8 uses 1 and 4-bit formats about half a byte. Memory for weights is therefore roughly parameters x bytes per parameter, before adding activations and the KV cache. Quantization stores weights with fewer bits so a large model fits on a smaller GPU, often with a modest quality drop. transformers integrates several methods. bitsandbytes quantizes on the fly at load time to 8-bit or 4-bit (NF4, a 4-bit format designed for normally distributed weights) and needs no calibration data. Methods such as GPTQ and AWQ quantize ahead of time using calibration data, and many pre-quantized checkpoints are published on the Hub. GGUF files belong to the llama.cpp ecosystem. Quantized weights mainly save memory; speed gains depend on kernels and hardware, and on some setups a quantized model runs slower than a bfloat16 one. Always re-check quality on your own evaluation set.

Back-of-the-envelope memory maths

Weights only, in GiB (1024^3 bytes), which is why 7e9 x 2 bytes = 14 GB shows as 13.0; leave headroom for activations and the KV cache during generation.

# Rough memory needed just to hold the weights
params = 7e9  # a 7-billion-parameter model
bytes_per_param = {"float32": 4, "bfloat16": 2, "int8": 1, "4-bit": 0.5}

for dtype, nbytes in bytes_per_param.items():
    gb = params * nbytes / 1024**3
    print(f"{dtype:>8}: {gb:5.1f} GiB")

Output:

 float32:  26.1 GiB
bfloat16:  13.0 GiB
    int8:   6.5 GiB
   4-bit:   3.3 GiB

Loading in 4-bit with bitsandbytes

Needs a CUDA GPU and the bitsandbytes package. Compute still happens in bfloat16; only storage is 4-bit.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model_id = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_id, quantization_config=bnb, device_map="auto")
tok = AutoTokenizer.from_pretrained(model_id)
print(model.get_memory_footprint() / 1024**3, "GB")

Pass a quantization_config, not loose flags

Older tutorials pass load_in_8bit=True directly to from_pretrained. Current versions expect a BitsAndBytesConfig via quantization_config; the loose flags are deprecated or removed.

Quick check: Roughly how much memory do the weights of a 7-billion-parameter model need in bfloat16?

• About 3.5 GB
• About 28 GB
• About 7 GB
• About 14 GB