GPU Offload and CPU/GPU Splits

Layers go where they fit

A transformer is a stack of repeated layers. Runtimes can place some layers in GPU memory and the rest in system RAM, computing each layer where its weights live. Ollama estimates this automatically from free VRAM; llama.cpp lets you choose with -ngl. Full offload is fastest. Partial offload works but the speed falls sharply, because the CPU-side layers run at RAM bandwidth and become the bottleneck for every token; a model with 70% of layers on GPU is often closer to CPU speed than to GPU speed. On Apple Silicon, unified memory means the GPU can use most system RAM, so larger models fit without splitting. With several GPUs, llama.cpp can split layers across them (--tensor-split). The practical rule: prefer a smaller model or lower quant that fits entirely on the GPU over a bigger one that spills, unless you need the bigger model's quality more than speed.

Measuring the effect of offload

llama-bench reports prompt-processing (pp) and token-generation (tg) speed for each setting.

# Compare 0, 16 and all layers on GPU; pp512 = prefill speed, tg128 = decode speed
./build/bin/llama-bench -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -ngl 0,16,99

# In Ollama, check the split instead
ollama ps

Integrated GPUs and drivers

If ollama ps shows 100% CPU on a machine with an NVIDIA GPU, check that the driver is installed and visible (nvidia-smi) and read the server log; Ollama falls back to CPU silently when it cannot use the GPU. Integrated laptop GPUs may help only a little, since they share the same RAM bandwidth as the CPU.

त्वरित जाँच: Why does partially offloading a model to the GPU often give disappointing speed?

• The GPU refuses to compute quantized layers
• Layers left in system RAM run at much lower bandwidth and bottleneck every token
• Partial offload disables the KV cache
• Ollama re-downloads the model each time