The GGUF File Format

One file with weights and metadata

GGUF is the file format used by llama.cpp and its underlying tensor library, ggml; Ollama also runs GGUF models. A GGUF file is a single binary containing the model's tensors (the weights, possibly quantized) together with metadata: the architecture, number of layers, context length it was trained for, the tokenizer vocabulary, special tokens, and usually the chat template. Bundling all of this in one file means a runtime can load a model without the separate config and tokenizer files that Hugging Face checkpoints use. The format is designed to be memory-mapped: the runtime can map the file into its address space and let the OS page tensors in as needed, which makes loading fast. Models published in other formats, such as safetensors from Hugging Face, are converted to GGUF with the conversion script in the llama.cpp repository and then optionally quantized.

From training checkpoint to runnable file

A typical model travels through these steps before you can run it locally. The stages, in order: Hugging Face safetensors checkpoint -> convert to GGUF (F16) -> quantize (for example Q4_K_M) -> load in llama.cpp or Ollama.

Converting and quantizing with llama.cpp

Run from a llama.cpp checkout. The first step writes an F16 GGUF, the second makes a 4-bit copy.

# 1. Convert a downloaded Hugging Face model folder to GGUF
python convert_hf_to_gguf.py ./my-model-hf --outfile my-model-f16.gguf --outtype f16

# 2. Quantize the F16 file to 4-bit (K-quant, medium)
./build/bin/llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_M

You rarely need to convert yourself

Popular models are already published as GGUF on Hugging Face in many quantization levels, and Ollama's library serves ready-made ones. Convert only when a model is new, private or fine-tuned by you.

त्वरित जाँच: Besides the weights, what does a GGUF file typically contain?

• The original training dataset
• Metadata such as architecture, tokenizer and chat template
• A Python virtual environment
• Compiled CUDA kernels for every GPU