QLoRA: Fine-Tuning on One Consumer GPU

Quantized base plus LoRA

QLoRA combines the two previous ideas. The frozen base model is loaded in 4-bit NF4, which slashes memory, and LoRA adapters are trained on top in higher precision. Gradients flow through the quantized weights into the adapters, but the base weights never change. This lets you fine-tune a model in the 7B to 8B range on a single GPU with roughly 16 to 24 GB of memory, which was impossible with full fine-tuning. Before training, prepare_model_for_kbit_training from peft adjusts a few things for stable training, such as keeping certain layers in higher precision and turning on gradient checkpointing by default. Gradient checkpointing trades extra compute for memory by recomputing activations during the backward pass instead of storing them. After training, you can keep the adapter separate, or load the base in bfloat16 and call merge_and_unload() to bake the adapter into a normal model for serving. Merging into a 4-bit base directly is lossy, so merge into a higher-precision copy of the base.

A QLoRA setup in outline

Continues from the 4-bit model above. Training then proceeds with Trainer or TRL's SFTTrainer.

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)   # model loaded with 4-bit config
# (this also enables gradient checkpointing unless use_gradient_checkpointing=False)

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules="all-linear", task_type="CAUSAL_LM")
model = get_peft_model(model, lora)
model.print_trainable_parameters()

# ... train ..., then save only the adapter
model.save_pretrained("qwen7b-support-adapter")

Merging the adapter for serving

Load the base in bfloat16, attach the adapter, merge, save a normal checkpoint.

import torch
from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct",
                                            dtype=torch.bfloat16, device_map="auto")
merged = PeftModel.from_pretrained(base, "qwen7b-support-adapter").merge_and_unload()
merged.save_pretrained("qwen7b-support-merged")

Pencil notes on a printed textbook

QLoRA is like writing pencil notes in the margins of a compressed pocket edition of a textbook. The printed text (base weights) never changes, the notes (adapters) are tiny, and when you want a clean copy you print a new edition with the notes typeset in, using the full-size original, not the pocket edition.

त्वरित जाँच: In QLoRA, what is stored in 4-bit precision?

• The LoRA adapter weights
• The frozen base model weights
• The optimizer state only
• The training dataset