LoRA and Parameter-Efficient Fine-Tuning

Train small adapters, freeze the giant

Full fine-tuning updates every weight, so a model with billions of parameters needs memory for the weights, their gradients and the optimiser state, often several times the model size. Parameter-efficient fine-tuning (PEFT) freezes the pretrained weights and trains only a small number of new ones. The most popular method, LoRA (Low-Rank Adaptation), adds to selected weight matrices W a learned update written as the product of two thin matrices, B times A, with a small rank r such as 8 or 16. Only A and B are trained, often well under one percent of the model's parameters. A scaling factor lora_alpha controls how strongly the update is applied. Benefits: far less GPU memory, small adapter files (megabytes instead of gigabytes), and the ability to keep one base model and swap adapters per task. After training you can merge the adapter into the base weights for inference with no extra latency. The trade-off is that a low rank may underfit tasks that need large behaviour changes.

Frozen base, small adapters

Only the adapter weights are trained; the base model stays untouched and can be shared by many adapters.

How few parameters LoRA trains

One 4096 x 4096 projection matrix, a typical size in a 7B-class model, with a rank-8 update.

# Trainable parameters: full weight matrix vs a LoRA update of rank r
d_in, d_out, r = 4096, 4096, 8

full = d_in * d_out
lora = r * d_in + d_out * r  # A is r x d_in, B is d_out x r

print(f"full matrix: {full:,}")
print(f"LoRA (r={r}): {lora:,}")
print(f"ratio: {lora / full:.2%}")

Output:

full matrix: 16,777,216
LoRA (r=8): 65,536
ratio: 0.39%

Wrapping a model with peft

target_modules names the layers to adapt; the names differ by architecture, so check print(model) first.

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype="auto")

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()  # prints trainable vs total parameter counts

Save adapters, not the whole model

model.save_pretrained("my-adapter") on a PEFT model writes only the adapter weights and config. Loading later needs the same base model id, so record it in your model card.

Quick check: In LoRA, which weights are updated during training?

• Only the small low-rank matrices added to selected layers
• All of the pretrained weights
• Only the embedding layer
• Only the tokenizer vocabulary