# LoRA and Parameter-Efficient Fine-Tuning — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-lora

> Explain how LoRA trains a tiny fraction of parameters and configure it with the peft library.

## Train small adapters, freeze the giant

Full fine-tuning updates every weight, so a model with billions of parameters needs memory for the weights, their gradients and the optimiser state, often several times the model size. **Parameter-efficient fine-tuning (PEFT)** freezes the pretrained weights and trains only a small number of new ones. The most popular method, **LoRA** (Low-Rank Adaptation), adds to selected weight matrices W a learned update written as the product of two thin matrices, B times A, with a small **rank** r such as 8 or 16. Only A and B are trained, often well under one percent of the model's parameters. A scaling factor `lora_alpha` controls how strongly the update is applied. Benefits: far less GPU memory, small adapter files (megabytes instead of gigabytes), and the ability to keep one base model and swap adapters per task. After training you can **merge** the adapter into the base weights for inference with no extra latency. The trade-off is that a low rank may underfit tasks that need large behaviour changes.

## Frozen base, small adapters

Only the adapter weights are trained; the base model stays untouched and can be shared by many adapters.

![Three connected stages: frozen base model weights, small trainable LoRA adapters, and the merged or adapter-attached model used for inference.](assets/figures/hugging-face/section-8-map.svg) — Figure 8.1: frozen base weights -> trainable LoRA adapters -> merged model for serving

## How few parameters LoRA trains

One 4096 x 4096 projection matrix, a typical size in a 7B-class model, with a rank-8 update.

```python
# Trainable parameters: full weight matrix vs a LoRA update of rank r
d_in, d_out, r = 4096, 4096, 8

full = d_in * d_out
lora = r * d_in + d_out * r  # A is r x d_in, B is d_out x r

print(f"full matrix: {full:,}")
print(f"LoRA (r={r}): {lora:,}")
print(f"ratio: {lora / full:.2%}")
```

Output:

```
full matrix: 16,777,216
LoRA (r=8): 65,536
ratio: 0.39%
```

## Wrapping a model with peft

`target_modules` names the layers to adapt; the names differ by architecture, so check `print(model)` first.

```python
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype="auto")

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()  # prints trainable vs total parameter counts
```

## Save adapters, not the whole model

`model.save_pretrained("my-adapter")` on a PEFT model writes only the adapter weights and config. Loading later needs the same base model id, so record it in your model card.

**Quiz:** In LoRA, which weights are updated during training?

- [x] Only the small low-rank matrices added to selected layers
- [ ] All of the pretrained weights
- [ ] Only the embedding layer
- [ ] Only the tokenizer vocabulary

*Answer:* Only the small low-rank matrices added to selected layers. LoRA freezes the base model and trains the low-rank A and B matrices, which is why memory use and file size drop so much.