LoRA and Parameter-Efficient Fine-Tuning

The needed change is often low-rank

Updating every weight is expensive and yields a full model copy per task. Parameter-efficient fine-tuning (PEFT) trains only a few added parameters. LoRA freezes each chosen weight matrix W and learns ΔW = A·B, where A is d×r and B is r×d with a small rank r (for example 8). That is r·2d trainable numbers instead of d²: for d = 4096, r = 8, about 65,000 instead of 16.8 million per matrix (0.39%). The bet is that the needed change is low-rank. The adapter is megabytes, can be swapped per task on one base model or merged for serving; QLoRA also stores the frozen base in 4-bit form.

Recovering a rank-2 change with a rank-2 adapter, run

I ran this with Python, numpy 2.5.3 and scikit-learn 1.9.1, with fixed random seeds. It trains small classical models, not a language model: the mechanics (gradient descent, learning rate, overfitting, forgetting, low-rank updates) are the same ideas that apply to fine-tuning an LLM, but the numbers are not LLM results. A frozen 32x32 matrix needs a rank-2 change. With no update the error is 3.29; a rank-1 adapter (64 trainable numbers, 6%) cuts it to 1.64; rank 2 (128 numbers, 12%) reaches essentially zero, and rank 4 also reaches zero.

import numpy as np
rng = np.random.default_rng(1)
d, true_rank = 32, 2

W0 = rng.normal(size=(d, d)) / np.sqrt(d)                      # frozen "pretrained" weight
delta = (rng.normal(size=(d, true_rank)) @ rng.normal(size=(true_rank, d))) * 0.3   # the change the new task needs: rank 2
W_target = W0 + delta

X = rng.normal(size=(400, d)); Y = X @ W_target.T               # task data produced by the target weights

def train_lora(r, steps=3000, lr=0.02):
    A = rng.normal(size=(d, r)) * 0.1; B = np.zeros((r, d))     # W = W0 + A @ B ; only A and B are trained
    for _ in range(steps):
        W = W0 + A @ B
        err = X @ W.T - Y                                       # (n, d)
        G = err.T @ X / len(X)                                   # gradient wrt W  (d, d)
        gA, gB = G @ B.T, A.T @ G
        A -= lr * gA; B -= lr * gB
    return float(np.mean((X @ (W0 + A @ B).T - Y) ** 2)), A.size + B.size

print("frozen W0 only: loss", round(float(np.mean((X @ W0.T - Y) ** 2)), 4), "| full matrix has", d * d, "parameters")
for r in (1, 2, 4):
    loss, params = train_lora(r)
    print(f"LoRA rank {r}: loss {loss:.5f} with {params} trainable parameters ({params / (d * d):.0%} of the full matrix)")

Output:

frozen W0 only: loss 3.2905 | full matrix has 1024 parameters
LoRA rank 1: loss 1.63657 with 64 trainable parameters (6% of the full matrix)
LoRA rank 2: loss 0.00000 with 128 trainable parameters (12% of the full matrix)
LoRA rank 4: loss 0.00000 with 256 trainable parameters (25% of the full matrix)

Training memory: full fine-tune versus LoRA, run

I ran this with plain Python 3 (standard library only), using example numbers. With the rough estimate of 16 bytes per trained parameter, a 7B model needs about 112 GB to fully fine-tune but about 14 GB when only a 0.4% adapter trains over a frozen 2-byte base. Rough planning numbers before activations and overhead.

# Rough training memory (mixed precision with Adam): weights 2 B + gradients 2 B + fp32 master weights 4 B + 2 optimizer states 8 B
# = about 16 bytes per TRAINED parameter, plus frozen weights at 2 B per parameter. Activations come on top.
def gb(x): return x / 1e9
def full_ft(params): return params * 16
def lora(params, trainable): return params * 2 + trainable * 16     # frozen base at 16-bit + trained adapters

for name, params in (("7B", 7e9), ("13B", 13e9), ("70B", 70e9)):
    trainable = params * 0.004                                       # ~0.4% trainable with a small-rank adapter (example)
    print(f"{name:4} full fine-tune ~ {gb(full_ft(params)):7.0f} GB   LoRA ~ {gb(lora(params, trainable)):6.0f} GB   (before activations)")

Output:

7B   full fine-tune ~     112 GB   LoRA ~     14 GB   (before activations)
13B  full fine-tune ~     208 GB   LoRA ~     27 GB   (before activations)
70B  full fine-tune ~    1120 GB   LoRA ~    144 GB   (before activations)

Quick check: What does LoRA train?

• Small low-rank matrices added to frozen weights
• Every weight of the model
• Only the tokenizer
• The user interface