Distillation: Teaching a Small Model From a Large One

Copy the behaviour, not the size

Distillation trains a small student to imitate a larger teacher. A common LLM recipe: collect realistic inputs, have a strong model answer them (with checks), review a sample, and fine-tune a smaller model on those pairs. On a narrow task the student can approach the teacher at far lower cost and latency. Cautions: the student inherits the teacher's mistakes; check the provider's terms on training other models with outputs; measure on human-verified test data, not just agreement with the teacher; keep the data diverse.

A student helped by teacher labels, run

I ran this with Python, numpy 2.5.3 and scikit-learn 1.9.1, with fixed random seeds. It trains small classical models, not a language model: the mechanics (gradient descent, learning rate, overfitting, forgetting, low-rank updates) are the same ideas that apply to fine-tuning an LLM, but the numbers are not LLM results. The linear teacher scores 0.885 on held-out data. A small tree trained on 300 noisy real labels scores 0.723; given teacher labels for 4,700 more unlabelled inputs it scores 0.791, improving but staying below the teacher.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(6)
X = rng.normal(size=(8000, 12)); w = rng.normal(size=12)
y = (X @ w + rng.normal(size=8000) * 1.2 > 0).astype(int)      # noisy real labels
Xtr, ytr, Xte, yte = X[:300], y[:300], X[5000:], y[5000:]

teacher = LogisticRegression(max_iter=2000).fit(Xtr, ytr)                       # the "big" model trained on the labelled data
Xunl = X[300:5000]                                                               # plenty of UNLABELLED inputs
teacher_labels = teacher.predict(Xunl)

student_alone = DecisionTreeClassifier(max_depth=6, random_state=0).fit(Xtr, ytr)
student_distilled = DecisionTreeClassifier(max_depth=6, random_state=0).fit(np.vstack([Xtr, Xunl]), np.concatenate([ytr, teacher_labels]))
print("teacher                         :", round(teacher.score(Xte, yte), 3))
print("small student, real labels only :", round(student_alone.score(Xte, yte), 3))
print("small student + teacher labels  :", round(student_distilled.score(Xte, yte), 3))

Output:

teacher                         : 0.885
small student, real labels only : 0.723
small student + teacher labels  : 0.791

Sample and read teacher outputs

Read at least a few dozen teacher answers before training; every systematic error you skip becomes the student's habit.

Quick check: How should a distilled student be evaluated?

• Only by how closely it copies the teacher
• On human-verified test data, not only agreement with the teacher
• Not at all
• Only by model size