Why Fine-Tuning Works

Reuse general knowledge, learn the specifics

Pretraining teaches a model general language patterns from a huge unlabelled corpus, for example by predicting masked or next tokens. This is the expensive part, done once by a lab with large compute. Fine-tuning continues training that model on your smaller, labelled dataset so it learns your specific task: classifying complaints, tagging medicine names or answering in your company's tone. Because the model already understands grammar and much world knowledge, a few thousand good examples are often enough, and training takes minutes to hours on one GPU instead of weeks on a cluster. This is transfer learning. The alternatives have trade-offs. Prompting a large general model needs no training but costs more per request and can be less consistent. Training from scratch only makes sense with very unusual data, such as a new language or a new modality, and very large resources. For most student and startup projects, the sensible first step is a small pretrained checkpoint fine-tuned on clean data.

The fine-tuning workflow

These four stages repeat for almost every supervised fine-tuning job.

A graduate joining a new company

A fresh engineering graduate already knows maths, programming and English. The company does not send them back to class 1; it gives a few weeks of induction on its own codebase. Pretraining is the degree; fine-tuning is the induction.

Data quality beats data quantity

A thousand consistently labelled examples usually beat ten thousand noisy ones. Before tuning hyperparameters, read fifty random training rows yourself and fix label mistakes and duplicates.

Quick check: You have 3,000 labelled customer emails in five categories and one GPU. What is the most sensible first approach?

• Train a new Transformer from scratch
• Fine-tune a small pretrained encoder such as a DistilBERT or BERT checkpoint
• Write regular expressions for each category
• Collect a billion more emails before starting