# Why Fine-Tuning Works — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-transfer-learning

> Explain transfer learning and decide between prompting, fine-tuning and training from scratch.

## Reuse general knowledge, learn the specifics

**Pretraining** teaches a model general language patterns from a huge unlabelled corpus, for example by predicting masked or next tokens. This is the expensive part, done once by a lab with large compute. **Fine-tuning** continues training that model on your smaller, labelled dataset so it learns your specific task: classifying complaints, tagging medicine names or answering in your company's tone. Because the model already understands grammar and much world knowledge, a few thousand good examples are often enough, and training takes minutes to hours on one GPU instead of weeks on a cluster. This is **transfer learning**. The alternatives have trade-offs. **Prompting** a large general model needs no training but costs more per request and can be less consistent. **Training from scratch** only makes sense with very unusual data, such as a new language or a new modality, and very large resources. For most student and startup projects, the sensible first step is a small pretrained checkpoint fine-tuned on clean data.

## The fine-tuning workflow

These four stages repeat for almost every supervised fine-tuning job.

![Four connected stages: a labelled dataset, tokenization, training with Trainer, and evaluation on held-out data.](assets/figures/hugging-face/section-7-map.svg) — Figure 7.1: labelled dataset -> tokenize -> train with Trainer -> evaluate on held-out data

## A graduate joining a new company

A fresh engineering graduate already knows maths, programming and English. The company does not send them back to class 1; it gives a few weeks of induction on its own codebase. Pretraining is the degree; fine-tuning is the induction.

## Data quality beats data quantity

A thousand consistently labelled examples usually beat ten thousand noisy ones. Before tuning hyperparameters, read fifty random training rows yourself and fix label mistakes and duplicates.

**Quiz:** You have 3,000 labelled customer emails in five categories and one GPU. What is the most sensible first approach?

- [ ] Train a new Transformer from scratch
- [x] Fine-tune a small pretrained encoder such as a DistilBERT or BERT checkpoint
- [ ] Write regular expressions for each category
- [ ] Collect a billion more emails before starting

*Answer:* Fine-tune a small pretrained encoder such as a DistilBERT or BERT checkpoint. A small pretrained encoder fine-tuned on a few thousand examples is fast, cheap and typically strong for fixed-label classification.