Regularisation: Dropout, Weight Decay and Augmentation

Make memorising harder

Large networks can memorise small datasets. Dropout randomly zeroes a fraction of activations during training, forcing redundancy; it is switched off in eval() mode. Weight decay penalises large weights. Data augmentation (random crops, flips, noise, for text paraphrasing or masking) creates varied training examples. Early stopping keeps the checkpoint with the best validation score. The best remedy remains more, and more varied, data.

Dropout with a tiny training set, run

I ran this on CPU with Python 3, PyTorch 2.14.1, numpy 2.5.3 and scikit-learn 1.9.1, with fixed seeds and one thread. A large network trained on only 200 digit images reaches 1.000 training accuracy either way. Without dropout it scores 0.934 on the test set; with 50% dropout 0.946. A small but real gain on this run.

import torch
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
torch.set_num_threads(1)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32); y = torch.tensor(y)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, train_size=200, random_state=0, stratify=y)   # tiny training set
def run(p):
    torch.manual_seed(0)
    m = torch.nn.Sequential(torch.nn.Linear(64, 512), torch.nn.ReLU(), torch.nn.Dropout(p),
                            torch.nn.Linear(512, 512), torch.nn.ReLU(), torch.nn.Dropout(p), torch.nn.Linear(512, 10))
    opt = torch.optim.Adam(m.parameters(), lr=1e-3)
    for _ in range(300):
        m.train(); opt.zero_grad()
        torch.nn.functional.cross_entropy(m(X_tr), y_tr).backward(); opt.step()
    m.eval()
    with torch.no_grad():
        return (m(X_tr).argmax(1) == y_tr).float().mean().item(), (m(X_te).argmax(1) == y_te).float().mean().item()
for p in [0.0, 0.5]:
    tr, te = run(p)
    print(f"dropout {p}: train accuracy {tr:.3f}  test accuracy {te:.3f}")

Output:

dropout 0.0: train accuracy 1.000  test accuracy 0.934
dropout 0.5: train accuracy 1.000  test accuracy 0.946

Augment with realistic changes

Only use augmentations that keep the label correct: flipping a digit 6 can turn it into a 9.

त्वरित जाँच: When is dropout active?

• Only during evaluation
• During training only; it is disabled in eval mode
• Never
• Only when saving the model