# Regularisation: Dropout, Weight Decay and Augmentation — Deep Learning & Neural Networks

Source: https://www.skillbyai.com/en/deep-learning/s-reg

> Fight overfitting in large networks.

## Make memorising harder

Large networks can memorise small datasets. **Dropout** randomly zeroes a fraction of activations during training, forcing redundancy; it is switched off in `eval()` mode. **Weight decay** penalises large weights. **Data augmentation** (random crops, flips, noise, for text paraphrasing or masking) creates varied training examples. **Early stopping** keeps the checkpoint with the best validation score. The best remedy remains more, and more varied, data.

## Dropout with a tiny training set, run

I ran this on CPU with Python 3, PyTorch 2.14.1, numpy 2.5.3 and scikit-learn 1.9.1, with fixed seeds and one thread. A large network trained on only 200 digit images reaches 1.000 training accuracy either way. Without dropout it scores 0.934 on the test set; with 50% dropout 0.946. A small but real gain on this run.

```python
import torch
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
torch.set_num_threads(1)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32); y = torch.tensor(y)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, train_size=200, random_state=0, stratify=y)   # tiny training set
def run(p):
    torch.manual_seed(0)
    m = torch.nn.Sequential(torch.nn.Linear(64, 512), torch.nn.ReLU(), torch.nn.Dropout(p),
                            torch.nn.Linear(512, 512), torch.nn.ReLU(), torch.nn.Dropout(p), torch.nn.Linear(512, 10))
    opt = torch.optim.Adam(m.parameters(), lr=1e-3)
    for _ in range(300):
        m.train(); opt.zero_grad()
        torch.nn.functional.cross_entropy(m(X_tr), y_tr).backward(); opt.step()
    m.eval()
    with torch.no_grad():
        return (m(X_tr).argmax(1) == y_tr).float().mean().item(), (m(X_te).argmax(1) == y_te).float().mean().item()
for p in [0.0, 0.5]:
    tr, te = run(p)
    print(f"dropout {p}: train accuracy {tr:.3f}  test accuracy {te:.3f}")
```

Output:

```
dropout 0.0: train accuracy 1.000  test accuracy 0.934
dropout 0.5: train accuracy 1.000  test accuracy 0.946
```

## Augment with realistic changes

Only use augmentations that keep the label correct: flipping a digit 6 can turn it into a 9.

**Quiz:** When is dropout active?

- [ ] Only during evaluation
- [x] During training only; it is disabled in eval mode
- [ ] Never
- [ ] Only when saving the model

*Answer:* During training only; it is disabled in eval mode. model.eval() turns dropout off.