# Loss Functions — Deep Learning & Neural Networks

Source: https://www.skillbyai.com/en/deep-learning/t-loss

> One number that says how wrong the model is.

## Match the loss to the task

A **loss function** measures how far predictions are from the targets, and training minimises it. For **multi-class classification**, the network outputs raw scores (**logits**); **softmax** turns them into probabilities and **cross-entropy** penalises low probability on the true class (heavily when the model is confidently wrong). For **binary** tasks use binary cross-entropy with logits; for **regression**, mean squared error or mean absolute error. In PyTorch, pass logits directly to `cross_entropy`, which applies softmax internally.

## Loss, gradients, updates

Training repeats: predict, measure the loss, compute gradients, and nudge the weights.

![Four ideas: loss, backpropagation, autograd, optimisers.](assets/figures/deep-learning/section-2-map.svg) — Figure 2.1 — Loss, backpropagation, autograd and optimisers.

## Softmax and cross-entropy, run

I ran this on CPU with Python 3, PyTorch 2.14.1, numpy 2.5.3 and scikit-learn 1.9.1, with fixed seeds and one thread. Logits of 2.0, 0.5 and -1.0 become probabilities 0.786, 0.175 and 0.039. If the true class is 0 the cross-entropy is 0.241; if it is class 2, the confident mistake costs 3.241. A regression example gives an MSE of 0.625.

```python
import torch
logits = torch.tensor([[2.0, 0.5, -1.0]])        # raw scores for 3 classes
probs = torch.softmax(logits, dim=1)
print("softmax probabilities:", probs.numpy().round(3))
for target in [0, 2]:
    ce = torch.nn.functional.cross_entropy(logits, torch.tensor([target]))
    print(f"true class {target}: cross-entropy {ce.item():.3f}")
pred, true = torch.tensor([2.5, 0.0]), torch.tensor([3.0, 1.0])
print("MSE for a regression output:", torch.nn.functional.mse_loss(pred, true).item())
```

Output:

```
softmax probabilities: [[0.786 0.175 0.039]]
true class 0: cross-entropy 0.241
true class 2: cross-entropy 3.241
MSE for a regression output: 0.625
```

## Do not softmax twice

Feed raw logits to cross_entropy; applying softmax first gives wrong, flat gradients.

**Quiz:** Which loss is standard for multi-class classification?

- [ ] Mean squared error on class ids
- [x] Cross-entropy on the logits
- [ ] Accuracy
- [ ] Number of parameters

*Answer:* Cross-entropy on the logits. Cross-entropy penalises low probability on the true class.