Loss Functions

Match the loss to the task

A loss function measures how far predictions are from the targets, and training minimises it. For multi-class classification, the network outputs raw scores (logits); softmax turns them into probabilities and cross-entropy penalises low probability on the true class (heavily when the model is confidently wrong). For binary tasks use binary cross-entropy with logits; for regression, mean squared error or mean absolute error. In PyTorch, pass logits directly to cross_entropy, which applies softmax internally.

Loss, gradients, updates

Training repeats: predict, measure the loss, compute gradients, and nudge the weights.

Softmax and cross-entropy, run

I ran this on CPU with Python 3, PyTorch 2.14.1, numpy 2.5.3 and scikit-learn 1.9.1, with fixed seeds and one thread. Logits of 2.0, 0.5 and -1.0 become probabilities 0.786, 0.175 and 0.039. If the true class is 0 the cross-entropy is 0.241; if it is class 2, the confident mistake costs 3.241. A regression example gives an MSE of 0.625.

import torch
logits = torch.tensor([[2.0, 0.5, -1.0]])        # raw scores for 3 classes
probs = torch.softmax(logits, dim=1)
print("softmax probabilities:", probs.numpy().round(3))
for target in [0, 2]:
    ce = torch.nn.functional.cross_entropy(logits, torch.tensor([target]))
    print(f"true class {target}: cross-entropy {ce.item():.3f}")
pred, true = torch.tensor([2.5, 0.0]), torch.tensor([3.0, 1.0])
print("MSE for a regression output:", torch.nn.functional.mse_loss(pred, true).item())

Output:

softmax probabilities: [[0.786 0.175 0.039]]
true class 0: cross-entropy 0.241
true class 2: cross-entropy 3.241
MSE for a regression output: 0.625

Do not softmax twice

Feed raw logits to cross_entropy; applying softmax first gives wrong, flat gradients.

त्वरित जाँच: Which loss is standard for multi-class classification?

• Mean squared error on class ids
• Cross-entropy on the logits
• Accuracy
• Number of parameters