# Evaluation and Metrics — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-metrics

> Write a compute_metrics function, choose the right metric and avoid evaluating on training data.

## Measuring what matters

During evaluation Trainer runs the model on `eval_dataset` and calls your `compute_metrics(eval_pred)` function with a pair: the model's **predictions** (logits as a NumPy array) and the **true labels**. You convert logits to predicted classes with argmax and return a dictionary of numbers, which Trainer logs with an `eval_` prefix. Pick metrics that match the problem. **Accuracy** is fine for balanced classes. **Precision** asks "of the items I flagged, how many were right?", **recall** asks "of the true items, how many did I catch?", and **F1** is their harmonic mean, useful for imbalanced data. NER is usually scored at the entity level with seqeval-style F1, and generation tasks use metrics such as ROUGE or BLEU, though human review often matters more. The `evaluate` library and scikit-learn provide tested implementations. Keep three splits: train to learn, validation to tune and pick checkpoints, and a **test** set you look at only once at the end.

## A compute_metrics function you can test offline

The same function Trainer would call, run on fake logits so you can check it without a GPU.

```python
import numpy as np

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    accuracy = (preds == labels).mean()
    tp = ((preds == 1) & (labels == 1)).sum()
    precision = tp / max((preds == 1).sum(), 1)
    recall = tp / max((labels == 1).sum(), 1)
    f1 = 2 * precision * recall / max(precision + recall, 1e-9)
    return {"accuracy": round(float(accuracy), 3), "f1": round(float(f1), 3)}

# What Trainer passes in: logits for each example and the true labels
logits = np.array([[2.0, -1.0], [0.3, 0.9], [-0.5, 1.5], [1.2, 0.4], [0.1, 0.2]])
labels = np.array([0, 1, 1, 1, 0])
print(compute_metrics((logits, labels)))
```

Output:

```
{'accuracy': 0.6, 'f1': 0.667}
```

## Using evaluate and early stopping

`evaluate` loads standard metric implementations; the callback stops training when the metric stops improving.

```python
import numpy as np
import evaluate
from transformers import EarlyStoppingCallback

f1 = evaluate.load("f1")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return f1.compute(predictions=preds, references=labels, average="macro")

# Needs load_best_model_at_end=True and metric_for_best_model="f1" in TrainingArguments
callbacks = [EarlyStoppingCallback(early_stopping_patience=2)]
```

**Quiz:** A spam filter is 98% accurate, but only 2% of emails are spam. Which metric reveals whether it actually catches spam?

- [x] Recall (or F1) on the spam class
- [ ] Training loss
- [ ] Accuracy on the training set
- [ ] Number of epochs

*Answer:* Recall (or F1) on the spam class. A model that predicts "not spam" for everything is also 98% accurate. Recall and F1 on the minority class show whether spam is caught.