Evaluation and Metrics

Measuring what matters

During evaluation Trainer runs the model on eval_dataset and calls your compute_metrics(eval_pred) function with a pair: the model's predictions (logits as a NumPy array) and the true labels. You convert logits to predicted classes with argmax and return a dictionary of numbers, which Trainer logs with an eval_ prefix. Pick metrics that match the problem. Accuracy is fine for balanced classes. Precision asks "of the items I flagged, how many were right?", recall asks "of the true items, how many did I catch?", and F1 is their harmonic mean, useful for imbalanced data. NER is usually scored at the entity level with seqeval-style F1, and generation tasks use metrics such as ROUGE or BLEU, though human review often matters more. The evaluate library and scikit-learn provide tested implementations. Keep three splits: train to learn, validation to tune and pick checkpoints, and a test set you look at only once at the end.

A compute_metrics function you can test offline

The same function Trainer would call, run on fake logits so you can check it without a GPU.

import numpy as np

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    accuracy = (preds == labels).mean()
    tp = ((preds == 1) & (labels == 1)).sum()
    precision = tp / max((preds == 1).sum(), 1)
    recall = tp / max((labels == 1).sum(), 1)
    f1 = 2 * precision * recall / max(precision + recall, 1e-9)
    return {"accuracy": round(float(accuracy), 3), "f1": round(float(f1), 3)}

# What Trainer passes in: logits for each example and the true labels
logits = np.array([[2.0, -1.0], [0.3, 0.9], [-0.5, 1.5], [1.2, 0.4], [0.1, 0.2]])
labels = np.array([0, 1, 1, 1, 0])
print(compute_metrics((logits, labels)))

Output:

{'accuracy': 0.6, 'f1': 0.667}

Using evaluate and early stopping

evaluate loads standard metric implementations; the callback stops training when the metric stops improving.

import numpy as np
import evaluate
from transformers import EarlyStoppingCallback

f1 = evaluate.load("f1")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return f1.compute(predictions=preds, references=labels, average="macro")

# Needs load_best_model_at_end=True and metric_for_best_model="f1" in TrainingArguments
callbacks = [EarlyStoppingCallback(early_stopping_patience=2)]

त्वरित जाँच: A spam filter is 98% accurate, but only 2% of emails are spam. Which metric reveals whether it actually catches spam?

• Recall (or F1) on the spam class
• Training loss
• Accuracy on the training set
• Number of epochs