Common Pitfalls and How to Spot Them

Bugs that produce no error

The worst transformers bugs run without errors and just give bad results. Mismatched tokenizer: loading a tokenizer from one checkpoint and the model from another scrambles every id; always load both from the same id or folder. Silent truncation: with truncation=True, text beyond the limit is dropped, so a label that depends on the end of a long document becomes unlearnable; check how many examples exceed max_length. Wrong padding side for decoder generation leads to odd outputs after pad tokens; use left padding. Missing pad token on many decoder tokenizers raises an error when batching; setting pad_token = eos_token is a common fix. Label mapping drift: if id2label does not match your dataset's integers, metrics look fine but predictions are named wrongly. Data leakage: near-duplicate rows shared between train and test inflate scores. Ignoring the chat template for instruct models degrades answers. A habit that catches most of these: decode a few tokenized examples back to text and eyeball them before training.

A five-line sanity check before training

Look at real model inputs, not your assumptions about them.

sample = train_ds[0]
print(tok.decode(sample["input_ids"]))                  # matches the raw text?
print(model.config.id2label, sample["labels"])          # label names line up?
lengths = [len(x) for x in train_ds["input_ids"]]
print(sum(l >= 256 for l in lengths), "of", len(lengths), "examples hit max_length")

Checking the question paper set

Before an exam, invigilators check that every student got the right paper set. Decoding your tokenized inputs is the same check: a few seconds of looking prevents hours of grading the wrong answers.

Quick check: A fine-tuned classifier scores 99% on the test set but performs poorly on new user data. Which pitfall is the most likely cause?

• Using safetensors instead of pickle
• Setting padding_side to left
• Running inference in bfloat16
• Duplicate or near-duplicate examples shared between train and test splits