Padding, Truncation and Batches

Why batches need padding and truncation

A tensor must be rectangular, but sentences have different lengths. Padding appends a special pad id to shorter sequences so every row in the batch has the same length, and sets the matching attention_mask entries to 0. padding=True (or "longest") pads to the longest sequence in the current batch, which is usually the most efficient. padding="max_length" pads everything to max_length, which wastes compute but gives fixed shapes that some hardware prefers. Truncation cuts sequences longer than a limit: truncation=True cuts to max_length or to the model's maximum input length. Without truncation, a long document can exceed the model's context and raise an error or produce silent problems. Pass return_tensors="pt" to get PyTorch tensors that can go straight into the model. For tasks that need the whole long document, split it into overlapping windows instead of truncating, so no evidence is thrown away.

Padding by hand

This is exactly what padding=True does: pad to the longest row and record the mask.

# What padding=True does to a batch of token-id lists
batch = [[101, 7592, 102], [101, 2129, 2024, 2017, 102]]
pad_id = 0
longest = max(len(ids) for ids in batch)

input_ids = [ids + [pad_id] * (longest - len(ids)) for ids in batch]
attention_mask = [[1] * len(ids) + [0] * (longest - len(ids)) for ids in batch]

for ids, mask in zip(input_ids, attention_mask):
    print(ids, mask)

Output:

[101, 7592, 102, 0, 0] [1, 1, 1, 0, 0]
[101, 2129, 2024, 2017, 102] [1, 1, 1, 1, 1]

The same thing with a real tokenizer

One call pads, truncates and returns PyTorch tensors ready for the model.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
batch = tok(
    ["Hi!", "How are you doing this semester?"],
    padding=True,          # pad to the longest in this batch
    truncation=True,       # never exceed the model's max length
    max_length=32,
    return_tensors="pt",
)
print(batch["input_ids"].shape)       # (2, length of the longer sentence)
print(batch["attention_mask"])

Decoder models generate with left padding

For batched generation with decoder-only models, set tokenizer.padding_side = "left" so every prompt ends at the last position, where the next token is appended. Many decoder tokenizers also have no pad token; a common fix is tokenizer.pad_token = tokenizer.eos_token.

त्वरित जाँच: You tokenize a batch with `padding=True`. What length does each row end up with?

• The model's maximum context length
• A fixed 512 tokens
• The length of the shortest sequence in the batch
• The length of the longest sequence in the batch