AutoTokenizer: Encode and Decode

What a tokenizer call returns

AutoTokenizer.from_pretrained(model_id) reads the checkpoint's tokenizer files and returns the right class, usually a fast tokenizer backed by the Rust tokenizers library. Calling the tokenizer on text returns a dictionary-like BatchEncoding. input_ids holds the token ids. attention_mask holds 1 for real tokens and 0 for padding, so the model can ignore padding. Some models, such as BERT, also return token_type_ids to mark sentence A and sentence B in a pair. The tokenizer also inserts special tokens the model was trained with: BERT wraps input in [CLS] and [SEP], while many decoder models add a beginning-of-sequence token. tokenizer.convert_ids_to_tokens(ids) shows the pieces, and tokenizer.decode(ids, skip_special_tokens=True) turns ids back into text. Fast tokenizers can also return offset_mapping, the character span of each token, which is how NER and question answering map predictions back to the original string.

Inspect what the model will see

Encode a sentence, look at the subword pieces, then decode back to text. Exact ids depend on the vocabulary, so print them rather than memorising them.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
enc = tok("Tokenization isn't magic!")

print(enc.keys())  # input_ids, token_type_ids, attention_mask
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# something like ['[CLS]', 'token', '##ization', 'isn', "'", 't', 'magic', '!', '[SEP]']
print(tok.decode(enc["input_ids"], skip_special_tokens=True))

A railway reservation chart

Think of input_ids as the PNR numbers on a reservation chart and the vocabulary as the master list that says which passenger each number belongs to. Use the chart from a different train (another model's tokenizer) and every number points to the wrong passenger, even though the chart itself looks perfectly valid.

त्वरित जाँच: In a BatchEncoding, what is the purpose of `attention_mask`?

• It stores the attention weights computed by the model
• It marks which positions are real tokens (1) and which are padding (0)
• It lists the special tokens used by the tokenizer
• It masks random tokens for masked-language-model training