# AutoTokenizer: Encode and Decode — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-autotokenizer

> Load the matching tokenizer for any checkpoint and read input_ids, attention_mask and special tokens.

## What a tokenizer call returns

`AutoTokenizer.from_pretrained(model_id)` reads the checkpoint's tokenizer files and returns the right class, usually a **fast** tokenizer backed by the Rust tokenizers library. Calling the tokenizer on text returns a dictionary-like **BatchEncoding**. `input_ids` holds the token ids. `attention_mask` holds 1 for real tokens and 0 for padding, so the model can ignore padding. Some models, such as BERT, also return `token_type_ids` to mark sentence A and sentence B in a pair. The tokenizer also inserts **special tokens** the model was trained with: BERT wraps input in `[CLS]` and `[SEP]`, while many decoder models add a beginning-of-sequence token. `tokenizer.convert_ids_to_tokens(ids)` shows the pieces, and `tokenizer.decode(ids, skip_special_tokens=True)` turns ids back into text. Fast tokenizers can also return `offset_mapping`, the character span of each token, which is how NER and question answering map predictions back to the original string.

## Inspect what the model will see

Encode a sentence, look at the subword pieces, then decode back to text. Exact ids depend on the vocabulary, so print them rather than memorising them.

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
enc = tok("Tokenization isn't magic!")

print(enc.keys())  # input_ids, token_type_ids, attention_mask
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# something like ['[CLS]', 'token', '##ization', 'isn', "'", 't', 'magic', '!', '[SEP]']
print(tok.decode(enc["input_ids"], skip_special_tokens=True))
```

## A railway reservation chart

Think of `input_ids` as the PNR numbers on a reservation chart and the vocabulary as the master list that says which passenger each number belongs to. Use the chart from a different train (another model's tokenizer) and every number points to the wrong passenger, even though the chart itself looks perfectly valid.

**Quiz:** In a BatchEncoding, what is the purpose of `attention_mask`?

- [ ] It stores the attention weights computed by the model
- [x] It marks which positions are real tokens (1) and which are padding (0)
- [ ] It lists the special tokens used by the tokenizer
- [ ] It masks random tokens for masked-language-model training

*Answer:* It marks which positions are real tokens (1) and which are padding (0). attention_mask is an input, not an output: 1 means attend to this position, 0 means it is padding and should be ignored.