# Why Models Read Subwords — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-subword

> Explain subword tokenization and how BPE, WordPiece and Unigram build a vocabulary.

## Words are too many, characters too few

A model cannot read text; it reads integers. A tokenizer splits text into **tokens** and maps each to an **id** in a fixed vocabulary. Splitting on whole words fails because the vocabulary would need millions of entries and any unseen word (a new slang term, a typo, a Hinglish spelling) becomes an unknown token. Splitting into single characters makes sequences very long and each token carries little meaning. **Subword tokenization** is the compromise: frequent words stay whole, rare words break into reusable pieces such as `token` + `ization`. Three algorithms dominate. **BPE** (byte-pair encoding, used by GPT-style models) starts from characters or bytes and repeatedly merges the most frequent adjacent pair. **WordPiece** (BERT) is similar but picks merges by a likelihood score and marks continuation pieces with `##`. **Unigram** (often via the SentencePiece library, used by T5 and many multilingual models) starts large and prunes tokens that least reduce likelihood. A tokenizer is learned from data, so it must always be paired with the model trained on it.

## From text to tensors

Tokenization is a fixed pipeline of its own, learned before the model was trained.

![Four connected stages: raw text, subword tokens, integer input IDs, and a padded batch of tensors with an attention mask.](assets/figures/hugging-face/section-3-map.svg) — Figure 3.1: raw text -> subword tokens -> input IDs -> padded batch tensors

## Training a toy BPE by hand

Three merge steps on a tiny corpus. Real tokenizers do the same thing tens of thousands of times over a huge corpus.

```python
from collections import Counter

# Toy BPE: start from characters, repeatedly merge the most frequent pair
words = [list("lower"), list("lowest"), list("low"), list("low")]

def most_frequent_pair(words):
    pairs = Counter()
    for w in words:
        pairs.update(zip(w, w[1:]))
    return pairs.most_common(1)[0][0]

def merge(words, pair):
    merged = []
    for w in words:
        out, i = [], 0
        while i < len(w):
            if i + 1 < len(w) and (w[i], w[i + 1]) == pair:
                out.append(w[i] + w[i + 1])
                i += 2
            else:
                out.append(w[i])
                i += 1
        merged.append(out)
    return merged

for step in range(3):
    pair = most_frequent_pair(words)
    words = merge(words, pair)
    print(f"merge {step + 1}: {pair} -> {words[0]}")
```

Output:

```
merge 1: ('l', 'o') -> ['lo', 'w', 'e', 'r']
merge 2: ('lo', 'w') -> ['low', 'e', 'r']
merge 3: ('low', 'e') -> ['lowe', 'r']
```

## Token counts are not word counts

Context limits and API bills are measured in tokens, not words. English prose is often a little over one token per word, while languages written in other scripts, including Hindi, can need noticeably more tokens for the same meaning with tokenizers trained mostly on English. Always count with the model's own tokenizer.

**Quiz:** What does a byte-pair encoding (BPE) tokenizer do repeatedly while it is being trained?

- [ ] Removes the least frequent words from the corpus
- [ ] Splits every word into single characters at inference time only
- [x] Merges the most frequent adjacent pair of symbols into a new token
- [ ] Assigns random ids to every word in the dictionary

*Answer:* Merges the most frequent adjacent pair of symbols into a new token. BPE training starts from small units and keeps merging the most frequent adjacent pair until the vocabulary reaches its target size.