Why Models Read Subwords

Words are too many, characters too few

A model cannot read text; it reads integers. A tokenizer splits text into tokens and maps each to an id in a fixed vocabulary. Splitting on whole words fails because the vocabulary would need millions of entries and any unseen word (a new slang term, a typo, a Hinglish spelling) becomes an unknown token. Splitting into single characters makes sequences very long and each token carries little meaning. Subword tokenization is the compromise: frequent words stay whole, rare words break into reusable pieces such as token + ization. Three algorithms dominate. BPE (byte-pair encoding, used by GPT-style models) starts from characters or bytes and repeatedly merges the most frequent adjacent pair. WordPiece (BERT) is similar but picks merges by a likelihood score and marks continuation pieces with ##. Unigram (often via the SentencePiece library, used by T5 and many multilingual models) starts large and prunes tokens that least reduce likelihood. A tokenizer is learned from data, so it must always be paired with the model trained on it.

From text to tensors

Tokenization is a fixed pipeline of its own, learned before the model was trained.

Training a toy BPE by hand

Three merge steps on a tiny corpus. Real tokenizers do the same thing tens of thousands of times over a huge corpus.

from collections import Counter

# Toy BPE: start from characters, repeatedly merge the most frequent pair
words = [list("lower"), list("lowest"), list("low"), list("low")]

def most_frequent_pair(words):
    pairs = Counter()
    for w in words:
        pairs.update(zip(w, w[1:]))
    return pairs.most_common(1)[0][0]

def merge(words, pair):
    merged = []
    for w in words:
        out, i = [], 0
        while i < len(w):
            if i + 1 < len(w) and (w[i], w[i + 1]) == pair:
                out.append(w[i] + w[i + 1])
                i += 2
            else:
                out.append(w[i])
                i += 1
        merged.append(out)
    return merged

for step in range(3):
    pair = most_frequent_pair(words)
    words = merge(words, pair)
    print(f"merge {step + 1}: {pair} -> {words[0]}")

Output:

merge 1: ('l', 'o') -> ['lo', 'w', 'e', 'r']
merge 2: ('lo', 'w') -> ['low', 'e', 'r']
merge 3: ('low', 'e') -> ['lowe', 'r']

Token counts are not word counts

Context limits and API bills are measured in tokens, not words. English prose is often a little over one token per word, while languages written in other scripts, including Hindi, can need noticeably more tokens for the same meaning with tokenizers trained mostly on English. Always count with the model's own tokenizer.

त्वरित जाँच: What does a byte-pair encoding (BPE) tokenizer do repeatedly while it is being trained?

• Removes the least frequent words from the corpus
• Splits every word into single characters at inference time only
• Merges the most frequent adjacent pair of symbols into a new token
• Assigns random ids to every word in the dictionary