The Transformer in Ten Minutes

Self-attention is the core idea

A Transformer turns text into a sequence of tokens, maps each token to a vector called an embedding, and then passes the vectors through stacked layers. The heart of each layer is self-attention: every token builds a query vector, and compares it with the key vectors of every token in the sequence (including itself) using a dot product. The scores are scaled by the square root of the vector size, turned into weights with softmax, and used to average the value vectors. So the word "bank" can borrow meaning from "river" elsewhere in the sentence. Because all tokens are processed in parallel rather than one by one as in older RNNs, Transformers train efficiently on GPUs. Since attention by itself ignores order, models add positional information to the embeddings. The cost of plain attention grows with the square of the sequence length, which is why every model has a maximum context length.

Attention weights by hand

A tiny version of scaled dot-product attention for one query over three keys. The weights always sum to 1.

import math

# Toy scaled dot-product attention for one query over three keys
query = [1.0, 0.0]
keys = {"the": [0.2, 0.1], "bank": [1.0, 0.3], "river": [0.9, 0.8]}
d_k = len(query)

scores = {w: sum(q * k for q, k in zip(query, vec)) / math.sqrt(d_k)
          for w, vec in keys.items()}
total = sum(math.exp(s) for s in scores.values())
weights = {w: math.exp(s) / total for w, s in scores.items()}

for word, w in weights.items():
    print(f"{word:>5}: {w:.2f}")

Output:

  the: 0.23
 bank: 0.40
river: 0.37

Encoder, decoder and both

The original Transformer had two halves. An encoder reads the whole input at once and every token can attend to every other token, which is ideal for understanding tasks such as classification, named-entity recognition and embeddings; BERT is the classic encoder-only model. A decoder generates text one token at a time and uses a causal mask so each position sees only earlier tokens; GPT-style and most modern chat models are decoder-only. Encoder-decoder models such as T5 and BART encode an input and then decode a new sequence, which suits translation and summarisation. Knowing which family a checkpoint belongs to tells you which Auto class to load, whether it can generate text, and how its tokenizer should pad. The library hides the maths, but these three words appear in every model card, so learn them now.

Do not confuse parameters with context length

A "7B" model has about seven billion learned numbers (parameters). Its context length is a different number: how many tokens it can attend to at once. Interviewers like to check that you keep these apart.

Quick check: Why does a decoder-only model use a causal mask in self-attention?

• So that each token can only attend to itself and earlier tokens while generating
• To make attention cost linear in sequence length
• To remove positional information from the embeddings
• To let the model read the whole input bidirectionally