# The Transformer in Ten Minutes — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-transformer-recap

> Explain self-attention, tokens and the encoder/decoder split well enough to read any model card.

## Self-attention is the core idea

A Transformer turns text into a sequence of **tokens**, maps each token to a vector called an **embedding**, and then passes the vectors through stacked layers. The heart of each layer is **self-attention**: every token builds a *query* vector, and compares it with the *key* vectors of every token in the sequence (including itself) using a dot product. The scores are scaled by the square root of the vector size, turned into weights with **softmax**, and used to average the *value* vectors. So the word "bank" can borrow meaning from "river" elsewhere in the sentence. Because all tokens are processed in parallel rather than one by one as in older RNNs, Transformers train efficiently on GPUs. Since attention by itself ignores order, models add **positional information** to the embeddings. The cost of plain attention grows with the square of the sequence length, which is why every model has a maximum context length.

## Attention weights by hand

A tiny version of scaled dot-product attention for one query over three keys. The weights always sum to 1.

```python
import math

# Toy scaled dot-product attention for one query over three keys
query = [1.0, 0.0]
keys = {"the": [0.2, 0.1], "bank": [1.0, 0.3], "river": [0.9, 0.8]}
d_k = len(query)

scores = {w: sum(q * k for q, k in zip(query, vec)) / math.sqrt(d_k)
          for w, vec in keys.items()}
total = sum(math.exp(s) for s in scores.values())
weights = {w: math.exp(s) / total for w, s in scores.items()}

for word, w in weights.items():
    print(f"{word:>5}: {w:.2f}")
```

Output:

```
  the: 0.23
 bank: 0.40
river: 0.37
```

## Encoder, decoder and both

The original Transformer had two halves. An **encoder** reads the whole input at once and every token can attend to every other token, which is ideal for *understanding* tasks such as classification, named-entity recognition and embeddings; BERT is the classic encoder-only model. A **decoder** generates text one token at a time and uses a *causal mask* so each position sees only earlier tokens; GPT-style and most modern chat models are decoder-only. **Encoder-decoder** models such as T5 and BART encode an input and then decode a new sequence, which suits translation and summarisation. Knowing which family a checkpoint belongs to tells you which Auto class to load, whether it can generate text, and how its tokenizer should pad. The library hides the maths, but these three words appear in every model card, so learn them now.

## Do not confuse parameters with context length

A "7B" model has about seven billion learned numbers (parameters). Its context length is a different number: how many tokens it can attend to at once. Interviewers like to check that you keep these apart.

**Quiz:** Why does a decoder-only model use a causal mask in self-attention?

- [x] So that each token can only attend to itself and earlier tokens while generating
- [ ] To make attention cost linear in sequence length
- [ ] To remove positional information from the embeddings
- [ ] To let the model read the whole input bidirectionally

*Answer:* So that each token can only attend to itself and earlier tokens while generating. Generation predicts the next token, so a position must not peek at future tokens. The causal mask hides them; it does not change the quadratic cost.