# Decoding Strategies: Greedy, Sampling, Beams — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-decoding

> Tune temperature, top-k, top-p and beam search and predict how each changes the output.

## Choosing the next token

**Greedy** decoding always takes the most likely token; it is deterministic but tends to repeat itself in long outputs. **Beam search** (`num_beams > 1`) keeps several candidate sequences and returns the one with the best overall score, which suits translation and summarisation where there is one "right" answer, but it costs more compute. **Sampling** (`do_sample=True`) draws the next token at random in proportion to its probability, which gives more varied, natural text. Three knobs shape sampling. **temperature** divides the logits before softmax: below 1 sharpens the distribution towards the top token, above 1 flattens it and adds risk. **top_k** keeps only the k most likely tokens. **top_p** (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p. `repetition_penalty` discourages tokens that already appeared. Most chat checkpoints ship a `generation_config.json` with tuned defaults, so change one knob at a time and compare outputs.

## What temperature does to probabilities

The same logits pushed through softmax at three temperatures.

```python
import math

def softmax(logits, temperature=1.0):
    scaled = [x / temperature for x in logits]
    m = max(scaled)
    exps = [math.exp(x - m) for x in scaled]
    total = sum(exps)
    return [e / total for e in exps]

tokens = ["chai", "coffee", "juice"]
logits = [2.0, 1.0, 0.1]

for t in (0.5, 1.0, 2.0):
    probs = softmax(logits, t)
    shown = ", ".join(f"{tok}={p:.2f}" for tok, p in zip(tokens, probs))
    print(f"T={t}: {shown}")
```

Output:

```
T=0.5: chai=0.86, coffee=0.12, juice=0.02
T=1.0: chai=0.66, coffee=0.24, juice=0.10
T=2.0: chai=0.50, coffee=0.30, juice=0.19
```

## Setting decoding options

Reuse a GenerationConfig so every call uses the same settings; set a seed when you need repeatable samples.

```python
from transformers import GenerationConfig, set_seed

set_seed(42)
creative = GenerationConfig(
    do_sample=True, temperature=0.8, top_p=0.9, top_k=50,
    repetition_penalty=1.1, max_new_tokens=60,
)
factual = GenerationConfig(do_sample=False, num_beams=4, max_new_tokens=60)

# out = model.generate(**inputs, generation_config=creative)
```

## temperature only matters when sampling

With `do_sample=False`, temperature, top_k and top_p are ignored and recent versions warn you about it. If two runs give identical text and you expected variety, check that sampling is actually on.

**Quiz:** What happens when you lower the sampling temperature from 1.0 to 0.3?

- [x] The distribution sharpens, so the most likely tokens are chosen far more often
- [ ] The model generates fewer tokens
- [ ] Every token becomes equally likely
- [ ] Beam search is switched on

*Answer:* The distribution sharpens, so the most likely tokens are chosen far more often. Dividing logits by a small temperature widens the gaps between them, so softmax concentrates probability on the top tokens.