Decoding Strategies: Greedy, Sampling, Beams

Choosing the next token

Greedy decoding always takes the most likely token; it is deterministic but tends to repeat itself in long outputs. Beam search (num_beams > 1) keeps several candidate sequences and returns the one with the best overall score, which suits translation and summarisation where there is one "right" answer, but it costs more compute. Sampling (do_sample=True) draws the next token at random in proportion to its probability, which gives more varied, natural text. Three knobs shape sampling. temperature divides the logits before softmax: below 1 sharpens the distribution towards the top token, above 1 flattens it and adds risk. top_k keeps only the k most likely tokens. top_p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p. repetition_penalty discourages tokens that already appeared. Most chat checkpoints ship a generation_config.json with tuned defaults, so change one knob at a time and compare outputs.

What temperature does to probabilities

The same logits pushed through softmax at three temperatures.

import math

def softmax(logits, temperature=1.0):
    scaled = [x / temperature for x in logits]
    m = max(scaled)
    exps = [math.exp(x - m) for x in scaled]
    total = sum(exps)
    return [e / total for e in exps]

tokens = ["chai", "coffee", "juice"]
logits = [2.0, 1.0, 0.1]

for t in (0.5, 1.0, 2.0):
    probs = softmax(logits, t)
    shown = ", ".join(f"{tok}={p:.2f}" for tok, p in zip(tokens, probs))
    print(f"T={t}: {shown}")

Output:

T=0.5: chai=0.86, coffee=0.12, juice=0.02
T=1.0: chai=0.66, coffee=0.24, juice=0.10
T=2.0: chai=0.50, coffee=0.30, juice=0.19

Setting decoding options

Reuse a GenerationConfig so every call uses the same settings; set a seed when you need repeatable samples.

from transformers import GenerationConfig, set_seed

set_seed(42)
creative = GenerationConfig(
    do_sample=True, temperature=0.8, top_p=0.9, top_k=50,
    repetition_penalty=1.1, max_new_tokens=60,
)
factual = GenerationConfig(do_sample=False, num_beams=4, max_new_tokens=60)

# out = model.generate(**inputs, generation_config=creative)

temperature only matters when sampling

With do_sample=False, temperature, top_k and top_p are ignored and recent versions warn you about it. If two runs give identical text and you expected variety, check that sampling is actually on.

त्वरित जाँच: What happens when you lower the sampling temperature from 1.0 to 0.3?

• The distribution sharpens, so the most likely tokens are chosen far more often
• The model generates fewer tokens
• Every token becomes equally likely
• Beam search is switched on