Sampling Parameters: Temperature, top_p and Friends

How the next token is chosen

After each forward pass the model outputs a score (a logit) for every token. The sampler turns those into a choice. Temperature divides the logits before softmax: low values (0.1 to 0.3) sharpen the distribution so the top token almost always wins, which suits extraction and code; higher values (0.8 to 1.2) flatten it for varied, creative text. At 0 most runtimes simply pick the top token (greedy). top_k keeps only the k most likely tokens; top_p (nucleus) keeps the smallest set whose probabilities add up to p; min_p drops tokens far less likely than the best one. repeat_penalty discourages recently used tokens to reduce loops. seed fixes the random generator so the same prompt and settings repeat output on the same build and hardware. num_predict caps generated tokens, and stop ends generation at given strings.

What temperature does to probabilities

Three candidate next tokens after "The capital of India is New", with made-up logits.

import math

def softmax(logits, temperature):
    scaled = [x / temperature for x in logits]
    m = max(scaled)
    exps = [math.exp(x - m) for x in scaled]
    total = sum(exps)
    return [e / total for e in exps]

tokens = ["Delhi", "Mumbai", "Pune"]
logits = [4.0, 3.0, 1.0]
for t in (0.2, 0.8, 1.5):
    probs = softmax(logits, t)
    print(f"T={t}: " + ", ".join(f"{w}={p:.2f}" for w, p in zip(tokens, probs)))

Output:

T=0.2: Delhi=0.99, Mumbai=0.01, Pune=0.00
T=0.8: Delhi=0.76, Mumbai=0.22, Pune=0.02
T=1.5: Delhi=0.61, Mumbai=0.31, Pune=0.08

A quiz-show contestant

Low temperature is a contestant who always says the answer they are most sure of. High temperature is one who sometimes gambles on the second or third guess. For a factual MCQ you want the first; for writing a poem the second is more fun.

Change one knob at a time

Beginners set temperature, top_k, top_p and min_p all at once and cannot tell which helped. Start from the model's defaults, change temperature first, and fix a seed while comparing prompts so differences come from your change, not randomness.

Quick check: For extracting fields from invoices as JSON, which setting is most appropriate?

• Temperature around 1.5 for variety
• A very high repeat_penalty
• A low temperature such as 0 to 0.2
• top_k of 1000