How generate() Writes Text

One token at a time

A causal language model does not write a paragraph in one go. Given a prompt, it runs a forward pass and produces logits for every token in the vocabulary as the possible next token. A decoding strategy picks one token, it is appended to the input, and the loop repeats until the model emits an end-of-sequence (EOS) token or a length limit is reached. This is called autoregressive generation. model.generate() implements the loop for you. Use max_new_tokens to cap how many tokens are added; it is clearer than max_length, which counts the prompt too. The output contains the prompt ids followed by the new ids, so slice off the prompt before decoding if you only want the reply. Re-running attention over the whole sequence each step would be wasteful, so generate keeps a KV cache of keys and values from earlier positions and only computes the newest token. Each step then does little arithmetic but must still read every weight from GPU memory, so generation speed is usually limited by memory bandwidth rather than raw compute.

The generation loop

Each new token goes back into the input, so the model sees its own previous choices.

Tracing greedy decoding

A toy model that scores the next word from the last word only. Follow how the loop grows the token list until it hits the end marker.

# Toy "model": maps the last token to next-token scores
table = {"<s>": {"I": 0.9, "you": 0.1},
         "I": {"love": 0.7, "eat": 0.3},
         "love": {"chai": 0.6, "</s>": 0.4},
         "chai": {"</s>": 0.8, "too": 0.2}}
tokens = ["<s>"]
while tokens[-1] != "</s>" and len(tokens) < 6:
    scores = table[tokens[-1]]
    next_token = max(scores, key=scores.get)
    tokens.append(next_token)
print(" ".join(tokens[1:-1]))

1. Line 6: Generation starts from a beginning-of-sequence marker, just as many real models prepend a BOS token.
2. Line 7: The loop runs while the last token is not the end marker and a length cap has not been hit; the cap plays the role of max_new_tokens.
3. Line 9: Greedy decoding takes the highest score: after <s>, I (0.9) beats you (0.1).
4. Line 10: The chosen token is appended, so the next step conditions on it.
5. Line 9: Now the last token is I, and love (0.7) beats eat (0.3). Each iteration only looks at the newest context.
6. Line 9: After love is appended, chai (0.6) narrowly beats </s> (0.4). Greedy never looks ahead; it just takes the local best.
7. Line 9: After chai is appended, </s> (0.8) beats too (0.2).
8. Line 10: The end marker is appended like any other token.
9. Line 7: The condition sees </s> as the last token, so the loop stops.
10. Line 11: The markers are stripped before printing, like skip_special_tokens=True in decode.

generate() with a small causal LM

Slice off the prompt tokens so only the newly generated text is decoded.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "openai-community/gpt2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tok("The best thing about studying in Pune is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=25, do_sample=False,
                     pad_token_id=tok.eos_token_id)

new_tokens = out[0, inputs["input_ids"].shape[1]:]
print(tok.decode(new_tokens, skip_special_tokens=True))

त्वरित जाँच: Why does `model.generate()` keep a KV cache?

• To store the vocabulary on the GPU
• To remember previous conversations across program runs
• To make sampling random
• To avoid recomputing keys and values for earlier tokens at every step