The Mental Model: Tokens, Weights and Inference

Next-token prediction in a loop

An LLM does one thing: given a sequence of tokens (sub-word pieces, roughly three-quarters of an English word on average), it outputs a probability for every token in its vocabulary being next. Inference has two phases. In prefill, the whole prompt is processed in one batched pass and the model stores each token's attention keys and values in the KV cache. In decode, the model produces one token at a time: a sampler picks a token from the probabilities, that token is appended, and the next forward pass reuses the KV cache instead of re-reading the whole prompt. The loop stops at an end-of-sequence token, a stop string, or a token limit. This explains most behaviour you will see later: long prompts slow the first token (prefill), every generated token costs another pass (decode), and the context window caps how many tokens fit in the cache.

Autocomplete on steroids

Your phone keyboard suggests the next word after "See you at the". An LLM is the same idea scaled up: billions of weights, a vocabulary of tens of thousands of tokens, and it keeps choosing and appending until the answer is finished. It is not looking things up in a database; it is predicting.

Prefill vs decode is an interview favourite

If someone asks why the first token is slow but later ones stream quickly, the answer is prefill: the whole prompt must be processed before the first output token. Decode speed (tokens per second) is a separate number, mostly limited by memory bandwidth. Benchmarks report them separately for this reason.

Quick check: What is the main purpose of the KV cache during generation?

• To store the downloaded model file on disk
• To cache HTTP responses from the server
• To keep attention keys and values for earlier tokens so each new token does not reprocess the whole sequence
• To save the user's chat history between sessions