# The Mental Model: Tokens, Weights and Inference — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-mental-model

> Describe how a model turns a prompt into tokens, runs a forward pass and samples the next token in a loop.

## Next-token prediction in a loop

An LLM does one thing: given a sequence of **tokens** (sub-word pieces, roughly three-quarters of an English word on average), it outputs a probability for every token in its vocabulary being next. Inference has two phases. In **prefill**, the whole prompt is processed in one batched pass and the model stores each token's attention keys and values in the **KV cache**. In **decode**, the model produces one token at a time: a **sampler** picks a token from the probabilities, that token is appended, and the next forward pass reuses the KV cache instead of re-reading the whole prompt. The loop stops at an end-of-sequence token, a stop string, or a token limit. This explains most behaviour you will see later: long prompts slow the first token (prefill), every generated token costs another pass (decode), and the context window caps how many tokens fit in the cache.

## Autocomplete on steroids

Your phone keyboard suggests the next word after "See you at the". An LLM is the same idea scaled up: billions of weights, a vocabulary of tens of thousands of tokens, and it keeps choosing and appending until the answer is finished. It is not looking things up in a database; it is predicting.

## Prefill vs decode is an interview favourite

If someone asks why the first token is slow but later ones stream quickly, the answer is prefill: the whole prompt must be processed before the first output token. Decode speed (tokens per second) is a separate number, mostly limited by memory bandwidth. Benchmarks report them separately for this reason.

**Quiz:** What is the main purpose of the KV cache during generation?

- [ ] To store the downloaded model file on disk
- [ ] To cache HTTP responses from the server
- [x] To keep attention keys and values for earlier tokens so each new token does not reprocess the whole sequence
- [ ] To save the user's chat history between sessions

*Answer:* To keep attention keys and values for earlier tokens so each new token does not reprocess the whole sequence. The KV cache holds per-token attention keys and values, so decode only computes the new token. It grows with context length, which is why long contexts need more memory.