# Common Interview and Exam Questions — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-interview

> Answer the local-LLM questions interviewers ask most, with crisp reasoning instead of buzzwords.

## Questions you should be able to answer aloud

**Why run locally?** Privacy, cost at volume, offline use and version control, at the price of capability and operations work. **What is quantization and its trade-off?** Storing weights in fewer bits to cut memory and raise decode speed, with some quality loss that grows below about 4 bits. **Why is the first token slow?** Prefill must process the entire prompt first. **What limits tokens per second?** Mostly memory bandwidth, since every token reads all weights. **Why does memory grow with context?** The KV cache stores keys and values per token per layer. **What is GGUF?** A single-file format with tensors and metadata for llama.cpp-based runtimes. **How do you guarantee JSON output?** Constrained decoding with a grammar or JSON schema, then validate. **How does tool calling work?** The model emits a structured call; your code executes it and returns the result. **How do you secure Ollama?** Keep it on localhost, and add auth via a proxy or tunnel.

## Answer with a number when you can

"An 8B model at 4-bit is about 5 GB, so it fits on an 8 GB GPU with room for a few thousand tokens of context" is far stronger than "it should fit". Practise the weights and KV-cache estimates until you can do them on paper.

## A classic trick question

"Will adding more CPU threads always speed up generation?" The expected answer is no: decode is bandwidth bound, so beyond the physical core count, extra threads add contention. Prefill, being compute heavy, benefits more from cores and GPUs.

## Explaining it to a non-technical relative

Think of the model as a very well-read friend who only remembers what fits on the notepad in front of them (the context). Quantization is writing their knowledge in shorthand so the book fits in a smaller bag, and a faster GPU is how quickly they can flip through that book for every word they say.

**Quiz:** An interviewer asks why a 70B model at 4-bit runs slowly on a 24 GB GPU even though inference "works". What is the best explanation?

- [ ] 70B models cannot be quantized
- [ ] The tokenizer is larger for big models
- [x] At roughly 40 GB the weights do not fit in 24 GB of VRAM, so many layers run from system RAM
- [ ] The GPU automatically lowers its clock for large models

*Answer:* At roughly 40 GB the weights do not fit in 24 GB of VRAM, so many layers run from system RAM. 70B x about 4.8 bits / 8 is roughly 40 GB. The overflow lives in system RAM, and those layers bottleneck every token.