Common Interview and Exam Questions

Questions you should be able to answer aloud

Why run locally? Privacy, cost at volume, offline use and version control, at the price of capability and operations work. What is quantization and its trade-off? Storing weights in fewer bits to cut memory and raise decode speed, with some quality loss that grows below about 4 bits. Why is the first token slow? Prefill must process the entire prompt first. What limits tokens per second? Mostly memory bandwidth, since every token reads all weights. Why does memory grow with context? The KV cache stores keys and values per token per layer. What is GGUF? A single-file format with tensors and metadata for llama.cpp-based runtimes. How do you guarantee JSON output? Constrained decoding with a grammar or JSON schema, then validate. How does tool calling work? The model emits a structured call; your code executes it and returns the result. How do you secure Ollama? Keep it on localhost, and add auth via a proxy or tunnel.

Answer with a number when you can

"An 8B model at 4-bit is about 5 GB, so it fits on an 8 GB GPU with room for a few thousand tokens of context" is far stronger than "it should fit". Practise the weights and KV-cache estimates until you can do them on paper.

A classic trick question

"Will adding more CPU threads always speed up generation?" The expected answer is no: decode is bandwidth bound, so beyond the physical core count, extra threads add contention. Prefill, being compute heavy, benefits more from cores and GPUs.

Explaining it to a non-technical relative

Think of the model as a very well-read friend who only remembers what fits on the notepad in front of them (the context). Quantization is writing their knowledge in shorthand so the book fits in a smaller bag, and a faster GPU is how quickly they can flip through that book for every word they say.

Quick check: An interviewer asks why a 70B model at 4-bit runs slowly on a 24 GB GPU even though inference "works". What is the best explanation?

• 70B models cannot be quantized
• The tokenizer is larger for big models
• At roughly 40 GB the weights do not fit in 24 GB of VRAM, so many layers run from system RAM
• The GPU automatically lowers its clock for large models