Cheat Sheet: Local LLMs in One Page

Everything on one page

Inference: prefill processes the prompt; decode generates one token per pass using the KV cache; decode speed is bounded by memory bandwidth divided by model size. Memory: weights = params x bits / 8; KV cache = 2 x layers x kv_heads x head_dim x context x bytes; add buffers and headroom. Files: GGUF holds weights plus metadata and chat template; Q4_K_M is the usual default, Q8_0 near-lossless. Ollama: server on 127.0.0.1:11434; pull, run, list, ps, show, stop, rm, create -f Modelfile; endpoints /api/chat, /api/generate, /api/embed, OpenAI-style /v1; streams NDJSON by default; tuning through OLLAMA_* variables. llama.cpp: CMake build; llama-cli, llama-server, llama-quantize, llama-bench; flags -m, -hf, -c, -ngl, -t, -np, --jinja. Apps: JSON schemas constrain shape, RAG grounds answers, tools are executed by your code. Safety: no built-in auth in Ollama, so never expose the raw port.

Command crib

The commands from this course in one block.

ollama pull qwen2.5:7b && ollama run qwen2.5:7b "hi"
ollama ps                                 # memory and CPU/GPU split
ollama create mybot -f Modelfile          # FROM / SYSTEM / PARAMETER
curl localhost:11434/api/chat -d '{"model":"qwen2.5:7b","messages":[{"role":"user","content":"hi"}],"stream":false}'

llama-server -m model.gguf -c 8192 -ngl 99 -np 2 --jinja --port 8080
llama-bench -m model.gguf -ngl 0,99
llama-quantize in-f16.gguf out-Q4_K_M.gguf Q4_K_M

Three numbers to always state

When describing any local setup in an interview or a design doc, state the model and quantization, the context length, and where it runs (full GPU, split, or CPU). Those three explain almost every speed and memory question that follows.

त्वरित जाँच: Which pair correctly matches tool and default address?

• Ollama at 127.0.0.1:11434, llama-server at 127.0.0.1:8080
• Ollama at 0.0.0.0:8080, llama-server at 127.0.0.1:11434
• Both on 0.0.0.0:80
• Ollama at 127.0.0.1:5000, llama-server at 127.0.0.1:3000