Local LLMs with Ollama & llama.cpp

What you'll learn

• Explain how local inference works: tokens, prefill and decode, the KV cache and why memory bandwidth limits speed.
• Choose a model family, size and GGUF quantization level that fits your RAM or VRAM and your task.
• Run and customize models with the Ollama CLI, Modelfiles, sampling parameters and chat templates.
• Call local models from code through the Ollama REST API, its Python library and OpenAI-compatible endpoints.
• Build and serve models directly with llama.cpp, llama-cli and llama-server.
• Build structured-output, RAG and tool-calling applications that run fully offline.
• Tune memory, GPU offload and concurrency, and deploy a local LLM server securely.

Syllabus

Why Run LLMs Locally

1. What Local LLMs Are and Why They Matter
2. The Mental Model: Tokens, Weights and Inference
3. Hardware Basics: RAM, VRAM and Bandwidth

Model Files, GGUF and Quantization

1. Weights, Parameters and Model Families
2. The GGUF File Format
3. Quantization Levels and Trade-offs

Getting Started with Ollama

1. Installing Ollama and the Server Model
2. Everyday Ollama CLI Commands
3. Choosing Models from the Library

Customizing Models with Modelfiles

1. Modelfile Basics: FROM, SYSTEM and PARAMETER
2. Sampling Parameters: Temperature, top_p and Friends
3. Chat Templates and Special Tokens

Programming Against the Ollama API

1. The REST API: generate, chat and streaming
2. The Official Python Library
3. OpenAI-Compatible Endpoints

Working Directly with llama.cpp

1. What llama.cpp Is and How to Build It
2. llama-cli and the Core Flags
3. llama-server: HTTP, Web UI and Parallel Slots

Building Applications on Local Models

1. Structured Output with JSON Schemas
2. Embeddings and Local RAG
3. Tool Calling with Local Models

Performance Tuning and Memory

1. Memory Math: Weights plus KV Cache
2. GPU Offload and CPU/GPU Splits
3. Context, Keep-Alive and Concurrency Settings

Deployment, Security and Pitfalls

1. Running Local LLMs in Docker and on a Team Server
2. Securing Local Inference
3. Common Pitfalls and Evaluating Quality

Revision and Interview Prep

1. Cheat Sheet: Local LLMs in One Page
2. Common Interview and Exam Questions
3. Mixed Self-Check

Related courses

• LLM APIs (27 lessons)
• LLMs (27 lessons)
• Agent Frameworks (25 lessons)
• Hugging Face Transformers (33 lessons)