# What Local LLMs Are and Why They Matter — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-why-local

> Explain what running a model locally means and judge when it beats a hosted API on privacy, cost, latency and control.

## Your laptop as the inference server

A **large language model (LLM)** is a file of learned numbers (the *weights*) plus a program that turns your text into tokens, runs them through those weights and predicts the next token, again and again. With a hosted API, that program runs in someone else's data centre and you pay per token. **Running locally** means the weights sit on your own disk and the inference program runs on your own CPU or GPU. The upside: prompts never leave your machine, there is no per-token bill, it works offline, and you choose the exact model version, so it never changes under you. The trade-off: local models are usually smaller than frontier hosted models, so they reason less well on hard tasks, and you become responsible for hardware, memory limits, updates and security. Local is a tool for the right jobs, not a replacement for everything.

## The local inference path

Everything in this course is some part of this path, and every part runs on your own hardware. The stages, in order: your prompt -> tokenizer -> model weights on your CPU/GPU -> generated tokens -> your application.

![5 connected stages, left to right: your prompt, tokenizer, model weights on your CPU/GPU, generated tokens, your application.](assets/figures/local-llms/section-1-map.svg) — Figure 1.1: your prompt -> tokenizer -> model weights on your CPU/GPU -> generated tokens -> your application

## Cooking at home vs ordering in

Ordering from a restaurant app (a hosted API) gives you a top chef with no setup, but every meal costs money and the restaurant sees your order. Cooking at home (a local LLM) needs a kitchen and some skill, the menu is simpler, but it is cheap per meal, private, and you can cook at 2 a.m. without asking anyone.

## When local is the right call

Good fits: sensitive data (medical notes, company code), offline or air-gapped environments, high-volume simple tasks like classification or extraction, and learning how LLMs really work. Poor fits: tasks that need the strongest available reasoning, or teams with no one to own the hardware. Many real systems mix both: a local model for cheap, private steps and a hosted model for the hard ones.

**Quiz:** Which is a genuine advantage of running an LLM locally rather than through a hosted API?

- [ ] Local models are always more accurate than hosted frontier models
- [x] Prompts and data never have to leave your machine
- [ ] You no longer need any RAM or VRAM planning
- [ ] Local models never need updating

*Answer:* Prompts and data never have to leave your machine. Privacy and control are the core advantages. Local models are typically smaller and less capable than frontier hosted models, and hardware planning becomes your job.