# Running Local LLMs in Docker and on a Team Server — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-docker-deploy

> Run Ollama in Docker with GPU access and persistent storage, and share one GPU box with a team.

## From laptop to a shared GPU box

Once a prototype works, teams often move inference to one machine with a good GPU and let everyone call it. The official `ollama/ollama` container image makes this repeatable. Three details matter. First, **persistence**: models are large, so mount a volume at `/root/.ollama` or every container restart re-downloads gigabytes. Second, **GPU access**: on Linux with NVIDIA, install the NVIDIA Container Toolkit and pass `--gpus=all`. Leave the flag out and the container quietly runs on CPU; pass it without the toolkit and Docker refuses to start the container. Third, **networking**: publishing port 11434 makes the API reachable from other machines, which is the goal but also the risk covered in the next topic. A web front end such as Open WebUI can sit beside it for non-programmers. For a single fixed model with tight control, a container running `llama-server` is an equally valid design. Either way, pin model tags, so a re-pull of `latest` does not change behaviour under your users.

## Ollama in Docker with a GPU and a volume

Requires Docker and, for the GPU flag, the NVIDIA Container Toolkit.

```bash
docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama

# Pull and test a model inside the container
docker exec -it ollama ollama run llama3.2 "Say hello"
```

## A small team deployment

Requests pass through an authenticating proxy before they reach the model server. The stages, in order: team clients -> reverse proxy with TLS and auth -> Ollama or llama-server container -> GPU and model volume.

![4 connected stages, left to right: team clients, reverse proxy with TLS and auth, Ollama or llama-server container, GPU and model volume.](assets/figures/local-llms/section-9-map.svg) — Figure 9.1: team clients -> reverse proxy with TLS and auth -> Ollama or llama-server container -> GPU and model volume

## Bind to localhost first

Notice `-p 127.0.0.1:11434:11434` above: the port is published only on the host's loopback interface. Open it wider only behind a proxy that adds authentication. A plain `-p 11434:11434` exposes the API to the whole network.

**Quiz:** Why mount a volume at /root/.ollama when running Ollama in Docker?

- [ ] It enables the GPU
- [ ] It is required for the OpenAI-compatible API
- [ ] It encrypts the prompts
- [x] So downloaded models persist across container restarts instead of being re-downloaded

*Answer:* So downloaded models persist across container restarts instead of being re-downloaded. Models live in that directory. Without a volume they vanish with the container and must be pulled again.