Lesson 25 / 30

Running Local LLMs in Docker and on a Team Server

Name: Local LLMs with Ollama & llama.cpp
Author: Rahul Agarwal

Run Ollama in Docker with GPU access and persistent storage, and share one GPU box with a team.

By Rahul Agarwal · 15 years in web development

From laptop to a shared GPU box

Once a prototype works, teams often move inference to one machine with a good GPU and let everyone call it. The official ollama/ollama container image makes this repeatable. Three details matter. First, persistence: models are large, so mount a volume at /root/.ollama or every container restart re-downloads gigabytes. Second, GPU access: on Linux with NVIDIA, install the NVIDIA Container Toolkit and pass --gpus=all. Leave the flag out and the container quietly runs on CPU; pass it without the toolkit and Docker refuses to start the container. Third, networking: publishing port 11434 makes the API reachable from other machines, which is the goal but also the risk covered in the next topic. A web front end such as Open WebUI can sit beside it for non-programmers. For a single fixed model with tight control, a container running llama-server is an equally valid design. Either way, pin model tags, so a re-pull of latest does not change behaviour under your users.

Ollama in Docker with a GPU and a volume

Requires Docker and, for the GPU flag, the NVIDIA Container Toolkit.

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama

# Pull and test a model inside the container
docker exec -it ollama ollama run llama3.2 "Say hello"

A small team deployment

Requests pass through an authenticating proxy before they reach the model server. The stages, in order: team clients -> reverse proxy with TLS and auth -> Ollama or llama-server container -> GPU and model volume.

4 connected stages, left to right: team clients, reverse proxy with TLS and auth, Ollama or llama-server container, GPU and model volume. — Figure 9.1: team clients -> reverse proxy with TLS and auth -> Ollama or llama-server container -> GPU and model volume

Bind to localhost first

Notice -p 127.0.0.1:11434:11434 above: the port is published only on the host's loopback interface. Open it wider only behind a proxy that adds authentication. A plain -p 11434:11434 exposes the API to the whole network.

Quick check: Why mount a volume at /root/.ollama when running Ollama in Docker?

It enables the GPU
It is required for the OpenAI-compatible API
It encrypts the prompts
So downloaded models persist across container restarts instead of being re-downloaded

Answer

So downloaded models persist across container restarts instead of being re-downloaded — Models live in that directory. Without a volume they vanish with the container and must be pulled again.