Securing Local Inference

Local does not mean safe by default

The Ollama API has no built-in authentication. By default it listens only on 127.0.0.1, which is safe, but setting OLLAMA_HOST=0.0.0.0 makes it reachable by anyone on the network, and internet scans regularly find such servers left open. An open server lets strangers use your GPU, pull or delete models, and read anything your prompts expose. Protect it with a firewall, an SSH tunnel, or a reverse proxy that adds TLS and authentication; llama-server can at least require a key with --api-key. Browser access is governed by OLLAMA_ORIGINS; avoid allowing every origin. Next, prompt injection applies locally too: text inside retrieved documents or web pages can instruct the model to misuse tools, so tools must be allow-listed, arguments validated and destructive actions confirmed by a human. Finally, treat model files as supply chain: download from sources you trust and keep runtimes updated, since file parsers can have security bugs.

Reach a remote server safely over SSH

The server keeps listening on localhost; the tunnel forwards your local port to it.

# On your laptop: forward local 11434 to the GPU box's localhost:11434
ssh -N -L 11434:localhost:11434 student@gpu-box

# Now local clients work as if the server were on your machine
curl http://localhost:11434/api/tags

# For llama-server, require a key
./build/bin/llama-server -m model.gguf --api-key "$LLAMA_API_KEY"

Never expose the raw port to the internet

Opening port 11434 on a cloud VM's security group or a home router is the single most common mistake. If people outside your machine need access, put an authenticating proxy or VPN in front, and log who calls what.

त्वरित जाँच: What is the main risk of starting Ollama with OLLAMA_HOST=0.0.0.0 on a cloud VM with an open port?

• Anyone who can reach the port can use the API, because Ollama has no built-in authentication
• The model will run only on CPU
• Generation becomes deterministic
• Models are automatically uploaded to the public library