# OpenAI-Compatible Endpoints — Local LLMs with Ollama & llama.cpp

Source: https://www.skillbyai.com/en/local-llms/ll-openai-compat

> Point existing OpenAI-SDK code at Ollama or llama-server by changing the base URL.

## Same client, different base URL

Many tools and libraries are written against the OpenAI Chat Completions API. Both Ollama and llama.cpp's `llama-server` expose **OpenAI-compatible endpoints** under `/v1`, such as `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings` and `/v1/models`. That means you can take code that uses the official `openai` SDK, set `base_url` to `http://localhost:11434/v1` (Ollama) or `http://localhost:8080/v1` (llama-server), and use a local model name. The SDK requires an API key string, but Ollama ignores its value, so any placeholder such as `"ollama"` works; llama-server checks it only if started with `--api-key`. Compatibility covers the common features (messages, streaming, temperature, tools, JSON mode) but is not complete: some parameters are ignored or behave differently, so test the features you rely on. This compatibility is what lets frameworks swap between hosted and local models with one configuration change.

## The openai SDK talking to Ollama

Requires `pip install openai` and a running Ollama server.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key ignored by Ollama

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Name three sorting algorithms."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)
```

## A universal charger

A USB-C phone charger works with any USB-C phone, whichever brand made it. The OpenAI-style API has become that common plug: write your app to it once, and you can plug in a hosted model or your local Ollama box.

**Quiz:** You have working code using the openai SDK. What is the minimum change to run it against local Ollama?

- [ ] Rewrite it with raw sockets
- [ ] Convert the model to safetensors
- [x] Set base_url to http://localhost:11434/v1, use a local model name, and pass any placeholder api_key
- [ ] Install CUDA on the client machine

*Answer:* Set base_url to http://localhost:11434/v1, use a local model name, and pass any placeholder api_key. Ollama serves an OpenAI-compatible API under /v1. The SDK needs some api_key value, which Ollama ignores.