OpenAI-Compatible Endpoints

Same client, different base URL

Many tools and libraries are written against the OpenAI Chat Completions API. Both Ollama and llama.cpp's llama-server expose OpenAI-compatible endpoints under /v1, such as /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models. That means you can take code that uses the official openai SDK, set base_url to http://localhost:11434/v1 (Ollama) or http://localhost:8080/v1 (llama-server), and use a local model name. The SDK requires an API key string, but Ollama ignores its value, so any placeholder such as "ollama" works; llama-server checks it only if started with --api-key. Compatibility covers the common features (messages, streaming, temperature, tools, JSON mode) but is not complete: some parameters are ignored or behave differently, so test the features you rely on. This compatibility is what lets frameworks swap between hosted and local models with one configuration change.

The openai SDK talking to Ollama

Requires pip install openai and a running Ollama server.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key ignored by Ollama

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Name three sorting algorithms."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)

A universal charger

A USB-C phone charger works with any USB-C phone, whichever brand made it. The OpenAI-style API has become that common plug: write your app to it once, and you can plug in a hosted model or your local Ollama box.

त्वरित जाँच: You have working code using the openai SDK. What is the minimum change to run it against local Ollama?

• Rewrite it with raw sockets
• Convert the model to safetensors
• Set base_url to http://localhost:11434/v1, use a local model name, and pass any placeholder api_key
• Install CUDA on the client machine