Running Multimodal Models Locally

Why and how to go local

Running a VLM locally keeps images on your machine, removes per-request API fees and works offline, which matters for private documents or field apps with poor connectivity. Tools such as Ollama and llama.cpp run quantised models on laptops and desktops. A llama.cpp-style vision model typically has two files: the language model weights and a separate multimodal projector (often called mmproj) containing the vision encoder and projector; both must match. Ollama hides this and lets you pull a vision model by name, then pass image file paths with your prompt; it also exposes an OpenAI-compatible endpoint, so code written for a hosted API can often point at localhost. The costs are on your side: memory for weights plus the image tokens in context, and slower generation than a data-centre GPU. Small local VLMs are good at captioning and simple questions, and noticeably weaker at dense text and multi-step visual reasoning.

Pull and query a local vision model

The first command downloads the model; the second asks about a local image. Output depends on the image, so it is not shown.

ollama pull llava
ollama run llava "Describe this photo in one sentence: ./market.jpg"

The same from Python

The ollama Python client accepts image paths (or bytes) in an images list on the message.

import ollama

response = ollama.chat(
    model="llava",
    messages=[{
        "role": "user",
        "content": "List the vegetables visible in this photo.",
        "images": ["./market.jpg"],
    }],
)
print(response["message"]["content"])

Home kitchen vs restaurant

A hosted API is a restaurant: the best chefs and equipment, but you pay per plate and the food leaves your house. A local model is your home kitchen: simpler dishes, slower, but nothing leaves home and the marginal cost is almost zero.

Quick check: In llama.cpp-style deployments of a vision model, what is the separate mmproj file?

• A config file for the web UI
• The vision encoder and projector that turn images into tokens for the LLM
• A dataset of example images
• A licence file