# Running Multimodal Models Locally — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-local-vlms

> Run a vision model on your own machine with Ollama and know what local deployment needs.

## Why and how to go local

Running a VLM locally keeps images on your machine, removes per-request API fees and works offline, which matters for private documents or field apps with poor connectivity. Tools such as **Ollama** and **llama.cpp** run quantised models on laptops and desktops. A llama.cpp-style vision model typically has two files: the language model weights and a separate **multimodal projector** (often called mmproj) containing the vision encoder and projector; both must match. Ollama hides this and lets you pull a vision model by name, then pass image file paths with your prompt; it also exposes an OpenAI-compatible endpoint, so code written for a hosted API can often point at localhost. The costs are on your side: memory for weights plus the image tokens in context, and slower generation than a data-centre GPU. Small local VLMs are good at captioning and simple questions, and noticeably weaker at dense text and multi-step visual reasoning.

## Pull and query a local vision model

The first command downloads the model; the second asks about a local image. Output depends on the image, so it is not shown.

```bash
ollama pull llava
ollama run llava "Describe this photo in one sentence: ./market.jpg"
```

## The same from Python

The ollama Python client accepts image paths (or bytes) in an `images` list on the message.

```python
import ollama

response = ollama.chat(
    model="llava",
    messages=[{
        "role": "user",
        "content": "List the vegetables visible in this photo.",
        "images": ["./market.jpg"],
    }],
)
print(response["message"]["content"])
```

## Home kitchen vs restaurant

A hosted API is a restaurant: the best chefs and equipment, but you pay per plate and the food leaves your house. A local model is your home kitchen: simpler dishes, slower, but nothing leaves home and the marginal cost is almost zero.

**Quiz:** In llama.cpp-style deployments of a vision model, what is the separate mmproj file?

- [ ] A config file for the web UI
- [x] The vision encoder and projector that turn images into tokens for the LLM
- [ ] A dataset of example images
- [ ] A licence file

*Answer:* The vision encoder and projector that turn images into tokens for the LLM. The multimodal projector file holds the vision side; it must match the language model weights.