# Open Models with Hugging Face — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-hf-tooling

> Run open vision-language and audio models with transformers pipelines and processors.

## Processors, models and pipelines

The Hugging Face **transformers** library hosts most open multimodal checkpoints. Three objects matter. A **processor** bundles everything that prepares inputs for a specific checkpoint: the image processor (resize, crop, normalise), the tokenizer, and for chat models the **chat template** that inserts image placeholders in the right place. A **model** class holds the weights; `AutoModel...` classes pick the correct architecture from the checkpoint's config. A **pipeline** wraps both into one call for a named task such as `zero-shot-image-classification`, `automatic-speech-recognition` or `image-text-to-text`. Pipelines are ideal for prototypes; for production you usually call the processor and model directly so you control batching, device placement and generation settings. Always load the processor from the same checkpoint as the model, since mismatched preprocessing silently degrades results. Weights download on first use and are cached locally, so check model size against your disk and GPU memory.

## Asking a VLM about an image

A chat-style call to an open VLM through the image-text-to-text pipeline. It downloads several gigabytes, so the output is not shown.

```python
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen2.5-VL-3B-Instruct")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "https://example.com/bus-timetable.jpg"},
        {"type": "text", "text": "When does the last bus to Pune leave?"},
    ],
}]
out = pipe(text=messages, max_new_tokens=80)
print(out[0]["generated_text"][-1]["content"])
```

## Read the model card first

The model card lists the expected prompt format, supported resolutions, licence and known limits. Licences vary: some open-weight models restrict commercial use. Check before you ship.

**Quiz:** What does a Hugging Face processor do for a multimodal checkpoint?

- [x] Prepares images, text and chat templates exactly as that checkpoint expects
- [ ] Trains the model on new data
- [ ] Converts the model to another framework
- [ ] Hosts the model on a remote server

*Answer:* Prepares images, text and chat templates exactly as that checkpoint expects. The processor bundles the image preprocessing, tokenizer and chat template matched to the checkpoint.