Open Models with Hugging Face

Processors, models and pipelines

The Hugging Face transformers library hosts most open multimodal checkpoints. Three objects matter. A processor bundles everything that prepares inputs for a specific checkpoint: the image processor (resize, crop, normalise), the tokenizer, and for chat models the chat template that inserts image placeholders in the right place. A model class holds the weights; AutoModel... classes pick the correct architecture from the checkpoint's config. A pipeline wraps both into one call for a named task such as zero-shot-image-classification, automatic-speech-recognition or image-text-to-text. Pipelines are ideal for prototypes; for production you usually call the processor and model directly so you control batching, device placement and generation settings. Always load the processor from the same checkpoint as the model, since mismatched preprocessing silently degrades results. Weights download on first use and are cached locally, so check model size against your disk and GPU memory.

Asking a VLM about an image

A chat-style call to an open VLM through the image-text-to-text pipeline. It downloads several gigabytes, so the output is not shown.

from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen2.5-VL-3B-Instruct")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "https://example.com/bus-timetable.jpg"},
        {"type": "text", "text": "When does the last bus to Pune leave?"},
    ],
}]
out = pipe(text=messages, max_new_tokens=80)
print(out[0]["generated_text"][-1]["content"])

Read the model card first

The model card lists the expected prompt format, supported resolutions, licence and known limits. Licences vary: some open-weight models restrict commercial use. Check before you ship.

त्वरित जाँच: What does a Hugging Face processor do for a multimodal checkpoint?

• Prepares images, text and chat templates exactly as that checkpoint expects
• Trains the model on new data
• Converts the model to another framework
• Hosts the model on a remote server