Vision-Language Model Architecture

Encoder, projector, language model

A vision-language model (VLM) answers free-form questions about images by generating text. The most common recipe, popularised by LLaVA, has three parts. A pretrained vision encoder, usually a CLIP-style ViT, turns the image into patch vectors. A small projector, a linear layer or a two-layer MLP, maps each patch vector into the embedding space of a language model. The projected vectors are then inserted into the prompt as if they were word embeddings, and a pretrained LLM reads text and image tokens together with ordinary self-attention and generates the answer. From the LLM's point of view an image is just a few hundred extra tokens. This design is popular because it reuses two expensive pretrained models and only needs to train the projector (and optionally fine-tune the LLM). Its cost is the token count: every image consumes context space and attention compute alongside the text.

A LLaVA-style VLM

Image patches flow through a frozen vision encoder and a trained projector, then join the text tokens inside the language model.

What the LLM actually receives

Conceptually the prompt is a single sequence. The placeholder is replaced by the projected image vectors before the LLM runs.

# Simplified view of a VLM input sequence
prompt = "USER: <image>\nWhat is the price on this menu board? ASSISTANT:"

# Inside the model:
# text_embeds  = llm.embed(tokenize(prompt))           # (T, d_llm)
# patch_feats  = vision_encoder(pixel_values)          # (576, d_vision)
# image_embeds = projector(patch_feats)                # (576, d_llm)
# sequence     = replace("<image>", image_embeds, text_embeds)
# answer       = llm.generate(inputs_embeds=sequence)

Chat templates matter

Each VLM expects its own prompt format and image placeholder. With Hugging Face, use the processor's apply_chat_template instead of hand-writing <image> strings; a wrong template often produces fluent but irrelevant answers rather than an error.

त्वरित जाँच: In a LLaVA-style VLM, what is the job of the projector?

• Map vision-encoder outputs into the language model's embedding space
• Compress the image to JPEG
• Generate the final answer text
• Translate text tokens into pixels