# Vision-Language Model Architecture — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-vlm-architecture

> Describe the encoder, projector and language model in a LLaVA-style VLM and how image tokens enter the prompt.

## Encoder, projector, language model

A **vision-language model (VLM)** answers free-form questions about images by generating text. The most common recipe, popularised by **LLaVA**, has three parts. A pretrained **vision encoder**, usually a CLIP-style ViT, turns the image into patch vectors. A small **projector**, a linear layer or a two-layer MLP, maps each patch vector into the **embedding space of a language model**. The projected vectors are then inserted into the prompt as if they were word embeddings, and a pretrained **LLM** reads text and image tokens together with ordinary self-attention and generates the answer. From the LLM's point of view an image is just a few hundred extra tokens. This design is popular because it reuses two expensive pretrained models and only needs to train the projector (and optionally fine-tune the LLM). Its cost is the token count: every image consumes context space and attention compute alongside the text.

## A LLaVA-style VLM

Image patches flow through a frozen vision encoder and a trained projector, then join the text tokens inside the language model.

![Four connected stages: image, vision encoder, projector, language model generating text.](assets/figures/multimodal-ai/section-5-map.svg) — Figure 5.1: image -> vision encoder (ViT) -> projector (MLP) -> language model reading image tokens + text tokens and generating the answer.

## What the LLM actually receives

Conceptually the prompt is a single sequence. The placeholder is replaced by the projected image vectors before the LLM runs.

```python
# Simplified view of a VLM input sequence
prompt = "USER: <image>\nWhat is the price on this menu board? ASSISTANT:"

# Inside the model:
# text_embeds  = llm.embed(tokenize(prompt))           # (T, d_llm)
# patch_feats  = vision_encoder(pixel_values)          # (576, d_vision)
# image_embeds = projector(patch_feats)                # (576, d_llm)
# sequence     = replace("<image>", image_embeds, text_embeds)
# answer       = llm.generate(inputs_embeds=sequence)
```

## Chat templates matter

Each VLM expects its own prompt format and image placeholder. With Hugging Face, use the processor's `apply_chat_template` instead of hand-writing `<image>` strings; a wrong template often produces fluent but irrelevant answers rather than an error.

**Quiz:** In a LLaVA-style VLM, what is the job of the projector?

- [x] Map vision-encoder outputs into the language model's embedding space
- [ ] Compress the image to JPEG
- [ ] Generate the final answer text
- [ ] Translate text tokens into pixels

*Answer:* Map vision-encoder outputs into the language model's embedding space. The projector converts patch vectors to the dimensionality and space the LLM's token embeddings use.