# Vision Encoders: CNNs and Vision Transformers — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-cnn-vs-vit

> Contrast CNN and ViT image encoders and explain why ViTs became the default in vision-language models.

## Two ways to turn pixels into features

An **encoder** turns raw input into a compact set of feature vectors. For images there are two main families. A **convolutional neural network (CNN)**, such as ResNet, slides small learned filters across the image; early layers detect edges and textures, deeper layers detect parts and objects. CNNs build in the assumption that nearby pixels matter most, which makes them data-efficient. A **Vision Transformer (ViT)** cuts the image into fixed-size patches, turns each patch into a vector, and runs a standard transformer over the sequence. It has fewer built-in assumptions, so it usually needs more training data, but it scales well and, crucially, it outputs a **sequence of tokens**. That sequence plugs directly into a language model or a cross-attention layer, which is why most current vision-language models, including CLIP variants and LLaVA-style models, use a ViT as the vision encoder.

## Inside a ViT encoder

A ViT cuts the image into patches, projects each patch to a vector, adds position information and runs transformer layers to produce one output token per patch.

![Four connected stages: image, patches, patch embeddings with positions, transformer output tokens.](assets/figures/multimodal-ai/section-3-map.svg) — Figure 3.1: image -> fixed-size patches -> patch embeddings + position -> transformer layers producing output tokens.

## Reading a newspaper

A CNN reads a newspaper with a magnifying glass, letter by letter, slowly building up words and headlines. A ViT cuts the page into squares and lets every square compare itself with every other square at once, which is costlier but sees the whole layout together.

**Quiz:** What property of a ViT's output makes it easy to connect to a language model?

- [ ] It outputs a single class label
- [ ] It outputs pixels at higher resolution
- [x] It outputs a sequence of token vectors
- [ ] It outputs text directly

*Answer:* It outputs a sequence of token vectors. A ViT produces one vector per patch, a token sequence that a projector can map into the language model's embedding space.