Vision Encoders: CNNs and Vision Transformers

Two ways to turn pixels into features

An encoder turns raw input into a compact set of feature vectors. For images there are two main families. A convolutional neural network (CNN), such as ResNet, slides small learned filters across the image; early layers detect edges and textures, deeper layers detect parts and objects. CNNs build in the assumption that nearby pixels matter most, which makes them data-efficient. A Vision Transformer (ViT) cuts the image into fixed-size patches, turns each patch into a vector, and runs a standard transformer over the sequence. It has fewer built-in assumptions, so it usually needs more training data, but it scales well and, crucially, it outputs a sequence of tokens. That sequence plugs directly into a language model or a cross-attention layer, which is why most current vision-language models, including CLIP variants and LLaVA-style models, use a ViT as the vision encoder.

Inside a ViT encoder

A ViT cuts the image into patches, projects each patch to a vector, adds position information and runs transformer layers to produce one output token per patch.

Reading a newspaper

A CNN reads a newspaper with a magnifying glass, letter by letter, slowly building up words and headlines. A ViT cuts the page into squares and lets every square compare itself with every other square at once, which is costlier but sees the whole layout together.

Quick check: What property of a ViT's output makes it easy to connect to a language model?

• It outputs a single class label
• It outputs pixels at higher resolution
• It outputs a sequence of token vectors
• It outputs text directly