How VLMs Are Trained

Align first, then teach to follow instructions

Training a LLaVA-style VLM usually happens in stages. Stage 1, alignment (feature pre-training): the vision encoder and LLM are frozen and only the projector is trained on image-caption pairs, so that projected image tokens start to mean something to the LLM. Stage 2, visual instruction tuning: the model is trained on conversations about images, questions, detailed descriptions, reasoning, OCR-style tasks, and the projector plus the LLM (fully or with adapters such as LoRA) are updated. The vision encoder is often kept frozen, though some recipes unfreeze it to improve fine detail. Instruction data is commonly a mix of human-written and model-generated conversations. Later stages may add preference tuning to reduce hallucination. Each stage changes fewer or more parameters for a reason: early on, unfreezing everything with a random projector would let noisy gradients damage the pretrained LLM.

Fine-tuning a VLM on a budget

For a domain task (reading electricity bills, grading answer sheets), start with a strong open VLM, freeze the vision encoder, and fine-tune with LoRA on a few thousand good examples. Data quality beats quantity here.

Training a new hire

Stage 1 is orientation: the new hire learns the vocabulary of the office. Stage 2 is on-the-job training with real tasks. You would not give someone live client calls on day one, and you do not unfreeze the LLM while the projector still outputs noise.

Quick check: In LLaVA-style stage-1 alignment, which component is usually trained?

• Only the vision encoder
• Only the projector
• Only the LLM's output layer
• All parameters from scratch