Cheat Sheet: Multimodal AI in One Page

The whole course on one page

Modalities become tensors: images as HWC pixels converted to normalised CHW; audio as 16 kHz waveforms converted to log-mel spectrograms; video as sampled frames plus a transcript. Encoders turn tensors into token sequences: a ViT makes (H/P) x (W/P) patch tokens. Fusion can be early, late or intermediate. Dual encoders such as CLIP are trained with a symmetric contrastive (InfoNCE) loss over in-batch negatives, giving a shared space for zero-shot classification and retrieval via cosine similarity. VLMs connect a vision encoder to an LLM with a projector (concatenated image tokens) or with cross-attention and resamplers; training goes alignment first, then visual instruction tuning. Diffusion generates images by learning to predict added noise, usually in a VAE latent space, steered by text through cross-attention and classifier-free guidance. Production concerns: image token cost, caching and batching, eval sets with recall@k and WER, hallucination checks, injection-safe actions, and privacy.

The course as a pipeline

Reading the course end to end mirrors a real multimodal system, from raw data to a safe, measured product.

Formulas worth memorising

Five relations that answer a surprising number of exam and interview questions.

patch tokens     = (H / P) * (W / P)
cosine(a, b)     = (a . b) / (|a| * |b|)
CLIP loss        = (CE(rows) + CE(columns)) / 2  over sim / temperature
CFG noise        = eps_uncond + w * (eps_cond - eps_uncond)
WER              = (S + D + I) / N_reference_words

Quick check: Which pairing of model family and training objective is correct?

• CLIP - predicts added Gaussian noise
• Diffusion - symmetric contrastive loss over captions
• Whisper - zero-shot classification with text prompts
• CLIP - contrastive loss; diffusion - noise prediction