# Cheat Sheet: Multimodal AI in One Page — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-cheat-sheet

> Recall the core vocabulary, shapes and formulas of multimodal AI quickly before an exam or interview.

## The whole course on one page

**Modalities** become tensors: images as HWC pixels converted to normalised CHW; audio as 16 kHz waveforms converted to log-mel spectrograms; video as sampled frames plus a transcript. **Encoders** turn tensors into token sequences: a ViT makes (H/P) x (W/P) patch tokens. **Fusion** can be early, late or intermediate. **Dual encoders** such as CLIP are trained with a symmetric contrastive (InfoNCE) loss over in-batch negatives, giving a shared space for zero-shot classification and retrieval via cosine similarity. **VLMs** connect a vision encoder to an LLM with a projector (concatenated image tokens) or with cross-attention and resamplers; training goes alignment first, then visual instruction tuning. **Diffusion** generates images by learning to predict added noise, usually in a VAE latent space, steered by text through cross-attention and classifier-free guidance. **Production** concerns: image token cost, caching and batching, eval sets with recall@k and WER, hallucination checks, injection-safe actions, and privacy.

## The course as a pipeline

Reading the course end to end mirrors a real multimodal system, from raw data to a safe, measured product.

![Five connected stages: represent data, encode, align, generate or answer, ship safely.](assets/figures/multimodal-ai/section-11-map.svg) — Figure 11.1: represent data (tensors) -> encode (ViT, audio encoder) -> align (contrastive, projector) -> generate or answer (VLM, diffusion) -> ship safely (cost, eval, security).

## Formulas worth memorising

Five relations that answer a surprising number of exam and interview questions.

```text
patch tokens     = (H / P) * (W / P)
cosine(a, b)     = (a . b) / (|a| * |b|)
CLIP loss        = (CE(rows) + CE(columns)) / 2  over sim / temperature
CFG noise        = eps_uncond + w * (eps_cond - eps_uncond)
WER              = (S + D + I) / N_reference_words
```

**Quiz:** Which pairing of model family and training objective is correct?

- [ ] CLIP - predicts added Gaussian noise
- [ ] Diffusion - symmetric contrastive loss over captions
- [ ] Whisper - zero-shot classification with text prompts
- [x] CLIP - contrastive loss; diffusion - noise prediction

*Answer:* CLIP - contrastive loss; diffusion - noise prediction. CLIP learns with a contrastive image-text loss; diffusion models learn to predict the noise added at each step.