33 questions from the Multimodal AI course, each with the correct answer and an explanation. Try to answer before you open the answer.
Answer: Answering "how many people are wearing helmets?" about a photo
Visual question answering needs the model to connect a text question with pixel content. The other three are text-only tasks.
Lesson: What Multimodal AI Means
Answer: They operate on any sequence of vectors, whatever modality produced it
Self-attention works on sequences of embeddings. Once images or audio are cut into vectors, the same layers apply.
Answer: Late fusion of the two models' scores
Late fusion reuses both models, needs little paired data and degrades gracefully when one modality is absent.
Answer: The colour channels (R, G, B)
In NCHW layout the second dimension is channels; an RGB image has three.
Lesson: Images as Tensors
Answer: 160,000
Sample rate times duration: 16,000 samples per second x 10 seconds = 160,000 samples.
Answer: Adjacent frames are highly redundant and every frame costs tokens and compute
Consecutive frames are almost identical, so sampling cuts cost sharply while keeping most of the information.
Answer: It outputs a sequence of token vectors
A ViT produces one vector per patch, a token sequence that a projector can map into the language model's embedding space.
Answer: 576
336 / 14 = 24 patches per side, and 24 x 24 = 576.
Lesson: Patch Embeddings Step by Step
Answer: A log-mel spectrogram of 16 kHz audio
Whisper resamples to 16 kHz and computes a log-mel spectrogram, processed in 30-second windows.
Lesson: Audio Encoders and Speech Models
Answer: They point in exactly the same direction
Cosine similarity is 1 when the angle between the vectors is zero, i.e. they point the same way.
Lesson: Shared Embedding Spaces
Answer: In the other captions in the same batch
Every other caption in the batch serves as a negative, which is why larger batches help.
Answer: A text prompt describing the class
Zero-shot classes are defined by text prompts; the text tower embeds any new description.
Answer: Map vision-encoder outputs into the language model's embedding space
The projector converts patch vectors to the dimensionality and space the LLM's token embeddings use.
Answer: The text tokens
Text tokens act as queries; image features supply keys and values, so each text token gathers relevant visual information.
Answer: Only the projector
Stage 1 freezes the encoder and LLM and trains just the projector on image-caption pairs.
Lesson: How VLMs Are Trained
Answer: The noise that was added at step t
The standard objective is to predict the added noise, trained with a mean squared error loss.
Answer: The latent is much smaller than the pixel image, so each step is cheaper
Compressing to a small latent makes every denoising step far cheaper; the VAE decoder restores full resolution at the end.
Answer: Latency adds up and the LLM loses tone and emotion from the speech
Each stage adds delay, and converting speech to text discards paralinguistic information before the LLM sees it.
Answer: Smaller images upload faster, fit size limits and usually cost fewer image tokens
Large images waste bandwidth, can exceed limits, and often cost more tokens without improving answers for typical questions.
Lesson: Sending Images to a Model API
Answer: Prepares images, text and chat templates exactly as that checkpoint expects
The processor bundles the image preprocessing, tokenizer and chat template matched to the checkpoint.
Lesson: Open Models with Hugging Face
Answer: The vision encoder and projector that turn images into tokens for the LLM
The multimodal projector file holds the vision side; it must match the language model weights.
Answer: Details the caption omitted cannot be retrieved
Retrieval only sees the generated caption, so anything the captioner skipped is invisible to search.
Answer: Verifying that extracted line items sum to the extracted total
Deterministic business-rule checks catch inconsistent values; asking the model to self-assess is unreliable.
Answer: Too few or no frames are being sent to the model for that segment
Visual questions need frames; a transcript-heavy pipeline with sparse or missing frames cannot see the screen.
Lesson: Video Understanding Applications
Answer: It quadruples
Patch count is (H/P) x (W/P), so doubling both sides multiplies it by four.
Answer: 0.3
WER = (S + D + I) / N = (1 + 1 + 1) / 10 = 0.3.
Lesson: Evaluating Multimodal Systems
Answer: Precompute and store image embeddings once; embed only the query at search time
Catalogue embeddings are static; compute them at ingest and only encode the query text online.
Lesson: Serving Fast and Cheap
Answer: Send a higher-resolution crop of the total region and allow a 'not readable' answer
Detail errors usually come from downsampling; a crop restores resolution and an escape option prevents guessing.
Answer: Require validation against business rules or human approval before any payment
Prompt instructions alone do not reliably stop injection; gating privileged actions limits the damage when the model is fooled.
Answer: EXIF may contain GPS location and device details that reveal personal information
EXIF often includes location and device data; removing it protects users when photos are stored or shared.
Answer: CLIP - contrastive loss; diffusion - noise prediction
CLIP learns with a contrastive image-text loss; diffusion models learn to predict the noise added at each step.
Answer: Each pair's negatives are the other items in the batch, so more items give more and harder negatives
In-batch negatives are the core of the loss; bigger batches supply more of them per step.
Answer: Asking for a clearer photo when image quality is low and allowing an 'unsure' answer
Detecting poor inputs and permitting uncertainty prevents the model from guessing a diagnosis from pixels it cannot see clearly.
Lesson: Mixed Self-Check