Prerequisites: Tensors, Embeddings and Transformers

The three ideas you need

First, a tensor is a multi-dimensional array of numbers; an image, a sound clip and a batch of sentences all become tensors before a model sees them. Second, an embedding is a learned vector that represents something, a word piece, an image patch or an audio frame, so that similar things end up close together. Distance between embeddings, usually measured with cosine similarity, is how models compare meaning. Third, the transformer processes a sequence of embeddings with self-attention: each position computes how much to look at every other position and mixes their information. Transformers do not care where the sequence came from. That single fact is why they dominate multimodal AI: if you can cut an image or a sound into a sequence of vectors, the same machinery that handles text can handle it too. The rest of this course is mostly about how each modality is turned into such a sequence and how the sequences are aligned.

Text becomes a sequence of vectors

Token ids index rows of an embedding table. The result is a (tokens, dim) matrix: the shape a transformer expects.

import numpy as np

vocab = {"a": 0, "red": 1, "car": 2}
E = np.random.default_rng(0).standard_normal((len(vocab), 4)).round(2)
ids = [vocab[w] for w in "a red car".split()]
print(ids)
print(E[ids].shape)        # (tokens, embedding_dim)

Output:

[0, 1, 2]
(3, 4)

Shapes first, always

Most multimodal bugs are shape bugs: channels in the wrong position, a missing batch dimension, an image token count that does not match what the language model expects. Print shapes at every step until the pipeline works.

त्वरित जाँच: Why are transformers a natural fit for multimodal models?

• They only work on text, so other modalities are converted to words
• They need no training data
• They run faster than any other architecture on images
• They operate on any sequence of vectors, whatever modality produced it