Shared Embedding Spaces

One space for pictures and words

If an image encoder and a text encoder both output vectors of the same length, we can train them so that an image and a caption describing it land close together, while unrelated pairs land far apart. The result is a shared (joint) embedding space. Closeness is usually measured with cosine similarity: the dot product of two vectors after each is scaled to length 1, giving a value between -1 and 1. Once such a space exists, many tasks become nearest-neighbour search: find images for a text query, find captions for an image, cluster a photo collection, or detect near-duplicates. The two encoders never generate anything; they only produce vectors. This is why joint-embedding models such as CLIP are cheap to use at scale: you embed your image collection once, store the vectors in a vector index, and every query costs one text encoding plus a fast similarity search.

Two towers, one space

A dual-encoder model runs an image tower and a text tower separately, then compares their normalised outputs in a shared space.

A map of a city

Think of the shared space as a city map where every photo and every sentence gets a pin. "Chai stall at a railway station" and a photo of exactly that are pinned on the same street, while a photo of a cricket match sits across town.

त्वरित जाँच: Two unit-length vectors have a cosine similarity of 1.0. What does that mean?

• They are unrelated
• They point in opposite directions
• One of them is a zero vector
• They point in exactly the same direction