Contrastive Training: How CLIP Learns

Learning from matching pairs

CLIP (Contrastive Language-Image Pre-training) was trained on a very large collection of image-caption pairs gathered from the web. For a batch of N pairs, both towers embed everything, and the model computes an N x N matrix of cosine similarities divided by a temperature. The diagonal holds the true pairs; every off-diagonal cell is a negative example. The loss is cross-entropy over each row (pick the right caption for each image) plus cross-entropy over each column (pick the right image for each caption), averaged. This is the symmetric InfoNCE loss. Two details matter. Larger batches give more negatives per step, which makes the task harder and the embeddings sharper, so contrastive training favours very large batches. The temperature controls how peaked the softmax is; CLIP learns it as a parameter. No human labels are needed beyond the naturally occurring captions.

The symmetric contrastive loss in NumPy

Three image-caption pairs. The diagonal of the logits is large because each image matches its own caption, so the loss is close to zero.

import numpy as np

def normalize(x):
    return x / np.linalg.norm(x, axis=1, keepdims=True)

# 3 images and their 3 matching captions (row i belongs with row i)
img = normalize(np.array([[0.9, 0.1, 0.0], [0.1, 0.8, 0.3], [0.0, 0.2, 0.9]]))
txt = normalize(np.array([[1.0, 0.0, 0.1], [0.0, 1.0, 0.2], [0.1, 0.1, 1.0]]))

temperature = 0.07
logits = img @ txt.T / temperature       # cosine similarity / temperature

def cross_entropy(logits, labels):
    logits = logits - logits.max(axis=1, keepdims=True)
    log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
    return -log_probs[np.arange(len(labels)), labels].mean()

labels = np.arange(3)                    # the diagonal is the correct pair
loss = (cross_entropy(logits, labels) + cross_entropy(logits.T, labels)) / 2
print(logits.round(1))
print("loss:", round(float(loss), 4))

Output:

[[14.1  1.5  1.6]
 [ 2.1 14.   6.4]
 [ 1.4  5.8 14.1]]
loss: 0.0002

Why not just use a normal classifier?

A classifier learns a fixed list of labels. Contrastive training learns from free-form captions, so the model picks up open-ended concepts ("a rangoli made of flower petals") that no fixed label set would contain. That is what makes zero-shot use possible.

Quick check: In CLIP's training, where are the negative examples for an image?

• In a separate hand-labelled dataset of wrong captions
• In the other captions in the same batch
• In random noise images
• There are no negatives in contrastive training