Cross-Attention and Other Connectors

Two ways to let text look at images

Concatenation (the LLaVA approach) puts image tokens directly in the LLM's sequence, so every layer's self-attention can see them. The alternative is cross-attention: text tokens provide the queries, image features provide the keys and values, and each text token pulls in a weighted mix of image information. Flamingo inserted gated cross-attention layers between the frozen layers of a language model and used a Perceiver Resampler to compress any number of image features into a small fixed set of tokens. BLIP-2 used a Q-Former, a small transformer with learned query vectors, to extract a fixed number of image tokens for a frozen LLM. Cross-attention and resamplers keep the LLM's context short, which helps with many images or video; concatenation is simpler and lets the LLM attend to fine detail but costs context. Both are intermediate fusion.

Cross-attention in NumPy

Two text tokens attend over five image patches. Each row of weights sums to 1, and the output has one vector per text token.

import numpy as np

rng = np.random.default_rng(1)
d = 8
text = rng.standard_normal((2, d))       # 2 text tokens   -> queries
image = rng.standard_normal((5, d))      # 5 image patches -> keys and values

Wq, Wk, Wv = (rng.standard_normal((d, d)) / np.sqrt(d) for _ in range(3))
Q, K, V = text @ Wq, image @ Wk, image @ Wv

scores = Q @ K.T / np.sqrt(d)            # (2 text, 5 patches)
weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights /= weights.sum(axis=1, keepdims=True)
out = weights @ V                        # each text token mixes image info

print(weights.shape, weights.sum(axis=1).round(3))
print(out.shape)

Output:

(2, 5) [1. 1.]
(2, 8)

Asking questions at a help desk

Cross-attention is like students (text tokens) queuing at a railway enquiry desk (the image). Each student asks their own question and takes away only the information relevant to them; the desk itself never joins the queue.

त्वरित जाँच: In cross-attention from text to image, which modality supplies the queries?

• The image patches
• Neither; queries are random
• Both, concatenated
• The text tokens