# Cross-Attention and Other Connectors — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-cross-attention

> Compare token concatenation with cross-attention connectors and implement cross-attention in NumPy.

## Two ways to let text look at images

Concatenation (the LLaVA approach) puts image tokens directly in the LLM's sequence, so every layer's self-attention can see them. The alternative is **cross-attention**: text tokens provide the **queries**, image features provide the **keys and values**, and each text token pulls in a weighted mix of image information. **Flamingo** inserted gated cross-attention layers between the frozen layers of a language model and used a **Perceiver Resampler** to compress any number of image features into a small fixed set of tokens. **BLIP-2** used a **Q-Former**, a small transformer with learned query vectors, to extract a fixed number of image tokens for a frozen LLM. Cross-attention and resamplers keep the LLM's context short, which helps with many images or video; concatenation is simpler and lets the LLM attend to fine detail but costs context. Both are intermediate fusion.

## Cross-attention in NumPy

Two text tokens attend over five image patches. Each row of weights sums to 1, and the output has one vector per text token.

```python
import numpy as np

rng = np.random.default_rng(1)
d = 8
text = rng.standard_normal((2, d))       # 2 text tokens   -> queries
image = rng.standard_normal((5, d))      # 5 image patches -> keys and values

Wq, Wk, Wv = (rng.standard_normal((d, d)) / np.sqrt(d) for _ in range(3))
Q, K, V = text @ Wq, image @ Wk, image @ Wv

scores = Q @ K.T / np.sqrt(d)            # (2 text, 5 patches)
weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights /= weights.sum(axis=1, keepdims=True)
out = weights @ V                        # each text token mixes image info

print(weights.shape, weights.sum(axis=1).round(3))
print(out.shape)
```

Output:

```
(2, 5) [1. 1.]
(2, 8)
```

## Asking questions at a help desk

Cross-attention is like students (text tokens) queuing at a railway enquiry desk (the image). Each student asks their own question and takes away only the information relevant to them; the desk itself never joins the queue.

**Quiz:** In cross-attention from text to image, which modality supplies the queries?

- [ ] The image patches
- [ ] Neither; queries are random
- [ ] Both, concatenated
- [x] The text tokens

*Answer:* The text tokens. Text tokens act as queries; image features supply keys and values, so each text token gathers relevant visual information.