# Multimodal Retrieval-Augmented Generation — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-multimodal-rag

> Design a RAG pipeline that retrieves images and text together and grounds a VLM's answer in them.

## Retrieval when the knowledge is visual

**Retrieval-augmented generation (RAG)** answers questions by first retrieving relevant content and then asking a model to answer using it. In many real collections the knowledge sits in charts, diagrams, slides and scanned pages that plain text extraction mangles. **Multimodal RAG** handles this in one of three common ways. First, **caption then embed**: a VLM writes a description of each image or page, and the text is indexed with a normal text embedder; simple, but details the caption missed are lost. Second, **joint embeddings**: index images directly with a CLIP-style model so text queries retrieve them. Third, **page-image retrieval**: embed whole page screenshots with a document-retrieval model and send the retrieved page images straight to a VLM. Whichever you choose, the final step passes the retrieved images and text to a VLM with the question, and asks it to cite which page or image supports the answer.

## Multimodal RAG pipeline

Content is embedded into one index; a query retrieves the closest pages and images; a VLM answers with citations.

![Four connected stages: ingest pages and images, embed into an index, retrieve top-k, VLM answers with citations.](assets/figures/multimodal-ai/section-8-map.svg) — Figure 8.1: ingest (pages, images, text) -> embed into a vector index -> retrieve top-k for the query -> VLM answers with citations.

## Retrieving images and text from one index

A chart image and a text chunk both score highly for a sales question because they share one embedding space.

```python
import numpy as np

def normalize(x):
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

# One shared embedding space: page images and text chunks live together.
items = ["page3.png (bar chart of sales)", "chunk: refund policy text",
         "page7.png (wiring diagram)", "chunk: sales grew in Q2"]
index = normalize(np.array([[0.9, 0.1, 0.1], [0.0, 0.2, 0.9],
                            [0.1, 0.9, 0.2], [0.8, 0.0, 0.3]]))

query = normalize(np.array([1.0, 0.0, 0.2]))   # "how did sales change?"
scores = index @ query
top = np.argsort(-scores)[:2]
for i in top:
    print(f"{scores[i]:.3f}  {items[i]}")
```

Output:

```
0.990  page3.png (bar chart of sales)
0.987  chunk: sales grew in Q2
```

## Keep the original, not just the caption

Store a pointer to the original image or page alongside every embedding. At answer time send the original pixels to the VLM; captions are useful for search but lose numbers, labels and layout.

**Quiz:** What is the main risk of the caption-then-embed approach to multimodal RAG?

- [x] Details the caption omitted cannot be retrieved
- [ ] It requires training a new LLM
- [ ] It cannot handle text queries
- [ ] It only works for video

*Answer:* Details the caption omitted cannot be retrieved. Retrieval only sees the generated caption, so anything the captioner skipped is invisible to search.