Multimodal Retrieval-Augmented Generation

Retrieval when the knowledge is visual

Retrieval-augmented generation (RAG) answers questions by first retrieving relevant content and then asking a model to answer using it. In many real collections the knowledge sits in charts, diagrams, slides and scanned pages that plain text extraction mangles. Multimodal RAG handles this in one of three common ways. First, caption then embed: a VLM writes a description of each image or page, and the text is indexed with a normal text embedder; simple, but details the caption missed are lost. Second, joint embeddings: index images directly with a CLIP-style model so text queries retrieve them. Third, page-image retrieval: embed whole page screenshots with a document-retrieval model and send the retrieved page images straight to a VLM. Whichever you choose, the final step passes the retrieved images and text to a VLM with the question, and asks it to cite which page or image supports the answer.

Multimodal RAG pipeline

Content is embedded into one index; a query retrieves the closest pages and images; a VLM answers with citations.

Retrieving images and text from one index

A chart image and a text chunk both score highly for a sales question because they share one embedding space.

import numpy as np

def normalize(x):
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

# One shared embedding space: page images and text chunks live together.
items = ["page3.png (bar chart of sales)", "chunk: refund policy text",
         "page7.png (wiring diagram)", "chunk: sales grew in Q2"]
index = normalize(np.array([[0.9, 0.1, 0.1], [0.0, 0.2, 0.9],
                            [0.1, 0.9, 0.2], [0.8, 0.0, 0.3]]))

query = normalize(np.array([1.0, 0.0, 0.2]))   # "how did sales change?"
scores = index @ query
top = np.argsort(-scores)[:2]
for i in top:
    print(f"{scores[i]:.3f}  {items[i]}")

Output:

0.990  page3.png (bar chart of sales)
0.987  chunk: sales grew in Q2

Keep the original, not just the caption

Store a pointer to the original image or page alongside every embedding. At answer time send the original pixels to the VLM; captions are useful for search but lose numbers, labels and layout.

Quick check: What is the main risk of the caption-then-embed approach to multimodal RAG?

• Details the caption omitted cannot be retrieved
• It requires training a new LLM
• It cannot handle text queries
• It only works for video