Retrieval Errors Versus Generation Errors

If it was not retrieved, the model cannot use it

A RAG system has two main stages: retrieval finds passages, generation writes the answer from them. When an answer is wrong, the cause is usually one of: the right passage was not retrieved (retrieval failure), it was retrieved but ranked too low or cut by the budget, it was in context but the model ignored or misread it (generation failure), or the knowledge base does not contain the answer at all. Evaluating only final answers mixes these up. Measure retrieval on its own with labelled relevant passages, then measure answers given the retrieved context, and you will know which stage to improve.

Two stages, two kinds of failure

A RAG answer can fail because the right text was never retrieved or because the model misused it; you need to measure both.

A failure triage table

Ask these questions about each wrong answer.

question                                       if NO ->
is the answer in the knowledge base?           content gap: add or fix documents
was a relevant passage in the top-k?           retrieval failure: query, chunking, retriever
was it inside the context after budgeting?     ranking / budget failure: rerank, k, compression
did the answer use it correctly?               generation failure: prompt, model, grounding rules

Log retrieved ids with every answer

Store the ranked passage ids for each production answer; without them you cannot tell which stage failed.

त्वरित जाँच: An answer is wrong and the relevant passage never appeared in the top-k. Which stage failed?

• The content was missing
• Generation
• Retrieval
• Output formatting