1. An answer is wrong and the relevant passage never appeared in the top-k. Which stage failed?
- The content was missing
- Generation
- Retrieval
- Output formatting
Show answer
Answer: Retrieval
The model could not use text it never received.
25 questions from the RAG Retrieval & Evaluation course, each with the correct answer and an explanation. Try to answer before you open the answer.
Answer: Retrieval
The model could not use text it never received.
Answer: Without shared rules, labels disagree and metrics become noisy
Metrics are only as consistent as the labels.
Lesson: What Counts as Relevant
Answer: They often reuse passage wording, making retrieval look easier than with real queries
Mix in and prefer real queries.
Lesson: Building an Evaluation Set
Answer: Mean reciprocal rank (MRR)
MRR rewards a relevant result at rank 1.
Answer: Whether the best passages are ranked near the top
nDCG discounts gains lower in the list.
Lesson: Graded Relevance and nDCG
Answer: Recall over the full candidate pool passed to the reranker
The reranker can only reorder what stage one found.
Answer: Recall can be bought by sending many irrelevant passages that add cost and distraction
Watch both sides of the trade-off.
Lesson: Context Precision and Noise
Answer: It halves it
Width scales with one over the square root of n.
Lesson: How Many Queries Do You Need?
Answer: Shared difficulty cancels out, so smaller real differences become detectable
Pairing reduces noise.
Answer: An average can hide a segment that performs badly
Look at the distribution, not only the mean.
Lesson: Slices: Averages Hide Weak Spots
Answer: Your queries and documents differ from benchmark data
Measure on the distribution you serve.
Answer: The matched text can be separated from the sentence holding the answer
Retrieval matches one chunk; the answer must be inside it.
Lesson: Chunk Size Experiments
Answer: When it improves the primary metric beyond noise at acceptable latency and cost
Ablations show what each stage contributes.
Lesson: Evaluating Rerankers, Hybrid Search and Query Rewriting
Answer: Long, freely worded answers where correct paraphrases share few words with the reference
Overlap is a proxy, not understanding.
Answer: Whether the answer's claims are supported by the retrieved context
Groundedness is about the context, not the world.
Answer: Many cited sources do not actually support the answer
Precision is about the citations given.
Lesson: Citation Precision and Recall
Answer: One narrow criterion per call with a rubric and structured verdict
Narrow, evidence-based questions give consistent grades.
Lesson: Designing a Judge Prompt
Answer: When most answers are good, a lenient judge agrees often by chance
Kappa and failure recall tell the real story.
Answer: Judge both orders and count only consistent verdicts
Order-swapping exposes and cancels position effects.
Lesson: Judge Biases and Mitigations
Answer: Faithfulness regressed beyond its allowed tolerance
Guardrail metrics matter as much as the headline.
Lesson: Evaluation as a Release Gate
Answer: Few users rate and those who do are not representative
Use them for trends and failure discovery.
Lesson: Online Signals and A/B Tests
Answer: New or changed documents and new kinds of user queries
Monitor the data, not only the code.
Lesson: Drift and Index Freshness
Answer: The causes of failures, so you can fix the most common one first
Counts by cause turn numbers into actions.
Lesson: Failure Analysis With a Taxonomy
Answer: Failure analysis to find the biggest cause
Diagnose before changing.
Answer: Measuring retrieval separately at the k actually sent to the model
Separate, quantified, repeatable measurement.
Lesson: A RAG Evaluation Checklist