Failure Analysis With a Taxonomy

Read, label causes, count

Metrics say how often the system fails; failure analysis says why. Take 50 to 100 failed questions, read each with its retrieved passages and answer, and assign one cause from a fixed taxonomy: missing content, query vocabulary mismatch, chunking split, ranking too low, outdated document retrieved, budget cut, generation ignored context, unsupported addition, wrong refusal. Count each cause and fix the largest bucket first, then re-run the evaluation. This is usually the fastest route to improvement.

Analyse, apply, check

Turn metrics into fixes with failure analysis, a worked case and a checklist.

A failure tally sheet

Example structure; counts come from your own review.

cause                              count   typical fix
missing content                    ___     add / fix documents
vocabulary mismatch                ___     hybrid search, query rewriting, synonyms
chunk split answer                 ___     larger chunks, overlap, headings in chunks
ranked too low / cut by budget     ___     reranker, higher k, compression
outdated version retrieved         ___     metadata filters, re-index
generation ignored context         ___     prompt order, reminder, stronger model
unsupported addition               ___     grounding rules, faithfulness check
wrong not-found                    ___     threshold tuning

One cause per failure

Pick the earliest stage that failed; double-counting makes the tally misleading.

त्वरित जाँच: What does failure analysis add to metrics?

• The causes of failures, so you can fix the most common one first
• A higher score automatically
• Faster retrieval
• More documents