What Counts as Relevant

Binary or graded, by written rules

Every retrieval metric depends on relevance judgements: for each query, which passages are relevant. Decide the rules first: is a passage relevant if it contains the answer, if it is useful context, or only if it is sufficient on its own? Use binary labels (relevant or not) for simple metrics, or graded labels (for example 0 to 3) when some passages are clearly better than others. Write examples of each grade, have two people label a sample, and measure agreement; vague definitions make metrics noisy and arguments endless.

A graded relevance rubric

Adapt the wording to your domain.

3  perfect   contains the complete answer to the question
2  useful    contains part of the answer or a necessary condition
1  marginal  on topic but does not help answer
0  not relevant

Rule: judge the passage alone, without other passages.
Rule: an outdated version of the policy is 0, even if on topic.

Label outdated content as not relevant

Make freshness part of the definition, or the metric will reward retrieving superseded documents.

Quick check: Why write relevance rules before labelling?

• Without shared rules, labels disagree and metrics become noisy
• Rules make the retriever faster
• Metrics do not use labels
• Rules replace the evaluation set