Citation Precision and Recall

Two sides of citation quality

When answers cite passage ids, measure citation precision (of the cited passages, how many really support the claims) and citation recall (of the passages needed to support the answer, how many were cited). Low precision means decorative citations users cannot trust; low recall means claims without visible support. Check basic validity in code too: every cited id must exist in the context that was sent. Answers to unanswerable questions should cite nothing and say not found.

Citation precision and recall over four answers, run

I ran this with Python 3 (numpy 2.5.3 and scikit-learn 1.9.1 where imported) on small made-up data, with fixed seeds where random. Across four answers, 3 of 4 cited documents truly support the answer (precision 0.75) and 3 of 5 needed documents were cited (recall 0.60). The last answer cited nothing.

cases = [  # (cited doc ids, gold supporting doc ids)
    ({"d1"}, {"d1"}),
    ({"d1", "d4"}, {"d1"}),
    ({"d2"}, {"d2", "d3"}),
    (set(), {"d5"}),
]
tp = fp = fn = 0
for cited, gold in cases:
    tp += len(cited & gold); fp += len(cited - gold); fn += len(gold - cited)
print(f"citation precision = {tp / (tp + fp):.2f}  (cited docs that really support the answer)")
print(f"citation recall    = {tp / (tp + fn):.2f}  (supporting docs that were cited)")

Output:

citation precision = 0.75  (cited docs that really support the answer)
citation recall    = 0.60  (supporting docs that were cited)

Validate ids in code

Reject or flag any answer citing an id that was not in the context; it is a fabricated citation.

त्वरित जाँच: Low citation precision means what?

• The answer was too short
• Too few sources were cited
• Many cited sources do not actually support the answer
• Retrieval recall is perfect