# Faithfulness: Is Every Claim Supported? — RAG Retrieval & Evaluation

Source: https://www.skillbyai.com/en/rag-evaluation/a-faith

> Check the answer against the retrieved context, claim by claim.

## Split into claims, verify each

**Faithfulness** (groundedness) asks whether every claim in the answer is supported by the retrieved context, regardless of whether it is true in the world. The standard approach: split the answer into atomic **claims**, then check each claim against the context, by a human, an NLI model, or an LLM judge given the context and one claim at a time. Report the share of supported claims and flag any unsupported ones. A crude word-overlap check can catch obvious additions but misses negations and paraphrases, so use it only as a cheap first filter.

## A crude claim-support check by word overlap, run

I ran this with Python 3 (numpy 2.5.3 and scikit-learn 1.9.1 where imported) on small made-up data, with fixed seeds where random. The answer has three claims. The 14-day claim is supported; return shipping is free contradicts the context, and refunds to the original card is not mentioned. Both are flagged because they contain words missing from the context. Word overlap would miss a contradiction that reuses the context's words, so real systems use NLI models or judges.

```python
import re
context = "Refunds are issued within 14 days of receiving the item. Return shipping is paid by the customer."
answer = "Refunds are issued within 14 days. Return shipping is free. Refunds go to the original card."
STOP = {"the", "is", "are", "by", "of", "to", "a", "within", "go"}
ctx = set(re.findall(r"[a-z0-9]+", context.lower()))
for claim in [c.strip() for c in answer.split(".") if c.strip()]:
    words = [w for w in re.findall(r"[a-z0-9]+", claim.lower()) if w not in STOP]
    missing = [w for w in words if w not in ctx]
    print(("SUPPORTED   " if not missing else "UNSUPPORTED ") + claim, "" if not missing else f"(not in context: {missing})")
```

Output:

```
SUPPORTED   Refunds are issued within 14 days 
UNSUPPORTED Return shipping is free (not in context: ['free'])
UNSUPPORTED Refunds go to the original card (not in context: ['original', 'card'])
```

## Separate faithful from correct

An answer can faithfully repeat an outdated document. Track faithfulness and correctness as different metrics.

**Quiz:** What does faithfulness measure?

- [x] Whether the answer's claims are supported by the retrieved context
- [ ] Whether the answer is long enough
- [ ] Whether retrieval was fast
- [ ] Whether the user liked the tone

*Answer:* Whether the answer's claims are supported by the retrieved context. Groundedness is about the context, not the world.