Drift and Index Freshness

Content, queries and models move

Retrieval quality drifts as documents are added, edited or removed, as users ask about new products, and as providers update models. Monitor index health (documents ingested, failed parses, age of the newest version per source), query mix (new topics, languages), and the online signals above. Re-run the offline evaluation on a schedule, refresh the evaluation set with recent real queries, and re-label passages when documents change so labels do not point at deleted chunks. Re-embed the index when you change embedding models and evaluate before switching traffic.

A weekly retrieval health report

Typical fields; thresholds depend on your system.

documents ingested this week: 412   parse failures: 3   stale sources (>30 days): 1
queries: 18,240   new-topic share: 7%   non-English share: 12%
not-found rate: 4.1% (last week 2.9%)  <- investigate
thumbs-down rate: 3.2%   escalations: 1.1%
offline eval (v31 set): recall@5 0.84 (baseline 0.85, within tolerance)

Version labels with documents

When a document is re-chunked, re-map its relevance labels, or recall will drop for reasons unrelated to quality.

Quick check: What can cause retrieval quality to drop with no code change?

• Adding more tests
• Nothing, quality is fixed
• Reading the logs
• New or changed documents and new kinds of user queries