Online vs Offline Evaluation

Two complementary loops

Offline evaluation runs on a fixed dataset before release. It is repeatable and comparable, but limited to the cases you thought of. Online evaluation scores live production traces, for example running a faithfulness judge on a sample of real answers, or checking every output for policy violations. It catches new kinds of questions and drift, but it is not a controlled comparison and it costs money for every judged trace, so teams usually sample. A third source is human review: annotation queues where reviewers label selected traces, typically those with negative feedback, low judge scores or high cost. The healthy loop: production traces feed online scores and review queues; reviewed failures become dataset examples; offline experiments verify the fix; the release is monitored online again. Each part covers a blind spot of the others.

Sample online judges by risk

Running a judge on every trace can cost as much as the app itself. Sample a small percentage of normal traffic, but evaluate a much larger share of risky slices such as new features, negative feedback or a specific high-value customer.

Mock tests and the real board exam

Mock tests (offline) let you compare preparation strategies fairly. The board exam (online) shows how you do on questions you never saw. Checking the real result paper (human review) tells you which topics to add to your next mock.

Quick check: Which is a strength of online evaluation that offline evaluation lacks?

• It gives a perfectly controlled comparison between two versions
• It is free to run
• It reveals how the system handles real, previously unseen user inputs
• It works without any traces