Evaluating Context Changes With Ablations

One variable per experiment

An ablation removes or changes one component and measures the effect: fewer chunks, no de-duplication, different ordering, no examples, compressed versus full documents. Use a fixed evaluation set of realistic questions with expected answers or grading criteria, run each variant, and compare quality, tokens, cost and latency together. Repeat runs when outputs vary, and look at failures by hand. Often the result is surprising: removing half the context can keep quality while halving cost.

Measure, apply, check

Context choices are hypotheses; test them with ablations and apply them with a checklist.

An ablation table template

Fill in your measured values; these are blanks, not results.

variant                       quality   avg input tokens   p95 latency
baseline (k=10, ranked order) ___       ___                ___
k=5                           ___       ___                ___
k=5 + dedupe                  ___       ___                ___
k=5 + dedupe + edge order     ___       ___                ___
k=5 + compression             ___       ___                ___
no few-shot examples          ___       ___                ___

Keep the eval set fixed

Change the context strategy, not the questions; otherwise you cannot tell what caused the difference.

Quick check: What is an ablation?

• Changing or removing one component and measuring the effect
• Changing everything at once
• Deleting the evaluation set
• Retraining the model