# Evaluating Context Changes With Ablations — Prompt & Context Engineering

Source: https://www.skillbyai.com/en/prompt-context-engineering/e-ablate

> Change one thing at a time and measure.

## One variable per experiment

An **ablation** removes or changes one component and measures the effect: fewer chunks, no de-duplication, different ordering, no examples, compressed versus full documents. Use a fixed evaluation set of realistic questions with expected answers or grading criteria, run each variant, and compare **quality, tokens, cost and latency** together. Repeat runs when outputs vary, and look at failures by hand. Often the result is surprising: removing half the context can keep quality while halving cost.

## Measure, apply, check

Context choices are hypotheses; test them with ablations and apply them with a checklist.

![Three ideas: ablation, case study, checklist.](assets/figures/prompt-context-engineering/section-8-map.svg) — Figure 8.1 — Ablation, case study and checklist.

## An ablation table template

Fill in your measured values; these are blanks, not results.

```text
variant                       quality   avg input tokens   p95 latency
baseline (k=10, ranked order) ___       ___                ___
k=5                           ___       ___                ___
k=5 + dedupe                  ___       ___                ___
k=5 + dedupe + edge order     ___       ___                ___
k=5 + compression             ___       ___                ___
no few-shot examples          ___       ___                ___
```

## Keep the eval set fixed

Change the context strategy, not the questions; otherwise you cannot tell what caused the difference.

**Quiz:** What is an ablation?

- [x] Changing or removing one component and measuring the effect
- [ ] Changing everything at once
- [ ] Deleting the evaluation set
- [ ] Retraining the model

*Answer:* Changing or removing one component and measuring the effect. One variable at a time isolates the cause.