# Attribution With a Comparison Group — Estimating AI Automation ROI

Source: https://www.skillbyai.com/en/ai-automation-roi/p-attrib

> Separate the AI effect from everything else.

## Difference-in-differences

After launch, metrics change for many reasons: new systems, seasonal mix, staff experience. Comparing only before and after for the team using AI attributes all of that to the AI. Use a **comparison group** (a team, region or time period without the tool) and compute the **difference-in-differences**: the change in the AI group minus the change in the comparison group. Randomised rollouts by team or region give the cleanest comparison.

## Did it actually pay off?

Measure results against a comparison, track the drivers, and reforecast honestly.

![Three ideas: attribution, tracking, reforecasting.](assets/figures/ai-automation-roi/section-7-map.svg) — Figure 7.1 — Attribution, tracking and reforecasting.

## Naive versus difference-in-differences, run

I ran this with plain Python 3. All figures belong to one worked example (an invoice-processing team) with invented but internally consistent numbers; prices are placeholders. The pilot team went from 6.4 to 3.1 minutes per invoice (3.3 minutes saved), but the comparison team also improved by 0.7 minutes because of a new ERP screen. Only about 2.6 minutes per invoice can be attributed to the AI tool.

```python
# minutes per invoice, before and after launch, for a team using the AI tool and a comparison team
pilot = {"before": 6.4, "after": 3.1}
control = {"before": 6.6, "after": 5.9}       # also improved: new ERP screen, seasonal mix
naive = pilot["before"] - pilot["after"]
did = naive - (control["before"] - control["after"])
print(f"naive saving (pilot before - after): {naive:.1f} min")
print(f"change in control team             : {control['before'] - control['after']:.1f} min")
print(f"difference-in-differences estimate : {did:.1f} min per invoice attributable to the AI tool")
```

Output:

```
naive saving (pilot before - after): 3.3 min
change in control team             : 0.7 min
difference-in-differences estimate : 2.6 min per invoice attributable to the AI tool
```

## Plan the comparison before launch

Choose the comparison group and metrics in advance; finding one afterwards invites cherry-picking.

**Quiz:** Why use a comparison group when measuring ROI?

- [ ] Comparison groups double savings
- [x] Other changes can improve metrics at the same time as the AI launch
- [ ] It is legally required
- [ ] It removes the need for data

*Answer:* Other changes can improve metrics at the same time as the AI launch. Isolate the AI effect.