Attribution With a Comparison Group

Difference-in-differences

After launch, metrics change for many reasons: new systems, seasonal mix, staff experience. Comparing only before and after for the team using AI attributes all of that to the AI. Use a comparison group (a team, region or time period without the tool) and compute the difference-in-differences: the change in the AI group minus the change in the comparison group. Randomised rollouts by team or region give the cleanest comparison.

Did it actually pay off?

Measure results against a comparison, track the drivers, and reforecast honestly.

Naive versus difference-in-differences, run

I ran this with plain Python 3. All figures belong to one worked example (an invoice-processing team) with invented but internally consistent numbers; prices are placeholders. The pilot team went from 6.4 to 3.1 minutes per invoice (3.3 minutes saved), but the comparison team also improved by 0.7 minutes because of a new ERP screen. Only about 2.6 minutes per invoice can be attributed to the AI tool.

# minutes per invoice, before and after launch, for a team using the AI tool and a comparison team
pilot = {"before": 6.4, "after": 3.1}
control = {"before": 6.6, "after": 5.9}       # also improved: new ERP screen, seasonal mix
naive = pilot["before"] - pilot["after"]
did = naive - (control["before"] - control["after"])
print(f"naive saving (pilot before - after): {naive:.1f} min")
print(f"change in control team             : {control['before'] - control['after']:.1f} min")
print(f"difference-in-differences estimate : {did:.1f} min per invoice attributable to the AI tool")

Output:

naive saving (pilot before - after): 3.3 min
change in control team             : 0.7 min
difference-in-differences estimate : 2.6 min per invoice attributable to the AI tool

Plan the comparison before launch

Choose the comparison group and metrics in advance; finding one afterwards invites cherry-picking.

Quick check: Why use a comparison group when measuring ROI?

• Comparison groups double savings
• Other changes can improve metrics at the same time as the AI launch
• It is legally required
• It removes the need for data