Position Bias in Pairwise Judging

Judge both orders

In pairwise comparisons ("which answer is better, A or B?") judges often favour whichever answer appears first (or second). Detect it by judging every pair in both orders: if the verdict flips with the order, the judge is responding to position, not quality. Count a win only when it is consistent across both orders and treat inconsistent pairs as ties. Report the consistency rate; a low rate means the judge cannot reliably tell these answers apart.

Swap consistency and first-slot preference, run

I ran this with Python 3 (scipy 1.18.1 where imported) on example or seeded simulated data, not results from a real product. Across 10 example pairs judged in both orders, only 4 verdicts are consistent, and 70% of wins go to whichever answer was shown first, a strong position bias.

# Each pair judged twice: answers shown as (A, B), then swapped (B, A). Record which slot won.
trials = [("first", "first"), ("first", "second"), ("first", "first"), ("second", "second"),
          ("first", "first"), ("first", "second"), ("first", "first"), ("second", "first"),
          ("first", "first"), ("first", "second")]
consistent = [t for t in trials if t[0] != t[1]]      # same answer won in both orders
first_slot = sum(s == "first" for t in trials for s in t) / (2 * len(trials))
print(f"verdicts consistent after swapping: {len(consistent)}/{len(trials)}")
print(f"share of wins going to whichever answer was shown first: {first_slot:.0%}")
print("judge has strong position bias" if first_slot > 0.6 else "no strong position bias")

Output:

verdicts consistent after swapping: 4/10
share of wins going to whichever answer was shown first: 70%
judge has strong position bias

Randomise order in every comparison

Even with swapping, randomise which answer appears first so any remaining bias does not favour one version.

Quick check: How do you detect position bias?

• Ask the judge if it is biased
• Use longer answers
• Judge each pair in both orders and check whether the verdict flips
• Only show one answer