Lesson 11 / 29
Self-Consistency: Sample Several, Take the Majority
Reduce random reasoning errors by voting.
By Rahul Agarwal · 15 years in web development
Many paths, one answer
With sampling randomness, the model may reason correctly on some runs and slip on others. Self-consistency runs the same prompt several times (for example 5 to 10) at a moderate temperature, extracts each final answer and returns the most common one. It helps most on problems with a single checkable answer (maths, classification, extraction) and costs several times as much per question, so use it for high-value cases. If agreement is low, treat that as a signal of uncertainty and escalate to a human or a stronger method.
A majority vote, run
I ran this plain-Python (standard library only) example. Seven sampled answers (invented to show the mechanics) give 42 four times, 41 twice and 40 once; the majority is 42 with 4 of 7 in agreement, a moderate level that you might want to review.
from collections import Counter
samples = ["42", "42", "41", "42", "40", "42", "41"]
counts = Counter(samples)
answer, votes = counts.most_common(1)[0]
print(dict(counts))
print("majority answer:", answer, f"({votes}/{len(samples)} agree)")
Output:
{'42': 4, '41': 2, '40': 1}
majority answer: 42 (4/7 agree)Quick check: What does low agreement among samples suggest?
- The API is down
- The answer is certainly right
- The prompt is too short to run
- The model is uncertain and the case may need review
Answer
The model is uncertain and the case may need review — Disagreement is a useful uncertainty signal.