Goodhart's Law: Overfitting to the Score

Keep the score honest

"When a measure becomes a target, it ceases to be a good measure." Repeatedly tweaking prompts to raise one test set's score overfits to that set: special-case wording for known inputs, verbose answers that please a length-biased judge, refusals that avoid failing strict checks. Protect against it: keep a held-out test set used rarely, refresh test sets with new production cases, combine several metrics (quality, length, refusals, cost), and confirm improvements with human review and live metrics.

Teaching to the test

Students drilled on last year's exam questions can score well without learning the subject; a new exam reveals the gap.

Keep a held-out set

Score candidate prompts on a hidden set only at release time; if gains disappear there, the prompt overfit.

त्वरित जाँच: Which practice helps avoid overfitting prompts to a test set?

• Never reviewing outputs
• Tuning on the same 20 cases forever
• Using only one metric
• Keeping a held-out test set and refreshing tests with new production cases