1. Why is checking a prompt change on a handful of examples risky?
- Small samples are always representative
- LLMs always give identical answers
- Outputs vary and a fix for one case can silently break others
- Prompts cannot be changed
Show answer
Answer: Outputs vary and a fix for one case can silently break others
Measure on a representative set.