Prompt Quality Scoring interview questions

25 questions from the Prompt Quality Scoring course, each with the correct answer and an explanation. Try to answer before you open the answer.

Why Score Prompts

1. Why is checking a prompt change on a handful of examples risky?

  1. Small samples are always representative
  2. LLMs always give identical answers
  3. Outputs vary and a fix for one case can silently break others
  4. Prompts cannot be changed
Show answer

Answer: Outputs vary and a fix for one case can silently break others

Measure on a representative set.

Lesson: The Problem With Eyeballing Prompts

2. Which level of scoring reflects what users actually experience most directly?

  1. Scoring the outputs (and system metrics in use)
  2. Counting words in the prompt
  3. The prompt file name
  4. The number of prompt versions
Show answer

Answer: Scoring the outputs (and system metrics in use)

Users see outputs, not prompts.

Lesson: What to Score: Prompt, Output, System

3. Why store raw outputs, not only final scores?

  1. It is required by Python
  2. Raw outputs are smaller
  3. Scores cannot be computed without deleting outputs
  4. So any score can be traced back and checked
Show answer

Answer: So any score can be traced back and checked

Traceability makes scores trustworthy.

Lesson: A Scoring Pipeline

Static Prompt Checks

4. What can a static prompt checklist NOT tell you?

  1. Whether an output format is specified
  2. Whether the outputs are actually correct
  3. Whether input is delimited
  4. Whether constraints are stated
Show answer

Answer: Whether the outputs are actually correct

Static checks are necessary, not sufficient.

Lesson: A Prompt Structure Checklist

5. How should a conflict like "brief but comprehensive" be fixed?

  1. Leave it for the model to decide
  2. Replace it with a measurable instruction or an explicit priority rule
  3. Repeat both words more often
  4. Remove all instructions
Show answer

Answer: Replace it with a measurable instruction or an explicit priority rule

Make instructions checkable.

Lesson: Vague Wording and Conflicting Instructions

6. Why log the prompt version with every output?

  1. It improves output quality automatically
  2. To make prompts shorter
  3. Models require version ids
  4. So scores and complaints can be traced to a specific version
Show answer

Answer: So scores and complaints can be traced to a specific version

Versioning enables comparison and rollback.

Lesson: Template Hygiene: Variables, Versions and Length

Deterministic Output Checks

7. Why report pass rates per check rather than one combined number?

  1. It shows which specific instruction the prompt fails to enforce
  2. Combined numbers are illegal
  3. Per-check rates are always higher
  4. It hides failures
Show answer

Answer: It shows which specific instruction the prompt fails to enforce

Specific failures suggest specific fixes.

Lesson: Format and Constraint Checks

8. Which metric suits extracted invoice totals?

  1. Word count
  2. BLEU score
  3. Numeric comparison with a tolerance
  4. Human vibes
Show answer

Answer: Numeric comparison with a tolerance

Compare numbers as numbers.

Lesson: Reference-Based Metrics

9. What is an assertion-based prompt test?

  1. A model training step
  2. A survey of users
  3. A specific expectation checked by code for a given input
  4. A random rating
Show answer

Answer: A specific expectation checked by code for a given input

Precise, repeatable checks.

Lesson: Assertion-Based Test Cases

Rubrics and Human Grading

10. Why prefer small scales with anchors over 1-10 ratings?

  1. They are applied more consistently by different graders
  2. They are always more lenient
  3. Larger scales are illegal
  4. Anchors make grading slower without benefit
Show answer

Answer: They are applied more consistently by different graders

Consistency matters more than resolution.

Lesson: Designing a Rubric

11. Why mark faithfulness as must-pass instead of just weighting it?

  1. Must-pass criteria are optional
  2. Faithfulness is unimportant
  3. Weights cannot be numbers
  4. An invented fact should fail the output regardless of other strengths
Show answer

Answer: An invented fact should fail the output regardless of other strengths

Deal-breakers are gates, not weights.

Lesson: Weighted Scores and Must-Pass Criteria

12. Why grade outputs blind to the prompt version?

  1. To make grading faster
  2. To avoid graders favouring the version they expect to be better
  3. Because versions have no names
  4. To hide the rubric
Show answer

Answer: To avoid graders favouring the version they expect to be better

Blinding removes expectation bias.

Lesson: Running Human Grading Well

LLM Judges

13. Which judge design tends to be more reliable?

  1. A single 1-10 score for "overall quality" with no rubric
  2. One criterion per call with evidence and a structured verdict
  3. No access to the source text
  4. Free-form essays as verdicts
Show answer

Answer: One criterion per call with evidence and a structured verdict

Narrow and evidence-based.

Lesson: Designing a Judge Prompt

14. What does quadratic weighted kappa reward compared with exact agreement?

  1. It measures speed
  2. Only identical ratings count
  3. It ignores chance agreement
  4. Near-miss ratings count as partial agreement, corrected for chance
Show answer

Answer: Near-miss ratings count as partial agreement, corrected for chance

Weighted agreement suits ordinal scales.

Lesson: Calibrating a Judge Against Humans

15. What happens if you optimise prompts against a length-biased judge?

  1. Outputs get longer and costlier without real quality gains
  2. Outputs always get shorter
  3. Nothing changes
  4. Costs drop automatically
Show answer

Answer: Outputs get longer and costlier without real quality gains

Biased judges steer optimisation.

Lesson: Length Bias

16. How do you detect position bias?

  1. Ask the judge if it is biased
  2. Use longer answers
  3. Judge each pair in both orders and check whether the verdict flips
  4. Only show one answer
Show answer

Answer: Judge each pair in both orders and check whether the verdict flips

Consistency under swapping reveals bias.

Lesson: Position Bias in Pairwise Judging

Comparing Prompt Versions

17. Why are pairwise comparisons often used for open-ended outputs?

  1. They remove all bias automatically
  2. They need no inputs
  3. Choosing the better of two is easier and more consistent than absolute scoring
  4. They measure cost
Show answer

Answer: Choosing the better of two is easier and more consistent than absolute scoring

Relative judgements are more reliable.

Lesson: Pairwise Comparisons and Ratings

18. Which test cases drive McNemar's test?

  1. The cases where exactly one version passes
  2. Cases where both pass
  3. Cases where both fail
  4. Only the first ten cases
Show answer

Answer: The cases where exactly one version passes

Concordant cases carry no information about the difference.

Lesson: Paired Significance Tests

19. What is an A/A test in prompt evaluation?

  1. Deleting the baseline
  2. Testing two unrelated prompts
  3. Testing only one input
  4. Comparing a prompt with itself to measure the noise in scores
Show answer

Answer: Comparing a prompt with itself to measure the noise in scores

Know your noise floor.

Lesson: Run-to-Run Variance and Repeated Sampling

Trade-Offs, Regression Gates and Goodhart

20. When is a prompt variant "dominated"?

  1. It has the highest quality
  2. Another variant is at least as good and no more expensive
  3. It is the cheapest
  4. It was written first
Show answer

Answer: Another variant is at least as good and no more expensive

Never choose a dominated option.

Lesson: The Quality-Cost Frontier

21. Why can the average stay the same while the gate blocks a change?

  1. Averages are always wrong
  2. Critical cases regressed even though others improved
  3. The gate ignores test cases
  4. Gates only check spelling
Show answer

Answer: Critical cases regressed even though others improved

Look at which cases changed, not only how many.

Lesson: Regression Gates in CI

22. Which practice helps avoid overfitting prompts to a test set?

  1. Never reviewing outputs
  2. Tuning on the same 20 cases forever
  3. Using only one metric
  4. Keeping a held-out test set and refreshing tests with new production cases
Show answer

Answer: Keeping a held-out test set and refreshing tests with new production cases

Fresh and held-out data keep scores honest.

Lesson: Goodhart's Law: Overfitting to the Score

Putting Scoring Into Practice

23. Why include cases where the correct behaviour is to decline?

  1. To check the prompt does not invent answers when it should not answer
  2. They make scores higher automatically
  3. Declining is always wrong
  4. They are easier to grade
Show answer

Answer: To check the prompt does not invent answers when it should not answer

Knowing when not to answer is part of quality.

Lesson: Building Test Sets and a Scoring Dashboard

24. In the case study, what found the invented order numbers?

  1. Counting words
  2. The prompt linter
  3. Rubric grading with faithfulness as a must-pass criterion
  4. Latency monitoring
Show answer

Answer: Rubric grading with faithfulness as a must-pass criterion

Must-pass criteria catch serious errors.

Lesson: Case Study: Scoring a Ticket-Summary Prompt

25. Which item belongs on a prompt scoring checklist?

  1. Optimising only the average score
  2. Trusting a single run on five inputs
  3. Calibrating the LLM judge against human grades
  4. Grading with the version names visible
Show answer

Answer: Calibrating the LLM judge against human grades

Calibrated, paired, gated.

Lesson: A Prompt Scoring Checklist

← Back to the course