Designing a Rubric

Specific criteria with anchored scales

A rubric lists the criteria that define a good output for this task: faithfulness to the source, addressing the user's request, correct format, concision, tone, safety. For each criterion define a small scale (pass/fail or 1 to 3 is easier to apply consistently than 1 to 10) with anchors: a written description and example of each level. Keep criteria independent where possible, so one problem is not counted several times.

Define good before you grade

A rubric breaks quality into criteria with weights and must-pass rules, applied consistently.

A rubric with anchors

Each level described in words.

criterion: faithful to source (pass/fail, MUST PASS)
  pass: every claim is supported by the ticket text
  fail: any claim not in the ticket (invented order numbers, promises)
criterion: addresses the ask (0 / 0.5 / 1)
  1: states what the customer wants clearly
  0.5: mentions it vaguely
  0: missing or wrong
criterion: concise (0 / 0.5 / 1)
  1: <= 60 words, no repetition   0.5: 60-100 words   0: > 100 words

Pilot the rubric

Have two people grade 20 outputs, discuss disagreements and refine anchors before scaling up.

Quick check: Why prefer small scales with anchors over 1-10 ratings?

• They are applied more consistently by different graders
• They are always more lenient
• Larger scales are illegal
• Anchors make grading slower without benefit