LLM-as-a-Judge Evaluators

A model grading a model

Human review does not scale to thousands of traces, so teams use LLM-as-a-judge: a second model call receives the question, the answer and sometimes the retrieved context or a reference answer, and returns a score with a reason. Typical criteria are faithfulness (is every claim supported by the context?), relevance, correctness against a reference, tone, and safety. Both Langfuse and LangSmith can run judge evaluators on traces or on dataset experiments. Judges are useful but imperfect. Known issues include position bias when comparing two answers, a preference for longer answers, leniency towards text written by similar models, and inconsistent scores on vague rubrics. Mitigations: write a specific rubric with examples, ask for a small discrete scale (such as 1-5 or pass/fail) with a reason first, use a capable judge model, and validate the judge against human labels on a sample before trusting its numbers.

Evaluation workflow

Traces and curated examples form a dataset, an experiment runs your app over it, evaluators score each output, and you compare scores between versions.

Parsing a judge's verdict safely

Judges sometimes return malformed output. Treat that as a separate "judge error" instead of silently counting it as a low score.

import json

def parse_judgement(text):
    try:
        data = json.loads(text)
        score = int(data["score"])
    except (ValueError, KeyError, TypeError):
        return None          # unparseable: record as "judge error", not as 0
    return score if 1 <= score <= 5 else None

print(parse_judgement('{"score": 4, "reason": "Cites policy, misses 7-day window."}'))
print(parse_judgement("Score: four"))
print(parse_judgement('{"score": 9}'))

Output:

4
None
None

Calibrate before you trust

Label 50-100 traces yourself, run the judge on the same traces, and check how often they agree. If agreement is poor, fix the rubric before using the judge's score in dashboards or release decisions.

Quick check: Which practice makes an LLM judge more reliable?

• Asking for a free-form opinion with no scale
• Always using the same model that generated the answer, at high temperature
• Hiding the retrieved context from a faithfulness judge
• A specific rubric with a small discrete scale, validated against human labels