Part 9 — Continuous Evaluation

Judge calibration

Scaling judgment·Evaluation·7 min read

An LLM judge is a system, so you evaluate it before you trust it. Calibration is measuring the judge against human ground truth — does it agree with people, and where is it biased — so its scores become a validated instrument instead of an unexamined oracle you're quietly optimising toward.

The previous chapter ended on a warning: an LLM judge is an unvalidated system whose scores are not ground truth. Calibration is how you retire that warning — by treating the judge as exactly what it is, a system to be evaluated, and running an evaluation on the judge itself. You measure how well its verdicts match human judgment, find its biases, and only then use it. Skip this and you're not measuring quality; you're measuring the judge's prejudices and calling them quality.

What you will understand by the end

  • Why the judge must itself be evaluated against human ground truth.
  • How to measure judge quality: agreement with humans, and bias probes.
  • What to do when the judge and humans disagree — fix the judge or the rubric.
  • Why a calibrated judge is a validated instrument, and an uncalibrated one is a liability.

Evaluate the judge like any system

Everything you know about evaluation applies to the judge. It has a task ("score this output against this rubric"), so it needs ground truth — a set of outputs with human verdicts — and a scoring rule: how often does the judge agree with the humans? Calibration is running that evaluation:

  • Assemble a calibration set of outputs with trusted human labels (ideally adjudicated, since the humans must agree first).
  • Run the judge on the same outputs.
  • Measure agreement between judge and humans — overall, and per slice.
Key idea

A judge you haven't measured against humans is an opinion of unknown quality. Calibration turns it into a validated instrument with a known agreement rate and known biases. Only once you know "this judge agrees with our raters 88% of the time and skews toward longer answers" can its scores carry weight in a comparison or a gate.

Measure agreement, and probe for bias

Two things to establish:

  • Agreement with humans. How often does the judge's verdict match the human verdict (adjusted for chance)? This is the judge's "accuracy," and it's bounded by human agreement itself — a judge can't be more reliable than the humans defining correct. Low agreement means the judge (or the rubric) isn't ready.
  • Bias probes. Deliberately test for the known failure modes: present the same pair in both orders (position bias), pad one answer with fluff (verbosity bias), include outputs from the judge's own model family (self-preference). A judge that flips its verdict when you swap positions has a bias you must mitigate or correct for.
Watch out

The most dangerous state is an uncalibrated judge wired into a gate: you're now optimising your system toward whatever the judge rewards, biases included. If the judge has verbosity bias, your releases will drift toward longer answers; if it has self-preference, toward one model's style. You won't see it in the judge's scores — they'll look great — you'll see it in a slow, unexplained drift. Calibrate before the judge influences any decision.

When judge and humans disagree

Disagreement between the judge and the humans is diagnostic, and the fix depends on the cause:

  • The rubric is vague. If humans interpret the criteria one way and the judge another, sharpen the rubric — often the judge is "wrong" because the instructions were ambiguous.
  • The judge is biased. If it systematically fails a bias probe, mitigate (position swaps, length-controlled prompts, a different judge model) and re-measure.
  • The humans disagree with each other. If the calibration labels themselves have low inter-annotator agreement, the ground truth isn't settled — fix that before blaming the judge.

Re-calibrate after any change to the judge model or its prompt: a "small" prompt tweak can move its biases, so its agreement number is only valid for the exact judge configuration you measured.

Observed evidence

This project sidesteps judge calibration entirely by using deterministic exact match — the "scorer" is a mechanical verifier with 100% agreement with itself and no bias to calibrate. That's the cleanest state of all: when a task can be scored deterministically, there's no judge to validate. Calibration is the price of using a judge, and it's a price worth avoiding when exact match is possible. A deterministic scorer needs no calibration →

Mental model

A judge is a system, so evaluate it: build a human-labelled calibration set, measure the judge's agreement with humans, and probe it for position, verbosity, and self-preference bias. Fix the rubric or mitigate the bias where they disagree, and re-calibrate after any judge change. A calibrated judge is a validated instrument; an uncalibrated one wired to a gate quietly optimises its own biases.

Common mistakes

  • Using a judge without calibrating it. Its scores are unvalidated opinion; you may be gating on bias.
  • No bias probes. Overall agreement can look fine while position or verbosity bias skews the decisions that matter.
  • Calibrating once, then changing the judge prompt. A prompt tweak can shift biases; agreement is only valid for the exact configuration measured.
  • Blaming the judge when the humans disagree. Low human agreement means the ground truth isn't settled — fix that first.

Practical guidance

  • Build a human-labelled calibration set (adjudicated so humans agree), run the judge on it, and report agreement overall and per slice.
  • Probe for bias explicitly — position (swap order), verbosity (pad length), self-preference (own-family outputs) — and mitigate what you find.
  • Read the judge's agreement as bounded by human agreement; a judge can't beat the humans who define correct.
  • Re-calibrate on any change to the judge model or prompt, and prefer deterministic scoring whenever the task makes a judge unnecessary.

Summary

  • A judge is a system; calibration is evaluating it against human ground truth before you trust it.
  • Measure agreement with humans (bounded by human agreement) and probe for bias (position, verbosity, self-preference).
  • Disagreement points at a vague rubric, a biased judge, or unsettled human labels — fix the right one and re-calibrate on any judge change.
  • A calibrated judge is a validated instrument; an uncalibrated one in a gate optimises its own biases — and a deterministic scorer needs no calibration at all.

Knowledge check

Your LLM judge agrees with human raters 90% overall, so you deploy it in a gate. A bias probe you skipped would have shown it flips its verdict 40% of the time when you swap the two answers' order. Why does the 90% not save you?

Because overall agreement hides the position bias: a judge that changes its answer 40% of the time on order alone is partly deciding by position rather than quality, and in pairwise gating that systematically favors whichever slot you put an option in — corrupting exactly the promotion decisions the gate makes. High aggregate agreement can coexist with a strong directional bias that only a targeted probe (swap positions and average) reveals. You must probe for the known biases specifically, not just measure overall agreement, before trusting a judge in a gate.

The judge and your human labels disagree on 25% of a calibration slice. Before concluding "the judge is bad," what should you check?

Whether the human labels themselves agree — the inter-annotator agreement on that slice. If the humans only agree with each other, say, 75% of the time, the ground truth isn't settled, so the judge can't meaningfully exceed that and the 25% "disagreement" partly reflects contested labels, not a bad judge. Also check whether the rubric is vague (humans and judge interpreting criteria differently). Fix unsettled ground truth and ambiguous rubrics first; only then is remaining judge-vs-human disagreement evidence of a biased or weak judge to mitigate.

Related chapters