Part 9 — Continuous Evaluation

Human review

Scaling judgment·Evaluation·6 min read

Automation does most of the work, but a continuous evaluation program still runs on human judgment at specific, high-leverage points — setting ground truth, adjudicating disagreement, calibrating judges, and spot-checking production. This chapter is about using scarce human attention where it's irreplaceable, and nowhere else.

Regression suites, gates, and LLM judges automate the bulk of continuous evaluation — but they all rest on human judgment at the bottom. Someone defined "correct," someone adjudicated the ambiguous cases, someone's labels calibrated the judge. Human review is expensive and doesn't scale, so the skill is not "review everything" but spending scarce human attention exactly where it's irreplaceable — and letting automation carry everything else. This chapter is that allocation problem.

What you will understand by the end

  • Why human judgment remains the foundation even in a heavily-automated program.
  • The high-leverage places to spend it: ground truth, adjudication, calibration, spot-checks.
  • How to make scarce human review efficient — sampling, guidelines, multiple raters.
  • Why humans and automation are a hierarchy, not a competition.

Automation rests on human judgment

Every automated check traces back to a human decision. A regression suite runs cases whose correct answers a human set. A gate enforces a threshold a human chose. An LLM judge is only trustworthy because humans calibrated it. Automation doesn't remove human judgment — it amplifies it, running the judgment humans encoded across millions of cases. So the question is never "humans or automation" but "where do the humans need to be?"

Key idea

Human review is the foundation automation stands on, not a competitor to it. The goal is to place human judgment where it defines or validates the automated machinery — and let the machinery handle the volume. Reviewing what a rule could have scored wastes the one resource (expert attention) the whole program depends on.

Where to spend it

Human attention pays off most at four points:

  • Setting ground truth. Authoring or verifying the correct answers for hard cases — the references everything else is scored against. Garbage here caps the whole program.
  • Adjudicating disagreement. Resolving the ambiguous cases where annotators or judges split — humans define the boundary the automation then enforces.
  • Calibrating and auditing judges. Producing the labels that validate an LLM judge, and periodically re-auditing it.
  • Spot-checking production. Reading a sample of real outputs — especially where automated signals are weak or absent — to catch what the metrics miss and to surface new failure modes.

Notice what's not on the list: scoring cases a deterministic rule or a calibrated judge already handles. That's where automation belongs.

Watch out

Two opposite failures. Over-using humans — reviewing everything by hand — doesn't scale, burns out reviewers, and adds [variance](/study/eval-scoring-types/) to things a rule would score consistently. Under-using them — fully automating with no human in the loop — means no one notices when the automation itself drifts or the ground truth rots. Spend human attention deliberately at the leverage points, not everywhere and not nowhere.

Make scarce review efficient

Because human review is costly, treat it as a scarce resource to deploy well:

  • Sample, don't census. For spot-checks, a well-chosen random (and stratified) sample tells you most of what a full review would, at a fraction of the cost.
  • Write guidelines. Clear instructions turn inconsistent individual judgment into repeatable labels and reduce rater disagreement.
  • Use multiple raters on the hard stuff. For ambiguous or high-stakes cases, more than one rater plus adjudication beats a single opinion — and measures agreement as a byproduct.
  • Prioritise by uncertainty and stakes. Review where automated confidence is low or the cost of error is high, not uniformly.
Observed evidence

This project's human judgment is concentrated up front, in designing the task and the expected answers: a person defined the QueryIntent schema and the one canonical intent per question, and from there a deterministic verifier scores everything with no human in the loop. That's the ideal allocation — humans set the ground truth once, automation runs it forever — and it's why the evaluation scales to every experiment without a review bottleneck. See the human-designed contract automation runs on →

Mental model

Automation amplifies human judgment; it doesn't replace it. Spend scarce human attention where it's irreplaceable — setting ground truth, adjudicating ambiguity, calibrating judges, spot-checking production — and let rules and calibrated judges handle the volume. Sample, write guidelines, use multiple raters on hard cases, and prioritise by uncertainty and stakes.

Common mistakes

  • Reviewing everything by hand. Doesn't scale, burns out reviewers, and adds variance a rule would avoid.
  • Fully automating with no human in the loop. No one catches automation drift or rotting ground truth.
  • Spending humans on machine-scorable cases. Wastes expert attention the whole program depends on.
  • Unstructured review. Without guidelines and sampling, human review is inconsistent and expensive for little signal.

Practical guidance

  • Place humans at the leverage points — ground truth, adjudication, judge calibration, production spot-checks — and automate the rest.
  • Sample and stratify for spot-checks; a good sample beats an exhausting census.
  • Give reviewers guidelines, use multiple raters on ambiguous/high-stakes cases, and prioritise by uncertainty and stakes.
  • Keep a standing human spot-check of production even when automation looks healthy — it's how you catch drift the metrics miss.

Summary

  • Even a heavily automated program rests on human judgment; automation amplifies it rather than replacing it.
  • Spend scarce human attention where it's irreplaceable: ground truth, adjudication, judge calibration, production spot-checks.
  • Make review efficient — sample, write guidelines, use multiple raters on hard cases, prioritise by uncertainty and stakes.
  • This project concentrates humans in designing the contract and answers, then lets deterministic scoring run forever — the ideal allocation.

Knowledge check

A team has a calibrated LLM judge and a deterministic verifier running on every change. Where, if anywhere, should humans still spend time?

At the foundation and audit points, not the volume: (1) setting/verifying ground truth for hard cases and adjudicating ambiguous ones; (2) periodically re-calibrating and auditing the judge, since its biases can drift with prompt or model changes; and (3) spot-checking production samples to catch failure modes and drift the automated signals miss. What they should not do is hand-score cases the verifier or calibrated judge already handles well — that wastes the expert attention the automation itself depends on.

Why is "automate everything, no human in the loop" a risky end state for an evaluation program?

Because all the automation ultimately encodes human judgments — the ground truth, the thresholds, the judge's calibration — and those can rot or drift: ground truth goes stale, the input distribution shifts, the judge's biases move after a model update. With no human ever looking, nothing catches the automation quietly measuring the wrong thing; the dashboards stay green while reality diverges. A standing human spot-check and periodic audits are what detect that drift, which is why the right answer is targeted human review, not zero.

Related chapters