Part 3 — Evaluation Foundations

The evaluation contract

Defining correctness·Evaluation·6 min read

Before you can score anything you must agree what 'correct' means — precisely enough that two people, or a machine, score the same output the same way. That agreement is the evaluation contract, and a fuzzy one turns every downstream number into noise.

You cannot measure "correct" until you have defined it. The definition — the exact task, the shape of a valid answer, and the rule that separates right from wrong — is the evaluation contract. It sounds like paperwork, but it is the load-bearing decision of the whole program: every score, comparison, and threshold inherits its precision from the contract. A vague contract doesn't give you a slightly-worse number; it gives you a number that means nothing.

What you will understand by the end

  • What the evaluation contract specifies, and why it must exist before you score.
  • Why an ambiguous contract makes scores unreproducible and comparisons meaningless.
  • How the contract connects the task, the ground truth, and the scoring rule.
  • What this project's contract is, concretely.

What the contract specifies

A complete contract answers four questions, unambiguously:

  • The task. What exactly is the system being asked to do, on what inputs? ("Turn a business question into a query intent," not "help with data.")
  • The input space. What inputs are in scope — and what is explicitly out of scope (so an out-of-scope input isn't scored as a failure).
  • The output form. What shape a valid answer takes — the schema, the fields, the allowed values.
  • The correctness rule. The precise condition under which an output counts as right. This is the sentence everything hinges on.
Key idea

The contract is the shared, written definition of correct — task, inputs, output form, and the exact right/wrong rule — agreed before any scoring. It is what makes a score a fact instead of an opinion: with it, the same output always earns the same score; without it, the score depends on who graded it.

Why ambiguity is fatal

If the correctness rule is fuzzy, two graders — or the same grader on two days, or a human and a script — will disagree on the same output. When that happens, the number is no longer a property of the system; it's a property of who graded it. Every comparison built on top ("A beats B by 3 points") becomes noise, because the 3 points could just be grading drift.

Watch out

The most common failure is scoring first and defining "correct" as you go, case by case. That silently changes the rule mid-run, so early and late scores aren't comparable and nobody can reproduce the number. Write the correctness rule down first, freeze it, and score every case against the same frozen rule — including the ones that make you want to bend it.

Ambiguity also has to be designed out in advance: real inputs are often genuinely ambiguous (two valid readings of one question), and the contract must say how those are handled — accept either reading, exclude them, or pick a canonical one. Deciding that at scoring time is how a contract quietly rots (more in ambiguity and annotation disagreement).

The contract ties the pieces together

The contract is the hinge between the other Part 4 concepts:

Get the contract right and those choices follow; get it wrong and no clever scoring rescues it.

Observed evidence

This project's contract is unusually crisp, which is why its numbers are trustworthy: the task is "produce a validated QueryIntent (pattern, metric, dimensions, filters, sort, limit) for a business question," the output form is that schema, and the correctness rule is exact match against the expected intent for each of the 88 tasks. Because the rule is frozen and mechanical, the 45.5%–95.5% spread is reproducible to the point — anyone re-running gets the same numbers. See the task and its terms →

Mental model

The contract is the written, frozen definition of correct — task, inputs, output form, and the exact right/wrong rule — agreed before scoring. It is what makes a score a property of the system instead of the grader. Fuzzy contract, meaningless numbers; crisp contract, reproducible ones.

Common mistakes

  • Scoring before defining. Deciding "correct" case by case changes the rule mid-run and destroys reproducibility.
  • Leaving ambiguity to scoring time. Genuinely ambiguous inputs need a rule in the contract, not an on-the-spot judgment.
  • No out-of-scope definition. Without one, inputs you never meant to support get counted as failures (or successes) arbitrarily.
  • A correctness rule no machine or second person can apply. If only its author can grade it, it isn't a contract — it's an opinion.

Practical guidance

  • Write the correctness rule in one sentence first, then pressure-test it against a handful of tricky outputs before you trust it.
  • Specify the output schema and the in/out-of-scope boundary explicitly; ambiguous inputs get an explicit handling rule.
  • Version the contract; if you change the definition of correct, that is a new contract and old scores don't carry over.
  • Prefer a contract a machine can apply where the task allows — it removes grader drift entirely.

Summary

  • The evaluation contract is the shared, frozen definition of correct: task, input space, output form, and the exact right/wrong rule.
  • It must exist before scoring; defining "correct" as you go makes numbers unreproducible.
  • It is the hinge between ground truth and the scoring method — both implement the contract.
  • This project's crisp contract (a schema plus exact-match) is why its numbers are trustworthy.

Knowledge check

Two engineers score the same 200 outputs and get 88% and 81%. The system didn't change. What's the root cause, and what's the fix?

The contract is ambiguous — there's no single frozen correctness rule, so the two graders applied slightly different definitions of "correct" (especially on borderline cases). The 7-point gap is grading drift, not a property of the system. Fix: write one explicit correctness rule, adjudicate the disagreements to refine it, re-score everything against the frozen rule (ideally mechanically), and treat genuinely ambiguous inputs with a rule stated in the contract.

Why is "produce a helpful answer" a poor correctness rule for a contract?

Because "helpful" isn't operational — two people will score the same answer differently, so it's not reproducible. A contract needs a rule precise enough that the same output always earns the same score. Either make it operational (a rubric of specific checkable criteria) or, if the task allows, define correctness structurally (matches the expected intent/label), so scoring doesn't depend on the grader's mood.

Related chapters