Part 3 — Evaluation Foundations

What an evaluation measures

What evaluation is·Evaluation·6 min read

Start the evaluation half of the book by pinning down what an evaluation actually is: a repeatable measurement of a fixed system, on a dataset with known-correct answers, under an explicit definition of correct — producing a number you can make a decision with, not a vibe.

The book so far built and ran a model. Now it turns to the question every serious LLM project lives or dies by: is it any good, and how do you know? "It seems to work" is not an answer you can ship, defend, or improve against. An evaluation replaces that feeling with a measurement — and this chapter defines exactly what is being measured, so the rest of the part can show you how.

What you will understand by the end

  • What an evaluation is — the three ingredients every real one has.
  • Why the thing measured is a system, not a model, and on your task, not in general.
  • The difference between a benchmark leaderboard and an evaluation that drives a decision.
  • The map for the rest of Part 4.

An evaluation is a measurement, and it needs three things

Strip away the tooling and an evaluation is always the same shape:

   a fixed SYSTEM  ─▶ run on ─▶  a DATASET with known-correct answers
                                        │
                                        ▼
                                  a SCORING rule  ─▶  a number you act on
  • A fixed system. The exact configuration under test — model, prompt, schema, decoding settings, parsing, and any correction policy — frozen so the result is attributable. This is the system-not-model point: you measure a configuration.
  • A dataset with known-correct answers. A set of inputs where you know what right looks like, so a score is possible at all. Building this well is its own craft (Part 5).
  • A scoring rule. An explicit definition of "correct" that turns each output into a score, the same way every time.

Take away any one and you don't have an evaluation — you have a demo, a hope, or a number nobody can reproduce.

Key idea

An evaluation is a repeatable measurement: the same system on the same dataset under the same scoring rule gives the same number. That repeatability is the whole point — it is what lets a number become a decision ("ship this config," "the guardrail helped," "don't regress below 90%") instead of an anecdote.

You are measuring your system on your task

There are two very different questions people call "evaluation," and confusing them is the first mistake:

  • General capability — "how good is this model at reasoning / coding / math?" — measured by public benchmarks. Useful for picking a starting model, but it is not about your task.
  • Your system on your task — "does this configuration produce the right output for the requests I actually get?" — measured by an eval you build. This is what tells you whether to ship.

This book is about the second. A model that tops a public leaderboard can still be wrong for your task, and a modest model wrapped in a good system can be exactly right.

Observed evidence

This project's leaderboard is an evaluation: six fixed systems, one 88-task dataset, one deterministic scoring rule — producing accuracies from 45.5% to 95.5%. None of those numbers is a claim about "which model is best in general"; each is a measurement of a configuration on this task. See the six systems measured →

A number is only useful if it drives a decision

The test of an evaluation is not "did we get a number" but "can we act on it." A good eval is built backward from the decision it must inform:

  • Should we ship config A or B? → measure both on the same dataset and compare.
  • Did the change help? → measure before and after, hold everything else fixed.
  • Are we allowed to deploy? → measure against a threshold you agreed in advance.

If a number can't change what you do, it isn't worth measuring. That is the difference between an evaluation and a vanity metric.

Mental model

An evaluation is a repeatable measurement of a fixed system, on a dataset with known-correct answers, under an explicit scoring rule — built to answer a specific decision. Not "is the model good," but "does this configuration do the right thing on my task, reproducibly enough to bet on."

Common mistakes

  • Measuring the model, not the system. The score belongs to the whole configuration; change the prompt or policy and it moves.
  • Borrowing a benchmark for your task. Public leaderboards measure general capability, not your inputs, your schema, or your definition of correct.
  • No fixed definition of correct. Without an explicit scoring rule, two people score the same output differently and the number means nothing.
  • Measuring what you can't act on. If the number can't change a decision, it's decoration.

Practical guidance

  • Write down the decision first, then design the smallest eval that answers it.
  • Pin the system (a versioned manifest) and the dataset, and state the scoring rule in one sentence before you run anything.
  • Separate "pick a base model" (use public benchmarks) from "is my system right" (build your own eval) — they need different measurements.
  • Treat every reported number as belonging to this system, this dataset, this rule — and say so.

Summary

  • An evaluation is a repeatable measurement: fixed system + dataset with known answers + scoring rule → a number you can act on.
  • You measure your system on your task, which is different from a model's general capability.
  • The value of a number is the decision it enables; if it can't change what you do, don't measure it.
  • The rest of Part 4 fills in each ingredient: modes, the contract, ground truth, scoring, the valid-vs-correct trap, and whole pipelines.

Knowledge check

A model tops a public reasoning benchmark. Your teammate says "great, it'll be the most accurate for our extraction task." What's the flaw?

The benchmark measures the model's general capability, not your system on your task. Your extraction task has its own inputs, output schema, and definition of correct, and your system wraps the model in a prompt, schema, and possibly a correction policy. The only way to know it's most accurate for you is to evaluate the competing configurations on your dataset under your scoring rule.

What makes a measurement an "evaluation" rather than a demo?

Repeatability tied to a decision: a fixed system, a dataset with known-correct answers, and an explicit scoring rule, so the same setup yields the same number and that number informs a specific choice (ship / don't ship / did it improve). A demo shows one output once; an evaluation produces a reproducible number you can bet on.

Related chapters