Ground truth and reference answers

To score an output you need something to score it against: the known-correct answer, the ground truth. It is the quiet foundation of every offline evaluation, and also its most common weak point — because ground truth has to be created, it costs real effort, it can be wrong, and for many tasks there isn't a single right answer at all. Where your ground truth comes from, and how trustworthy it is, sets a ceiling on everything above it.

What you will understand by the end

• What ground truth is and the main ways it is produced.
• Why ground truth is expensive, and why it can itself be wrong (label noise).
• The crucial split between tasks with one correct answer and tasks with many.
• How the shape of your ground truth decides which scoring is possible.

Where ground truth comes from

The known-correct answers don't appear for free. They come from one of a few sources, each with a cost/quality trade:

• Expert labelling. People who know the domain write or verify the correct answer per input. Highest quality for judgment-heavy tasks; slowest and most expensive.
• Deterministic derivation. When the correct answer can be computed — e.g. the intent that a canonical query for this question would use — you derive it mechanically. Cheap, exact, and reproducible, but only possible when the task has a computable right answer.
• Consensus / adjudication. Multiple annotators label independently and you resolve disagreements. Reduces individual bias at the cost of coordination.
• Existing signals. Sometimes reality provides labels — a user's correction, a downstream success/failure — usable as (noisy) ground truth.

Ground truth can be wrong

It is tempting to treat the reference answers as infallible. They are not: labels carry noise — honest mistakes, stale answers, and genuine disagreement between annotators. When a system "fails" a case, the first question is sometimes was the model wrong, or was the label wrong?

One right answer, or many?

The single most consequential property of a task, for evaluation, is how many correct answers an input has:

• Single reference. One canonical correct answer per input — a classification label, a structured intent, a numeric result. Scoring can be exact and mechanical: match or not.
• Many valid references. Open-ended tasks — a summary, an explanation, a piece of code — have many answers that are all correct while wording differently. There is no single string to match, so exact match is hopeless; you need semantic or rubric scoring, or human judgment.

   single valid answer            many valid answers
   ─────────────────────          ─────────────────────
   classification label           a summary
   a structured intent            an open-ended answer
   a computed result              working code
   → exact match works            → exact match fails; need semantic / rubric / human

This split decides the whole scoring strategy of the next two chapters. Tasks with a single reference are a gift — you can score them cheaply and reproducibly; tasks with many valid answers are where evaluation gets hard.

Mental model

Ground truth is the manufactured set of known-correct answers you score against. It costs effort, it can be wrong, and — most importantly — a task has either one canonical answer (score by matching) or many valid ones (needs semantic/rubric/human judgment). That single property drives your whole scoring strategy.

Common mistakes

• Treating references as infallible. Labels have noise; a strong system's "failures" are worth auditing for bad ground truth.
• Underfunding ground truth. It caps the whole evaluation; skimping on it caps your ceiling invisibly.
• Forcing exact match on a many-answer task. Open-ended outputs have many valid forms; exact match will mark correct answers wrong.
• Ignoring the single-vs-many question. It determines which scoring is even possible; decide it before choosing a method.

Practical guidance

• Pick the cheapest trustworthy source: derive ground truth deterministically when the task allows it; use experts/consensus when it doesn't.
• Audit the references a strong system disagrees with — you'll find (and fix) label noise and discover a real ceiling.
• Decide early whether your task has one correct answer or many, and let that choose your scoring approach.
• Where you can, design the task to have a single canonical answer (a structured intent, a label) — it makes everything downstream cheaper and more reproducible.

Summary

• Ground truth is the manufactured set of correct answers; its quality caps the evaluation.
• It is expensive and can be wrong — label noise creates ceilings you must audit for.
• The decisive property is single vs many valid answers: single → exact mechanical scoring; many → semantic/rubric/human.
• This project's single-reference intents (by design) are why its scoring is mechanical and reproducible.

Knowledge check

Your best system scores 92% and the 8% "failures" are concentrated on a handful of inputs it's very confident about. What should you check first?

The ground truth on those inputs. A strong, confident system disagreeing with the label is a classic signal of label noise — the reference may be wrong, stale, or ambiguous. Audit those cases: if the labels are bad, the system was actually right and your real accuracy is higher, and you've found a hidden ceiling. Don't assume the system is wrong just because it disagrees with the reference.

Why can't you score a summarization task with exact match, and what does that imply about its ground truth?

Because a summary has many valid forms — countless different wordings are all correct — so there's no single reference string to match; exact match would fail correct summaries. That means its ground truth can't be a single canonical answer; you need reference-free or rubric-based scoring (criteria a summary must satisfy) or human judgment, which is exactly why open-ended tasks are harder and costlier to evaluate than single-reference ones.

Related chapters

• The evaluation contract — the definition of correct that ground truth encodes
• Deterministic and subjective scoring — what the single-vs-many split lets you do
• Exact match, semantic scoring and rubric scoring — methods for each kind of reference
• Ground truth and reference construction — building references at dataset scale