Part 4 — Evaluation Data and Dataset Design

Taxonomy and label design

Labels and ambiguity·Evaluation·6 min read

Before you can label data or score outputs, you need a set of categories — a taxonomy. Good label design is quietly decisive: categories that are clear, distinct, complete, and aligned with the decision you'll make turn a task into something measurable; bad ones make every downstream number unstable.

Many evaluation tasks reduce to assigning things to categories — the label a classifier predicts, the pattern a query maps to, the error type a failure gets. That set of categories is a taxonomy, and designing it well is one of the least glamorous, most consequential steps in building an evaluation. A muddled taxonomy — overlapping, incomplete, or mis-grained categories — makes labels inconsistent and scores unstable no matter how good your model or your scoring rule is.

What you will understand by the end

  • What a taxonomy is and where it shows up in evaluation.
  • The four properties of a good label set: distinct, complete, right-grained, actionable.
  • Why a bad taxonomy corrupts labels and scores, upstream of everything else.
  • How the schema in this project is a taxonomy in disguise.

What a taxonomy is, and where it hides

A taxonomy is the fixed set of categories a task uses. It appears all over evaluation:

  • The classes a classifier chooses between.
  • The patterns a structured-output task maps inputs to (this project's query intents).
  • The error types a failure taxonomy sorts failures into.
  • The segments you stratify a dataset by.

In every case the same design question applies: are these the right buckets, defined clearly enough that two people put the same item in the same one?

Four properties of a good label set

  • Distinct (mutually exclusive). Each item belongs in exactly one category. Overlapping categories force arbitrary choices, and the same item lands in different buckets on different days.
  • Complete (collectively exhaustive). Every item that can occur has a home — including an explicit "other" or "out of scope" so real inputs aren't jammed into a category that doesn't fit.
  • Right-grained. Not so coarse that important distinctions collapse into one bucket, not so fine that categories are indistinguishable and labels become guesses. Granularity should match the decisions you'll make.
  • Actionable. The categories should map to things you'd do differently. A distinction that changes no decision adds labelling cost and noise for no benefit.
Key idea

A good taxonomy is distinct, complete, right-grained, and actionable — its categories are clear enough that labellers agree, cover everything that occurs, cut at the granularity your decisions need, and each map to a different action. Get those four right and labelling is consistent; get any wrong and every label and score built on top inherits the confusion.

A bad taxonomy corrupts everything downstream

Label design sits upstream of ground truth and scoring, so its flaws propagate. Overlapping categories create annotation disagreement that looks like annotator error but is really a design bug. An incomplete set forces real inputs into ill-fitting buckets, poisoning both the labels and any per-category metric. Wrong granularity either hides failures (too coarse) or manufactures noise (too fine). No amount of careful scoring recovers from a taxonomy that was wrong to begin with.

Watch out

When annotators keep disagreeing on the same kind of item, the reflex is to write more labelling guidelines. Often the real fix is upstream: the taxonomy has overlapping or ill-defined categories. Redesigning the buckets — splitting, merging, adding an explicit "other," sharpening definitions — fixes the disagreement at its source, where more guidelines just paper over it.

The project's schema is a taxonomy

Observed evidence

This project's QueryIntent is a taxonomy in disguise: each question maps to one pattern (e.g. top_n_by_metric vs metric_by_dimension_with_filter), and those patterns are designed to be distinct and exhaustive so exact-match scoring is well-defined. The whole "ranking vs breakdown" failure is a boundary between two taxonomy categories — which is only a crisp, measurable failure because the categories themselves are cleanly designed. See the pattern taxonomy →

Mental model

A taxonomy is the fixed set of categories a task uses — classes, patterns, error types, segments. Good ones are distinct, complete, right-grained, and actionable, so labellers agree and metrics mean something. Because it sits upstream of labels and scores, a bad taxonomy corrupts everything below it — and persistent disagreement is usually a taxonomy bug, not an annotator bug.

Common mistakes

  • Overlapping categories. Items fit more than one bucket; labels become arbitrary and unstable.
  • No "other"/out-of-scope bucket. Real inputs get forced into ill-fitting categories, poisoning labels and per-class metrics.
  • Wrong granularity. Too coarse hides real distinctions; too fine makes categories indistinguishable and labels a guess.
  • Categories no one acts on differently. Distinctions that change no decision add cost and noise for nothing.

Practical guidance

  • Design the taxonomy to be mutually exclusive and collectively exhaustive, with an explicit other/out-of-scope category.
  • Choose granularity by the decisions you'll make with the labels — no finer, no coarser.
  • Pilot the taxonomy on real data and measure agreement; fix persistent disagreement by redesigning categories, not just adding rules.
  • Treat the taxonomy as versioned: changing categories changes the meaning of every label and every per-category metric.

Summary

  • A taxonomy is the category set behind classes, patterns, error types, and segments — and it sits upstream of everything.
  • Good label sets are distinct, complete, right-grained, and actionable.
  • A bad taxonomy corrupts labels and scores and shows up as annotation disagreement that redesign, not more guidelines, actually fixes.
  • This project's pattern schema is a clean taxonomy — which is what makes its boundary failures measurable.

Knowledge check

Annotators keep disagreeing on which of two categories a certain kind of input belongs to. Your teammate proposes writing detailed tie-breaking rules. What's the more fundamental fix to check first?

Whether the taxonomy is the problem — the two categories likely overlap or are ill-defined, so the item genuinely fits both. The more fundamental fix is to redesign the buckets: sharpen the definitions, split or merge categories, or add an explicit boundary rule at the taxonomy level. Tie-breaking guidelines paper over a design flaw; fixing the categories removes the disagreement at its source (and makes every future label consistent).

Why can a taxonomy that's too fine-grained be as harmful as one that's too coarse?

Too coarse collapses distinctions you care about into one bucket, hiding real differences in behaviour. Too fine creates categories so similar that labellers can't reliably tell them apart, so labels become guesses and per-category metrics turn to noise — and it adds distinctions no decision depends on. Granularity should match the decisions you'll make: fine enough to separate what you'd act on differently, coarse enough that labellers agree.

Related chapters