Part 4 — Evaluation Data and Dataset Design

Representative test construction

Representative and adversarial data·Evaluation·6 min read

An offline score only predicts production if the test set looks like production. This chapter is about building a dataset that mirrors the real input distribution — sampling from reality, covering the segments that matter, and balancing frequency against importance — so your lab number means something outside the lab.

An offline evaluation measures your system on your dataset, and then you act as if that number describes the real world. It only does so to the extent the dataset looks like the real world. A test set built from convenient or imagined examples produces a confident number that doesn't survive contact with production. This chapter is about closing that gap: constructing a dataset that is genuinely representative of the inputs your system will actually face.

What you will understand by the end

  • Why "representative" is the property that lets an offline number predict production.
  • How to build a dataset from the real input distribution, not convenient examples.
  • The tension between frequency (what's common) and importance (what's costly to get wrong).
  • Why a dataset is a model of reality you must maintain, not build once.

Representative means "shaped like production"

The input distribution your system meets in production has structure: common request types, rare ones, easy and hard cases, different user segments, different phrasings. A representative test set reproduces that structure. If production is 60% simple lookups and 5% multi-condition queries, a test set that is 5% lookups and 60% multi-condition measures a different system than the one users experience — and its number will mislead in whichever direction the mismatch runs.

Key idea

A test set is a sample of production, and an offline score is only a prediction of production performance to the degree the sample matches the real distribution. "Representative" isn't a nicety — it's the property that makes the offline number mean anything about the world outside your lab.

Build from reality, not imagination

The reliable way to get a representative set is to draw from real inputs:

  • Sample real traffic (logs, historical requests, user queries) rather than inventing examples — invented examples encode your assumptions, not users' behaviour.
  • Stratify by the segments that matter — request type, difficulty, user cohort, input length — so each important slice is present in roughly its real proportion.
  • Include the messy — typos, ambiguous phrasing, out-of-scope requests — because production is messy and a clean test set overstates quality.

When you can't sample real traffic (a new product), synthesise deliberately across the expected distribution and correct the mix as real data arrives.

Watch out

The easiest test set to build — the examples that came to mind, or the ones your system already handles well — is the least representative. It flatters the system by over-sampling the easy, familiar cases and under-sampling the long tail where failures live. If building the dataset felt effortless, suspect it mirrors your assumptions, not production.

Frequency versus importance

Two forces decide how much of each case type to include, and they pull apart:

  • Frequency — how often a case occurs. Common cases dominate the user experience, so under-representing them makes the aggregate number lie.
  • Importance — how costly a case is to get wrong. A rare request whose failure is a disaster deserves representation beyond its frequency.

A pure-frequency set can put near-zero weight on rare-but-critical cases; a pure-importance set over-weights edge cases and misstates typical quality. The usual answer is to cover both — represent the common cases in proportion, and guarantee coverage of the rare-but-critical ones (often as a separate slice, like the adversarial set), and report them so one doesn't hide the other.

Observed evidence

This project separates a held-out (60) set meant to reflect the ordinary distribution of business questions from an adversarial (8) set of deliberately hard boundary cases. That split is the frequency-vs-importance balance made concrete: the held-out number estimates typical performance, while the adversarial slice guarantees the rare-but-costly boundary cases are measured rather than averaged away. Compare the ordinary and adversarial slices →

A dataset is a model you maintain

Production drifts — new request types, changing phrasing, new user segments — so a test set that was representative last quarter may not be now. Representativeness is a property you maintain, refreshing the set as the input distribution moves (and as production failures reveal what you missed — coverage and blind spots).

Mental model

A test set is a sample of production; the offline number predicts production only as well as the sample matches it. Build from real traffic, stratify by the segments that matter, include the mess, and balance frequency (what's common) against importance (what's costly). Then keep it representative as reality drifts.

Common mistakes

  • Inventing examples. They encode your assumptions, not users' behaviour; sample real inputs instead.
  • Over-sampling the easy cases. The convenient set flatters the system and hides the long tail.
  • Pure frequency or pure importance. One buries rare-but-critical cases; the other misstates typical quality. Cover both, reported separately.
  • Building it once. Production drifts; an un-maintained set slowly stops predicting reality.

Practical guidance

  • Sample from real traffic where you can; stratify by request type, difficulty, and segment to match the real proportions.
  • Deliberately include messy and out-of-scope inputs so the number reflects production, not a clean lab.
  • Represent common cases in proportion and guarantee coverage of rare-but-critical ones as a separate, separately-reported slice.
  • Refresh the set as the distribution drifts and as blind spots surface.

Summary

  • A test set is a sample of production; the offline number predicts reality only if the sample matches it.
  • Build from real inputs, stratified by the segments that matter, including the mess.
  • Balance frequency (common cases, in proportion) against importance (rare-but-costly, guaranteed coverage) and report them separately.
  • Representativeness is maintained, not built once — this project's held-out vs adversarial split is that balance made concrete.

Knowledge check

Your system scores 94% offline but users complain constantly. The test set was built from examples the team wrote by hand. What's the most likely problem?

The hand-written set probably isn't representative: it over-samples the clean, familiar cases the team thought of and under-samples the messy, long-tail inputs real users send, so the 94% describes an easier distribution than production. The fix is to rebuild the set from real traffic, stratified to match the true mix (including typos, ambiguity, and out-of-scope requests), so the offline number tracks what users actually experience.

Why isn't sampling test cases purely by frequency always the right call?

Because a rare case can be critically important — its failure is catastrophic even though it occurs seldom — and pure-frequency sampling would give it almost no weight, so a system that fails it entirely still scores well. You want the common cases represented in proportion (so the aggregate is honest) and guaranteed coverage of the rare-but-costly cases (often as a separate slice), reported separately so neither hides the other.

Related chapters