Part 4 — Evaluation Data and Dataset Design

Adversarial and boundary cases

Representative and adversarial data·Evaluation·6 min read

A representative set tells you average quality; it won't tell you where the system breaks. For that you deliberately construct hard cases — boundary conditions, near-miss distinctions, and edge inputs — and score them separately, so failure modes surface instead of being averaged into a comfortable mean.

A representative test set answers "how good is the system on typical inputs?" It is quiet about the more useful question: "where does it break, and how close is the edge?" Average-case data averages failures away. To find the cliffs, you build a set on purpose out of the hardest cases — the boundaries where two answers are nearly identical, the edges of the input space, the near-misses — and you keep its score separate so it does its job instead of muddying the headline number.

What you will understand by the end

  • Why average-case data hides failure modes, and adversarial data exposes them.
  • The kinds of hard case: boundary distinctions, edge inputs, and near-misses.
  • Why the adversarial score is reported separately, not blended into the aggregate.
  • How adversarial cases turn into a tool for improving the system, not just grading it.

Why you need hard cases on purpose

Failures cluster. They don't spread evenly across the input space; they pile up at specific boundaries — the points where the correct answer flips from one thing to a very similar other thing. A representative set contains a few of these by chance, but their signal drowns in the easy majority. Building a set targeted at the boundaries concentrates the signal: instead of one failure in a hundred easy cases, you get a slice designed to make the system slip if it's going to.

Key idea

Average-case accuracy tells you how the system does on the inputs it will mostly see; adversarial accuracy tells you where its understanding is thin. The first is what you report to stakeholders; the second is what tells you what to fix. You need both, and they are different datasets built for different jobs.

The kinds of hard case

  • Boundary distinctions. Two readings that are semantically adjacent, where a small cue decides which is correct — a ranking vs a breakdown, a filter vs a group-by. These are where "valid but wrong" failures live.
  • Edge inputs. The extremes of the input space — very long, very short, empty, malformed, multi-part, or out-of-scope requests — that a middle-of-the-distribution set never probes.
  • Near-misses. Inputs a hair away from a case the system handles well, designed to test whether it understood the concept or just pattern-matched the easy version.
Watch out

Don't fold the adversarial score into your headline accuracy. Adversarial cases are deliberately harder than production, so blending them understates typical quality and hides the boundary signal in the average. Report adversarial separately — "94% held-out, 62% adversarial" says far more than a merged 90% that means neither.

Boundaries are the instructive failures

The most valuable adversarial cases sit exactly on a semantic boundary, because a failure there is diagnostic: it tells you the precise distinction the system hasn't learned. A model that handles clear rankings and clear breakdowns but fails on the phrasings that sit between them has told you exactly what to teach it. This is the direct feeder into semantic boundary failures in the next part.

Observed evidence

This project keeps a dedicated adversarial (8) slice of boundary cases separate from the held-out (60). The signature failure it targets is real: on "Revenue by region", the boundary between a ranking (top-N) and a breakdown (group-by) — a small phrasing cue with a large consequence — is exactly where raw models slipped, producing valid-but-wrong intents. The adversarial slice makes that fragility measurable instead of letting it hide in the average. See boundary failures →

From grading to improving

Adversarial cases aren't only a grade — they're a backlog. Each boundary the system fails on is a concrete thing to fix (a prompt clarification, a correction rule, a training example), and once fixed it becomes a regression test that stops the failure returning. The best source of new adversarial cases is production: real failures, distilled to their boundary, added to the set (the loop closed in turning incidents into test cases).

Mental model

Representative data measures typical quality; adversarial data — boundary distinctions, edge inputs, near-misses — measures where the system breaks. Build hard cases on purpose, score them separately from the headline, and treat each failure as both a diagnosis and a future regression test.

Common mistakes

  • Only testing average cases. Failures cluster at boundaries; an average-case set averages them away.
  • Blending adversarial into the headline. It understates typical quality and buries the boundary signal; report it separately.
  • Adversarial cases that are just weird, not boundary. Random strangeness is less useful than targeted near-misses at real decision boundaries.
  • Grading with them but never fixing. The point is to turn each failure into a fix and a regression test, not just a lower score.

Practical guidance

  • Build an adversarial slice targeted at real decision boundaries, edge inputs, and near-misses — not random oddities.
  • Report it separately from representative accuracy; two numbers, clearly labelled.
  • Mine production failures for new adversarial cases, distilled to the boundary they expose.
  • Turn each fixed adversarial failure into a regression test so it can't silently return.

Summary

  • Average-case data hides failure modes; adversarial data — boundaries, edges, near-misses — exposes them.
  • Report adversarial accuracy separately; blending it understates typical quality and buries the signal.
  • Boundary failures are the most instructive — they name the exact distinction to fix and feed semantic boundary analysis.
  • Adversarial cases double as a backlog and regression suite, best sourced from real production failures.

Knowledge check

A system scores 96% on held-out data and a stakeholder wants to ship. What one number would you add before deciding, and why?

Its adversarial accuracy, reported separately. The 96% is typical-case performance; it says nothing about the boundary cases where the system is most likely to fail catastrophically. If the adversarial slice is, say, 55%, you know exactly which distinctions are fragile and can decide whether those failures are acceptable for launch — a judgment the blended headline number would have hidden entirely.

Why is a failure on a boundary case more useful than a failure on a random weird input?

Because a boundary failure is diagnostic: it pinpoints the specific semantic distinction the system hasn't learned (e.g. ranking vs breakdown), which tells you exactly what to fix — a prompt clarification, a correction rule, a training example. A random weird input may fail for reasons that don't generalise or matter, giving you a lower score but no clear action. Targeted near-misses convert failures into fixes.

Related chapters