Part 10 — Integrated Case Studies

Designing the intent schema

Setting up the case study·Evaluation·6 min read

The QueryIntent schema is where the case study's evaluability was engineered. This chapter walks its design — the pattern taxonomy and the fields — and shows how a well-designed schema makes wrong answers hard to represent and turns the whole task into something exact match can score.

The use case hinged on the model producing a QueryIntent. That object isn't incidental — its design is where most of the case study's good properties come from. A schema built well makes the right answer easy to express, the wrong answer hard to represent, and the whole task scorable by exact match. This chapter walks that design: the pattern taxonomy at its heart and the fields that surround it, and why each choice matters for evaluation.

What you will understand by the end

  • The structure of the QueryIntent schema — pattern plus fields.
  • Why the pattern is a taxonomy, and why that taxonomy is the crux.
  • How the schema makes wrong answers hard to represent.
  • Why this design is what enables deterministic scoring.

The schema: a pattern and its parameters

A QueryIntent captures everything the deterministic builder needs to construct the query:

{ "pattern_id": "metric_by_dimension_with_filter",
  "metric": "revenue",
  "dimensions": ["region"],
  "filters": [ ... ],
  "sort": [{ "field": "region", "direction": "asc" }],
  "limit": 100 }
  • pattern_id — the shape of the query: is this a ranking (top_n_by_metric), a breakdown (metric_by_dimension_with_filter), a simple aggregate, and so on. This is the classification core.
  • metric, dimensions, filters — the extraction: what is measured, grouped by, and restricted to.
  • sort, limit — how results are ordered and truncated.

The model's job is to fill this in correctly; deterministic code does the rest.

The pattern is a taxonomy, and it's the crux

The pattern_id is a taxonomy — a fixed set of query shapes designed to be distinct and exhaustive. That design decision is load-bearing: because the patterns are cleanly defined, a question maps to one correct pattern, which is what makes exact-match scoring well-defined and what makes the signature failure — choosing top_n_by_metric (ranking) when the answer was metric_by_dimension_with_filter (breakdown) — a crisp, measurable error rather than a fuzzy judgment call.

Key idea

The pattern_id field is a query-shape taxonomy, and its quality determines everything downstream. Because the patterns are distinct and exhaustive, each question has one correct pattern — so the boundary between a ranking and a breakdown is a sharp, scorable line, not a matter of opinion. A muddier taxonomy would have made the whole experiment unmeasurable.

Making wrong answers hard to represent

A good output schema narrows the model's room for error, and this one is designed to do exactly that. By enumerating the allowed patterns, metrics, and dimensions, the schema makes many conceivable wrong answers unrepresentable — the model can't invent a metric that doesn't exist or a pattern outside the set. What remains representable is a small space of meaningful choices (which pattern, which metric), which is precisely where the interesting semantic failures live. The schema pushes the difficulty from "anything could go wrong" down to "the few decisions that matter."

Watch out

A schema constrains shape, never meaning — and this one is a perfect illustration. It guarantees a valid pattern_id from the allowed set, but it cannot guarantee the right pattern for the question. That residual gap — valid pattern, wrong pattern — is the entire semantic-error surface of the case study, and no schema design closes it; only measuring meaning does.

Why the design enables deterministic scoring

Pulling it together: because the schema has a canonical structure (normalized fields, an enumerated pattern), two intents can be compared by exact structural equality — after normalizing harmless variation like field order. That's what makes the deterministic verifier possible, which is what makes the whole six-system comparison reproducible. The schema design and the scoring method are two sides of one decision: build the output to have a canonical form, and scoring becomes mechanical.

Observed evidence

The schema did its job: across all six systems, output was 100% schema-valid — the enumerated patterns and typed fields left no room for malformed intents — so every failure that remained was a semantic one (wrong pattern, wrong filter), scorable by exact match against the one expected intent. Structure solved by design; meaning left as the thing to measure. See the schema and its patterns →

Mental model

The QueryIntent schema is a query-shape taxonomy (pattern_id) plus extraction fields (metric, dimensions, filters, sort, limit). Its cleanly-designed pattern taxonomy gives each question one correct answer, makes many wrong answers unrepresentable, and — via a canonical structure — lets exact match score it. The schema's design is where the case study's evaluability was engineered.

Common mistakes

  • A muddy pattern taxonomy. Overlapping or vague query shapes would make the ranking-vs-breakdown boundary unscorable and the whole experiment fuzzy.
  • Under-constraining the fields. A loose schema lets valid-but-unusable intents through; enumerate allowed values to make wrong answers hard to represent.
  • Expecting the schema to guarantee correctness. It guarantees a valid pattern, not the right one; meaning is still a separate measurement.
  • Non-canonical structure. Without normalization, exact match manufactures false negatives on harmless variation.

Practical guidance

  • Design the model's output around a clean taxonomy for the key decision (here, query shape) — distinct, exhaustive, scorable.
  • Enumerate allowed values (patterns, metrics, dimensions) so many wrong answers become unrepresentable and the error surface shrinks to what matters.
  • Give the schema a canonical form and normalize before comparing, so exact match is both possible and fair.
  • Accept that the schema handles shape; plan a separate semantic check for the residual wrong-but-valid space.

Summary

  • The QueryIntent schema is a pattern taxonomy plus extraction fields; the pattern_id is the crux.
  • A cleanly-designed pattern taxonomy gives each question one correct answer and makes the boundary errors crisp and scorable.
  • The schema makes many wrong answers unrepresentable, shrinking the error surface to the decisions that matter.
  • Its canonical structure is what enables deterministic exact-match scoring — schema design and scoring are one decision.

Knowledge check

How does the design of the `pattern_id` taxonomy make the "ranking vs breakdown" failure measurable rather than a judgment call?

Because the patterns (e.g. top_n_by_metric for a ranking vs metric_by_dimension_with_filter for a breakdown) are designed to be distinct and exhaustive, each question maps to exactly one correct pattern. So when a model outputs a ranking pattern where the expected answer is a breakdown, that's an unambiguous mismatch on a well-defined category — a crisp, exact-match-scorable error — rather than a fuzzy "is this close enough?" judgment. A muddy taxonomy with overlapping shapes would have made the same failure a matter of opinion; the clean taxonomy is what turns it into a measurement.

The schema guarantees 100% valid output. What does that guarantee, and what does the case study's 45.5%–95.5% accuracy range prove it does not guarantee?

It guarantees shape: every output is a well-formed QueryIntent with a legal pattern_id and typed fields from the allowed sets — no malformed or out-of-schema output. It does not guarantee meaning: the 45.5%–95.5% accuracy range (all at 100% validity) proves that a schema-valid intent can still choose the wrong pattern or filter for the question. Structure and correctness are independent axes; the schema closes the structural gap entirely and leaves the entire semantic gap open, which is exactly the space where the case study's failures live.

Related chapters