Part 2 — Complete LLM Systems
The model is only one component
Understand why a measured accuracy belongs to a whole versioned configuration — model, prompt, schema, decoding, provider settings, parsing, validation, and correction policy — not to 'the model,' and why conflating the two is the most common evaluation mistake.
You are about to compare some language models. You send each one the same question, score the answers, and rank them. It feels like you are measuring the models.
You are not. You are measuring systems — and the difference is the first thing a serious evaluator has to internalise, because almost every misleading benchmark you will ever read comes from forgetting it.
What you will understand by the end
- Why an accuracy number belongs to a configuration, not to a model.
- The seven-or-so parts that make up the thing you are actually measuring.
- How to tell, from a single trial, whether "the model" or "the system around it" produced a result.
- Why comparing a bare model against a model-plus-policy is an unfair fight — and how to compare fairly instead.
Core concept: the unit of evaluation is a configuration
When a request turns into a score, a lot has happened between the two. A prompt was assembled. A schema constrained the output. The model decoded tokens under a temperature and a provider's settings. The raw text was parsed and validated. Maybe a correction policy rewrote part of it. Only then was the result compared to a known-correct answer.
Every one of those steps can change the score. So the score does not describe the model; it describes the whole pipeline, frozen at specific versions:
A measured result belongs to a versioned configuration: model + prompt + output schema + decoding parameters + provider settings + parsing + validation + correction policy. Change any one and you may be measuring a different system.
Call that configuration a system. It is the atom of evaluation. "Claude Haiku scored 95%" is shorthand; the honest statement is "this configuration, which happens to use Claude Haiku as its model, scored 95% on this dataset under these settings."
Nothing here is specific to query generation. A RAG system is retriever + reranker + prompt + model + post-processing; an agent is tools + memory + planner + model; a classifier is prompt + model + a decision threshold; a summarizer is prompt + model + length/format constraints. In every case the score belongs to the whole configuration, and every lesson in this chapter transfers unchanged. This book uses one running case study — a natural-language-to-query system — but you should read "system" as your system.
The case study on this site makes the point concrete: a leaderboard row is never labelled with just a model name — each row is a system, and two of the rows use the same family of open model wrapped in different policy, which is only sensible if the system, not the model, is the unit being compared.
Mental model: two dials that are easy to confuse
Hold two quantities apart in your head:
- Raw model quality — what the model produced on its own, validated but untouched.
- Corrected system quality — what the whole pipeline produced after the correction policy did its work.
Reporting one number and calling it "the model's accuracy" hides which dial moved. A system can look strong because its model is strong, or because a deterministic policy patched a weak model's mistakes. Those are completely different engineering situations, and only reporting both dials tells them apart.
The rule that falls out of this: always report raw model accuracy and corrected system accuracy separately. The gap between them is the value your engineering added — and sometimes the gap is negative, which you would never discover if you only tracked the final number.
A concrete example: one trial, two answers
Here is a single real trial from the A1.5 experiment. The task looks trivial.
Question: "Revenue by region last quarter" Split: development
The correct answer is a breakdown — group revenue by region and list every
region. In the intent schema this project uses, that is the pattern
metric_by_dimension_with_filter, sorted by region, no truncation:
{ "pattern_id": "metric_by_dimension_with_filter",
"metric": "revenue", "dimensions": ["region"],
"sort": [{ "field": "region", "direction": "asc" }],
"limit": 100 }
The raw Qwen 3B model produced this instead — valid, schema-compliant, and semantically wrong:
{ "pattern_id": "top_n_by_metric",
"metric": "revenue", "dimensions": ["region"],
"sort": [{ "field": "revenue", "direction": "desc" }],
"limit": 10 }
It heard "by region" as a ranking — top-10 regions by revenue, descending — rather than a breakdown of all of them. Different query, different rows, wrong answer.
Then a deterministic correction policy in the system noticed the mismatch (a
top_n prediction with no ranking language in the question) and downgraded the
pattern back to metric_by_dimension_with_filter. The system's final answer was
correct.
On this trial the raw model was wrong and the complete system was right. If you had recorded only "Qwen 3B: correct," you would have credited the model for work the policy did. Open every trial the policy touched →
The same example also demolishes a comforting assumption. Every one of the six systems produced 100% schema-compliant JSON — every output parsed, every field validated. Yet raw semantic accuracy ranged from 45.5% (Qwen 3B) to 95.5% (the best hosted systems). Valid structure told you nothing about whether the answer meant the right thing.
Schema compliance is a syntax check, not a correctness check. 100% valid JSON and 45% correct answers coexisted in this very experiment.
Experiment walkthrough: A1.5, the six-system comparison
The question. Given a fixed natural-language-to-query task, how should we close the semantic-quality gap — a bigger open model, a deterministic guardrail, or a hosted frontier model?
The systems. Six complete configurations on the same contract: Qwen 3B + policy, Qwen 7B + policy, Claude Haiku, Claude Sonnet, and two OpenAI models. The model differs; so does whether a correction policy is present. That asymmetry is deliberate and only legitimate because we are comparing systems.
The dataset & method. 88 tasks split into development (20), held-out (60), and adversarial (8). Each system produced one attempt per task; a deterministic verifier scored each attempt by exact match against the expected intent, and recorded both raw and corrected outcomes.
The result. Two systems tied for the top at 95.5%. The open models improved a lot from 3B to 7B but stayed below the gate. The 3B guardrail added 25 points on its own model — and, when blindly transferred to 7B, mostly regressed it.
It does not prove one model is "more accurate" than another in general. It compares six configurations on one 88-task dataset with one attempt each. Move the dataset, the prompt, or the policy and the numbers move with them. The result is a provisional routing decision, not a universal model ranking.
Common mistakes this concept prevents
- Crediting the model for the system's work. "Qwen 3B got 70%" — no; the system did, and 25 of those points came from a hand-written rule, not the model.
- Unfair comparisons. Benchmarking a bare hosted model against your model-plus-guardrail and declaring your stack "as good as" a frontier model. You compared different kinds of thing.
- Transferring a policy blindly. The 3B guardrail helped 3B and hurt 7B. A policy is part of a system, tuned to that model's failure mode; it is not a free add-on.
- Trusting valid JSON. Structured output guarantees shape, not meaning. Treating a schema pass as a correctness signal is how "100% valid" hides "45% right."
Practical guidance
When you run an evaluation, treat the system as the unit:
- Record every part of the configuration — model id, prompt version, schema, decoding params, provider settings, parser, validator, and policy — as one versioned manifest attached to the result.
- Freeze it for the duration of a comparison. If you must change something, change one part and re-run, so a delta is attributable.
- Always report raw and corrected accuracy. The gap is your policy's real effect — including when it is negative.
- Compare like with like. If one system has a correction policy, either give the others a fair equivalent or clearly mark the asymmetry.
- State what a result licenses. "This configuration is the provisional default on this dataset" is defensible. "This model is the best" usually is not.
Summary
- The thing you measure is a system — a versioned configuration — not a model.
- Keep two dials apart: raw model quality and corrected system quality; report both.
- A single trial ("Revenue by region") showed the raw model wrong and the system right, and showed 100% valid JSON sitting on top of 45% correctness.
- The A1.5 comparison is a comparison of configurations, and its result is a provisional routing decision, not a model ranking.
Knowledge check
A system scores 70%. Its raw model scores 45%. What produced the other 25 points, and why must you report it separately?
The correction policy did. Reported as one number, the 25 points look like model quality; they are engineering added around a weak model. Only separating raw from corrected reveals that the model is actually weak and the system is being carried by a deterministic rule.
Every system emits 100% valid JSON. Can you conclude anything about correctness?
No. Schema compliance is a syntax guarantee. In this experiment 100% valid JSON coexisted with raw accuracy from 45.5% to 95.5%. Validity and meaning are different axes.
Why is it unfair to compare a bare hosted model against your model-plus-guardrail?
They are different systems. The guardrail is part of your configuration; the hosted model has no equivalent. A fair comparison either equips both with comparable policy or explicitly marks the asymmetry so the reader is not misled.
Related evidence
Follow these into the live project to see the concept as data, not prose: