Part 3 — Evaluation Foundations

Evaluating complete pipelines

Valid is not correct·Evaluation·6 min read

Real systems are multi-stage — retrieve, prompt, model, parse, validate, correct, execute — and a single end-to-end score hides which stage failed. This chapter closes Part 4 by scoring the whole pipeline both ways: end-to-end for the user outcome, and stage-by-stage to locate the fault.

The chapters so far scored an output against an answer. But an LLM application is not one output — it is a pipeline of stages, and a failure at any one of them shows up as "the answer was wrong." A single end-to-end number tells you that the system failed; it never tells you where. Evaluating a complete pipeline means measuring it two ways at once: end-to-end for what the user experienced, and stage-by-stage for where to fix it. This closes Part 4 and opens the door to failure analysis.

What you will understand by the end

  • Why a single end-to-end score hides where a multi-stage system failed.
  • The difference between end-to-end and stage-level evaluation, and why you need both.
  • How the raw-vs-corrected split you already met is a two-stage pipeline view.
  • How this connects Part 4 to failure analysis.

An LLM system is a pipeline of scorable stages

Recall the anatomy: a request flows through retrieval, prompt assembly, the model, parsing, validation, a correction policy, and execution. Each stage takes an input and produces an output — which means each stage can be scored on its own:

   retrieve ─▶ prompt ─▶ model ─▶ parse ─▶ validate ─▶ correct ─▶ execute ─▶ answer
      │           │        │        │          │           │          │
   recall?    complete?  intent  parses?   schema-ok?   fixed vs   query runs?
   right docs?           right?             (syntax)    broke it?  right rows?
      └──────────── stage-level metrics localise the failure ──────────────┘
   └────────────────────── end-to-end metric: did the user get the right answer? ─────────┘
Key idea

An end-to-end score answers "did it work?"; stage-level scores answer "where did it break?". You need both. End-to-end is the outcome the user actually got; stage-level is the diagnostic that turns a red number into an action. Reporting only end-to-end is like a smoke alarm with no location — you know there's a fire, not which room.

Why end-to-end alone misleads

Two systems can share the same end-to-end accuracy for completely different reasons — one has a weak retriever and a strong model, the other the reverse — and you'd treat them identically if all you saw was the final number. Worse, a stage can mask another: a good correction policy can hide a weak model, and a lenient final check can hide upstream damage. Only per-stage measurement separates these.

Watch out

An empty or wrong final answer is not automatically "the model was wrong." It might be a retrieval miss (nothing relevant fetched), a parsing bug (good output dropped), or a schema that under-constrained. Diagnosing a pipeline means asking which stage failed — and you can only ask that if you measured each stage. This is the core habit of failure analysis.

The raw-vs-corrected split is a two-stage view

You have already seen the smallest version of pipeline evaluation. In The model is only one component, reporting raw model accuracy and corrected system accuracy separately is exactly a two-stage measurement: score the model stage, then score after the correction stage, and the gap is that stage's contribution.

Observed evidence

This project measures both stages, which is the only reason its results are interpretable: scoring before and after the correction stage shows exactly how many points that stage contributed — something the end-to-end number alone would hide. What that contribution turns out to be (a large fix on one model, a regression on another) is the subject of guardrail intervention analysis; here the point is only that a two-stage measurement is what reveals it. See the correction stage's trials →

Mental model

A pipeline is a chain of scorable stages. Measure it end-to-end (the user's outcome, what you gate on) and stage-by-stage (where it broke, what you fix). A single final number hides which stage failed and lets one stage mask another; the raw-vs-corrected gap is the simplest example of reading a pipeline two ways.

Common mistakes

  • Only measuring end-to-end. You learn that it failed, never where; every fix becomes a guess.
  • Letting a stage mask another. A strong correction policy or lenient final check hides an upstream weakness unless you measure stages separately.
  • Not attributing a change. If you can't see which stage moved, you can't tell whether a gain came from the change you made or something else.

Practical guidance

  • Instrument each stage's input and output, and define a metric per stage (retrieval recall, parse rate, intent accuracy, execution success) alongside the end-to-end score.
  • Gate on end-to-end, debug with stage-level; when end-to-end moves, check the stage metrics to confirm it moved for the intended reason.
  • Always report raw and corrected (or pre/post any policy stage) — the gap is that stage's real effect, including when it's negative.
  • Feed stage-level failures into failure analysis — locating the stage is the first step of root-causing it.

Summary

  • LLM systems are pipelines of individually scorable stages; a single end-to-end score hides where they fail.
  • Measure both: end-to-end (the user outcome you gate on) and stage-level (the diagnostic you fix with).
  • The raw-vs-corrected split is the simplest pipeline view; its gap is a stage's contribution.
  • Locating the failing stage is the handoff from evaluation foundations into failure analysis.

Knowledge check

Two RAG systems both score 78% end-to-end. Why might you still strongly prefer one, and what would reveal the difference?

Because the same 78% can come from very different stage profiles — one might have 95% retrieval recall and a weaker generator, the other a great generator starved by 60% retrieval. They need opposite fixes and have different ceilings. Only stage-level metrics (retrieval recall vs answer groundedness, measured separately) reveal which is which; the end-to-end number treats them as identical.

Your system's end-to-end accuracy is 88%, and a teammate credits "the model." How does the raw-vs-corrected view test that claim?

Score the pipeline at two stages: the raw model output and the post-correction output. If raw is 88% too, the model earned it; if raw is 63% and correction lifts it to 88%, then 25 of those points are the correction stage, not the model. The gap between the two stage scores attributes the result to the correction stage rather than the model.

Related chapters