Part 5 — Failure Analysis and System Improvement

Aggregate metrics versus individual failures

Reading failures·Evaluation·6 min read

The average is for tracking; the individual failures are for understanding. This chapter is about holding both — using the aggregate to know whether you're improving, and the distribution of failures underneath it to know what's actually broken — and never letting the mean flatten a lopsided failure profile.

The previous chapter said: open the individual trials. This one is about the relationship between those trials and the number on top of them. An aggregate metric and the individual failures are not rivals — they answer different questions. The aggregate tells you whether the system is getting better; the individual failures tell you what is wrong. Confuse their jobs — optimise the mean without reading the cases, or drown in cases without tracking the mean — and you either improve the wrong thing or can't tell if you improved at all.

What you will understand by the end

  • What the aggregate is good for, and what it structurally hides.
  • Why two systems with the same average can have completely different failure profiles.
  • How to use aggregate and individual views together, each for its job.
  • Why optimising the mean blindly can make the system worse where it matters.

The average is a summary, and summaries hide

An aggregate is a compression: it throws away everything about which cases failed and how to produce one number. That compression is exactly what makes it useful for tracking — one number you can watch over time — and exactly what makes it blind to structure. A mean of 80% is consistent with "fails evenly and mildly everywhere" and with "perfect on 80% and catastrophically wrong on a critical 20%." Those are different systems with the same number.

Key idea

The aggregate answers "am I improving?"; the individual failures answer "what is broken?". Use the mean to track direction over time and the case-level view to understand and fix. Neither replaces the other — a number with no cases behind it can't be acted on, and cases with no number can't tell you if a change helped.

Same average, different systems

Because the mean flattens the distribution, equal aggregates routinely hide opposite realities:

  • One system fails a little on many inputs; another fails totally on a concentrated few. Same average; the second has a critical hole the first doesn't.
  • One system's failures are spread across all segments; another's are all in one user cohort. Same average; the second is unusable for that cohort.
  • One system's errors are near-misses; another's are wild. Same average; very different user harm.

Only looking under the average — by segment, by failure type, by severity — separates these. A single number can't, and treating it as if it could is how a "fine" system ships with a catastrophic blind spot.

Watch out

Optimising the aggregate blindly can degrade the cases that matter most. A change that lifts the mean by fixing many easy inputs while breaking a few critical ones looks like progress and is a regression where it counts. Always check the aggregate move against the distribution of what changed — which cases got better, which got worse.

Use both, each for its job

The discipline is to keep both views live:

  • Aggregate — track it over time and across configs to answer "is this change an improvement, and is it real?" (with the uncertainty from Part 5 attached).
  • Distribution — break the number down by segment, failure type, and severity, so you see where the failures concentrate and how bad they are.
  • Individual traces — read the actual cases to understand the mechanism.

Move from the number, to the breakdown, to the cases, and back. The number tells you something moved; the breakdown and cases tell you whether it moved the right way.

Observed evidence

In this project, the aggregate said the 3B guardrail added ~25 points — a clear improvement. But the same guardrail, applied to the 7B model, regressed it: the aggregate went the wrong way because the rule broke cases the stronger model already got right. Only looking under the mean — which cases the rule changed, for better and worse — explains why one number rose and another fell from the "same" change. See which cases the policy moved →

Mental model

The aggregate is for tracking (am I improving?); the individual failures are for understanding (what's broken?). The mean flattens the failure profile, so equal averages can hide opposite realities — spread-thin vs concentrated-critical. Watch the number, but always check the distribution and the cases behind it, especially when optimising.

Common mistakes

  • Treating the average as the whole story. It hides which cases fail and how badly; equal means can be very different systems.
  • Optimising the mean blindly. Lifting the average while breaking critical cases is a regression dressed as progress.
  • Only reading cases, never tracking the mean. Without the aggregate you can't tell if your fixes are net improvements.
  • Not breaking the number down. A single aggregate with no per-segment/per-type view can't reveal concentrated failure.

Practical guidance

  • Keep the aggregate for direction (over time, across configs) and the distribution + traces for understanding; use each for its job.
  • Break every headline number down by segment, failure type, and severity before trusting it.
  • When a change moves the mean, inspect which cases changed — confirm nothing critical regressed.
  • Weight by severity, not just count — a few catastrophic failures can matter more than many mild ones the mean treats equally.

Summary

  • The aggregate tracks improvement; the individual failures explain it — different jobs, both needed.
  • The mean flattens the failure profile, so equal averages can hide opposite realities (spread vs concentrated, mild vs catastrophic).
  • Optimising the mean blindly can regress the cases that matter; check the distribution of what changed.
  • This project's guardrail — +25 on one model, a regression on another — is the mean hiding opposite case-level effects.

Knowledge check

Two classifiers both score 90% on your test set. Why might you strongly prefer one, and what view reveals the difference?

Because 90% can hide very different failure profiles: one might fail mildly and evenly across all classes, while the other is perfect on common classes and fails completely on a rare but critical one. Same mean, very different systems — the second has a hole the aggregate can't show. Breaking the number down by class and severity (and reading the failing cases) reveals it; the single 90% treats the two as identical.

A change raises your aggregate accuracy by 3 points. Why isn't that automatically good news?

Because the mean can rise while the change breaks the cases that matter most — e.g. it fixes many easy inputs (lifting the average) but regresses a few high-severity or high-frequency-critical ones. Averaged together that reads as progress but is a regression where it counts. You have to inspect the distribution of what changed — which cases improved, which got worse, weighted by severity — before calling a 3-point aggregate gain an improvement.

Related chapters