Part 5 — Failure Analysis and System Improvement

Error taxonomies

Classifying failures·Evaluation·6 min read

A pile of individual failures is overwhelming until you sort it. An error taxonomy — a set of failure types built from real traces — turns that pile into a ranked, countable backlog, so you fix the failure classes that matter most instead of reacting to whichever case you saw last.

Reading individual traces surfaces the failures; but fifty individual failures are fifty problems, and you can't fix fifty things at once. The move that turns reading into action is grouping — sorting failures into a small set of types, an error taxonomy. Once failures are typed and counted, the chaos becomes a backlog you can rank: this type accounts for 40% of failures, that one for 5%. You stop reacting to the last case you saw and start fixing the classes that actually move the number.

What you will understand by the end

  • What an error taxonomy is and why it turns failures into a prioritisable backlog.
  • Why you build it bottom-up from real traces, not top-down from imagination.
  • The properties it shares with any good taxonomy — distinct, complete, actionable.
  • How typed, counted failures drive what you fix first.

From a pile of failures to a set of types

An error taxonomy is a small set of failure categories — "wrong pattern," "wrong filter," "parse error," "retrieval miss," "hallucinated field" — that every failure can be sorted into. Tagging each failing trace with a type converts an unstructured pile into a distribution: how many failures of each kind. That distribution is the thing you can act on — it tells you which failure class is worth an engineer's week and which is a rounding error.

Key idea

An error taxonomy turns "we have lots of failures" into "40% are wrong-pattern, 25% are wrong-filter, 15% are retrieval misses, and the rest are a long tail." That ranking is what lets you fix by impact instead of by recency — attacking the failure class that accounts for the most failures, not whichever trace you happened to read last.

Build it bottom-up from real traces

The temptation is to invent the failure categories up front, from how you think the system fails. Resist it: a taxonomy dreamed up in advance inherits your assumptions and misses the failure modes you didn't anticipate — the same blind-spot problem as imagined datasets. Build it the other way:

  • Read a batch of real failing traces without categories in mind.
  • Let the types emerge — cluster failures that share a mechanism.
  • Name and define each type crisply enough that two people tag the same failure the same way.
  • Iterate as new failures reveal types you didn't have; the taxonomy is a living artifact.
Watch out

An error taxonomy is a taxonomy, so it inherits the same failure modes: overlapping types (a failure that fits two) make counts arbitrary, an incomplete set forces failures into an ever-growing "misc" bucket, and non-actionable types (categories no fix maps to) waste the effort. Apply the distinct-complete-right-grained-actionable test to your failure types too.

Typed failures rank the work

Once failures are typed and counted, prioritisation becomes arithmetic. Rank types by frequency × severity and you have a backlog: the wrong-pattern class causing 40% of failures on critical inputs is your first target; the rare parse error is not. Each type also points at a kind of fix — a semantic error type suggests prompt or training work, a parse type suggests plumbing, a retrieval type suggests the index — which is the handoff to root-cause analysis.

Observed evidence

This project's failures sort into a small, telling taxonomy: the dominant type is wrong pattern — choosing top_n_by_metric when the answer needed metric_by_dimension_with_filter — with wrong-filter and other types trailing. Because failures are typed, the fix was targeted: a correction rule aimed precisely at the dominant "ranking-vs-breakdown" class, which is exactly why the 3B guardrail could add 25 points. Typing the failures is what made the fix specific. Sort the failing trials by type →

Mental model

An error taxonomy sorts failures into a small set of types, turning an overwhelming pile into a counted distribution you can rank by frequency × severity. Build it bottom-up from real traces (not imagination), keep the types distinct-complete-actionable, and let the ranking decide what you fix first and what kind of fix each type needs.

Common mistakes

  • No taxonomy — fixing case by case. Without grouping you react to the last failure you saw, not the biggest class.
  • Inventing types top-down. Imagined categories miss the real failure modes; build from traces.
  • Overlapping or misc-heavy types. Ambiguous categories make counts meaningless; a bulging "misc" bucket means the taxonomy is incomplete.
  • Types that map to no fix. If a category doesn't suggest an action, it's costing you effort for nothing.

Practical guidance

  • Read failing traces first, categorise second — let the types emerge from the data, then name and define them crisply.
  • Count each type and rank by frequency × severity; fix the top classes, not the newest case.
  • Keep the taxonomy distinct, complete (with a small honest "other"), and actionable, and revise it as new failures appear.
  • Use each type to point at a kind of fix and stage, feeding root-cause analysis.

Summary

  • An error taxonomy sorts failures into types, turning a pile into a counted, rankable backlog.
  • Build it bottom-up from real traces; imagined categories miss the real failure modes.
  • Keep types distinct, complete, and actionable — it's a taxonomy and inherits those requirements.
  • Ranking by frequency × severity decides what to fix first — this project's dominant "wrong-pattern" type is what made its guardrail targeted and effective.

Knowledge check

You've read 60 failing traces. Why is the next step to categorise them rather than start fixing the ones you found most interesting?

Because fixing by interest (or recency) attacks whichever cases caught your eye, not the failures that most affect the score. Categorising into a taxonomy converts the 60 into a distribution — e.g. 40% wrong-pattern, 25% wrong-filter, a long tail — so you can rank by frequency × severity and spend effort on the class that accounts for the most (or the most critical) failures. Grouping turns reading into a prioritised backlog; fixing ad hoc just chases individual cases.

Why build the error taxonomy from real traces instead of listing likely failure types up front?

Because a taxonomy invented in advance inherits your assumptions about how the system fails and misses the modes you didn't anticipate — the same blind-spot problem as imagined datasets. The failures that matter are often ones you wouldn't have listed. Reading real traces and letting the categories emerge captures the actual failure modes, including the surprising ones, and yields types that map to real fixes rather than imagined ones.

Related chapters