Part 2 — Complete LLM Systems

Guardrails, retries, fallbacks and correction policies

Reliability and configuration·Core·7 min read

The model is unreliable; the system around it doesn't have to be. This chapter is the reliability layer — deterministic guardrails, retries, fallbacks, and correction policies that catch and handle known failures — and the discipline that keeps that layer from quietly making things worse.

A model produces the wrong thing some fraction of the time, and you can't make that fraction zero. What you can do is build a reliability layer around it — deterministic machinery that catches known failures and handles them: guardrails that check output, retries that try again, fallbacks when a step fails, and correction policies that fix predictable mistakes. This is where a system converts an unreliable model into a dependable product. It's also, as failure analysis warned, where you can quietly make things worse — so the layer needs the same discipline as anything else you measure.

What you will understand by the end

  • The four reliability mechanisms and what each handles.
  • Why every one of them is a trade, not a free improvement.
  • How this design-side view connects to the evaluation-side lesson.
  • The discipline that keeps the reliability layer a net positive.

The four mechanisms

Each addresses a different kind of known failure:

  • Guardrails — deterministic checks on input or output that block or flag bad cases: reject output that fails validation, catch a disallowed value, refuse an out-of-scope request. A gate that stops known-bad things from proceeding.
  • Retries — when a step fails (a parse error, a transient provider error, an empty result), try again, possibly with a tweaked prompt or higher effort. Handles transient and sometimes-works failures.
  • Fallbacks — when the primary path fails after retries, fall back to a safer alternative: a simpler prompt, a different model, a default answer, or a graceful "I can't do that." Ensures the system degrades gracefully instead of crashing.
  • Correction policies — deterministic rules that rewrite a known-wrong output into the right one: detect a characteristic mistake (root-caused earlier) and fix it in code. Turns a predictable model error into a reliable result.
Key idea

The reliability layer moves the system's dependability from the model (which you can't perfect) to deterministic code around it (which you can). Each mechanism handles a known failure class: guardrails block bad output, retries handle flaky steps, fallbacks degrade gracefully, correction policies fix predictable mistakes. Reliability is engineered around the model, not demanded of it.

Every mechanism is a trade

None of these is free, and treating them as pure wins is how the reliability layer backfires:

  • A guardrail that's too aggressive blocks good output; too lax, it lets bad through.
  • Retries cost latency and money, can mask a real problem you should have fixed, and — at worst — amplify load during an incident.
  • Fallbacks can hide that the primary path is broken (everything "works" via the fallback while quality quietly craters), and a silent default can be worse than an honest failure.
  • Correction policies are the sharpest trade of all: as Guardrails: fixes and regressions showed, a rule that fixes some cases regresses others, and one tuned to a model that fixed 3B by 25 points broke 7B.
Watch out

The reliability layer is itself a source of failures. A retry storm during an outage, a fallback that masks a broken primary, a correction rule that regresses cases it shouldn't touch — each is the reliability layer causing the incident. Measure the net, severity-weighted effect of every mechanism, and inspect the cases each one changes; a mechanism that isn't clearly net-positive is doing harm you can't see.

Design side meets evaluation side

This chapter and Guardrails: fixes and regressions are two views of one thing. This is the design side — how you build the reliability layer; that was the evaluation side — how you measure whether it helped. They're inseparable: you can only build these mechanisms responsibly if you measure them the way failure analysis prescribed — raw vs corrected accuracy, every changed case inspected, re-measured per model. Build and measure are the same loop.

Observed evidence

This project's correction policy is a textbook reliability mechanism and a textbook trade: it detects the ranking-vs-breakdown mistake and rewrites the intent, adding ~25 points to the 3B system it was tuned for. The design lesson is inseparable from the evaluation one — the same rule regressed the 7B model, which you only see because the system reports raw and corrected accuracy separately and every intervened case is openable. Reliability engineered and measured together. See the correction policy's fixes and regressions →

Mental model

Reliability is engineered around the model in deterministic code: guardrails block bad output, retries handle flaky steps, fallbacks degrade gracefully, correction policies fix predictable mistakes. Every one is a trade — it can block good output, mask real problems, or regress cases — so measure each mechanism's net, severity-weighted effect and inspect what it changes. Build and measure are one loop.

Common mistakes

  • Treating a mechanism as a free win. Guardrails, retries, fallbacks, and corrections all trade something; count the downside.
  • Fallbacks that hide a broken primary. If everything "works" via fallback, quality can crater invisibly; alert on fallback rate.
  • Retries that mask or amplify. They can hide a bug you should fix and worsen load during incidents; bound them and watch retry rate.
  • Shipping a correction policy on catch-count. Its value is net of regressions and is tuned to one model; measure raw vs corrected, per model.

Practical guidance

  • Build the layer from root-caused failure classes — each mechanism should target a known, characterised failure, not a vague fear.
  • Measure every mechanism's net, severity-weighted effect (raw vs corrected), inspect the cases it changes, and re-measure it per model.
  • Bound retries and alert on fallback/retry/guardrail rates — a spike is often the reliability layer masking or causing an incident.
  • Prefer an honest failure over a silent bad default; degrade gracefully and visibly.

Summary

  • The reliability layer — guardrails, retries, fallbacks, correction policies — engineers dependability into deterministic code around an imperfect model.
  • Each mechanism handles a known failure class, and each is a trade, not a free win.
  • It's the design side of the evaluation-side guardrail lesson — build and measure together (raw vs corrected, per model, inspect changed cases).
  • This project's correction policy — +25 on 3B, a regression on 7B — is the reliability layer's two-sided nature in one artifact.

Knowledge check

Your system's dashboards are all green, but users report degraded answers. You find the fallback path is serving 40% of traffic. What happened?

The fallback is masking a broken primary path. Something in the main pipeline is failing often enough that 40% of requests silently route to the safer-but-worse fallback, which "succeeds" (so the top-line success metric stays green) while actually delivering degraded answers. The reliability layer turned a loud failure into a silent quality crater. The fix: alert on fallback rate as a first-class signal, diagnose and repair the primary path, and make sure fallbacks are visible in metrics rather than quietly absorbing failures. A fallback should be a rare safety net, not a load-bearing path nobody noticed.

Why can't you evaluate a correction policy just by counting the failures it fixes?

Because a correction policy fires on a condition that never perfectly matches the failure it targets, so it also fires on some already-correct cases and regresses them. Its true value is the net — fixes minus regressions, weighted by severity — and it's tuned to a specific model's failure mode, so it can help one model and harm another (this project's rule added 25 points to 3B but regressed 7B). Counting only fixes measures half its effect; you must report raw vs corrected accuracy, inspect every case it changes, and re-measure it on each model it's applied to.

Related chapters