Guardrails: fixes and regressions

Once failure analysis has named a failure type, the tempting fix is a guardrail — a deterministic correction policy that catches that failure and rewrites it. Guardrails are powerful; the ranking-vs-breakdown rule in this project added 25 points. But every guardrail is a two-sided trade: the same rule that fixes the cases you targeted can break cases you didn't, and a guardrail measured only by what it catches hides what it breaks. This chapter is about evaluating guardrails honestly — by their net effect and the full set of cases they touch.

What you will understand by the end

• What a guardrail is and why it's an attractive, high-leverage fix.
• Why every guardrail risks regressions — breaking cases it shouldn't touch.
• Why you measure the net effect and inspect every case it changed, not just its catches.
• Why a guardrail is tuned to one system and doesn't transfer for free.

A guardrail catches a known failure

A guardrail is a deterministic rule bolted onto the pipeline that detects a specific, characterised failure and corrects it — downgrade a top_n prediction to a breakdown when the question has no ranking language, drop an out-of-range value, retry on a parse error. Because failure analysis gave you the failure's exact signature, the rule can be precise, and precise rules can move the number a lot for little cost. That's the appeal.

Every guardrail can regress

Here's the catch. A guardrail fires on a condition, and no condition perfectly matches the failure it targets. It will sometimes fire on cases that were already correct and break them — a regression. So a guardrail has two effects that must both be counted:

• Fixes — previously-failing cases the rule now gets right.
• Regressions — previously-passing cases the rule now gets wrong.

The rule is only worth it if fixes outweigh regressions, and even a net-positive rule may be unacceptable if its regressions land on critical cases. A guardrail evaluated only by its catches — "it fixed 30 cases!" — is measured on half its effect.

The clearest possible warning: it doesn't transfer

The sharpest evidence that a guardrail is a two-sided trade is what happens when you move it to a different model. A rule tuned to one system's failure mode encodes assumptions about that system, so on another it can fire in the wrong places and do net harm.

Mental model

A guardrail deterministically corrects a known failure — high leverage, but two-sided. It fires on a condition that never perfectly matches, so it fixes some cases and regresses others; its worth is the net, weighted by severity, and it's tuned to one system so it doesn't transfer. Measure raw vs corrected, inspect every case it changed, and re-measure per system.

Common mistakes

• Counting fixes, ignoring regressions. A guardrail's value is net; "it fixed 30" is half the story.
• Transferring a guardrail blindly. Tuned to one system's failures, it can net-harm another — as the 3B rule did to 7B.
• Not looking at what it broke. A net-positive rule can still regress critical cases; inspect the changed trials.
• Treating the guardrail as free. It adds complexity and its own failure mode; it's part of the system, versioned with it.

Practical guidance

• Always report raw and corrected accuracy; the gap — positive or negative — is the guardrail's real effect.
• Inspect every intervened case for fixes and regressions, and weight regressions by severity, not just count.
• Re-tune and re-measure the guardrail on each system it's applied to; don't port it unchanged.
• Add a guardrail only when its net, severity-weighted effect is clearly positive, and keep it versioned as part of that system.

Summary

• A guardrail deterministically corrects a known failure — high leverage, but a two-sided trade: fixes and regressions.
• Its worth is the net effect, weighted by severity; counting only catches measures half of it.
• It's tuned to one system and doesn't transfer — this project's 3B guardrail (+25) regressed the 7B model.
• Measure raw vs corrected, inspect every changed case, and re-measure per system.

Knowledge check

A guardrail "fixes 40 previously-failing cases." Your colleague wants to ship it. What two things must you check before agreeing?

(1) The regressions: how many previously-passing cases does the rule now break, and how severe are they? The net (40 fixes minus regressions, severity-weighted) is what matters — a rule that also breaks 35, or breaks 3 critical cases, may be net-negative. (2) That it was measured on this exact system: a guardrail tuned to another model can fire wrongly here. Confirm net-positive, severity-weighted, on the actual target system, by inspecting the changed cases — not just the catch count.

Why did the same guardrail add 25 points to the 3B model but regress the 7B model?

Because the guardrail was tuned to the 3B model's specific failure mode — cases where 3B chose ranking instead of breakdown. The stronger 7B model already handled many of those cases correctly, so when the rule fired on them it "corrected" already-right outputs into wrong ones — a regression. A guardrail encodes assumptions about one system's failures; move it to a system that fails differently and its firing condition lands in the wrong places. It's part of a system, not a portable upgrade.

Related chapters

• The model is only one component — raw vs corrected, the gap a guardrail creates
• Aggregate metrics versus individual failures — why a guardrail's mean can hide its regressions
• Guardrails, retries, fallbacks and correction policies — designing the policies themselves
• Regression suites — catching guardrail regressions automatically