Part 9 — Continuous Evaluation

Release gates

The safety net·Evaluation·6 min read

A regression suite only protects you if failing it stops a release. A release gate is the automated quality bar a change must clear to ship — a threshold on the metrics that matter, agreed in advance — turning evaluation from advice into a decision that actually blocks bad changes.

A regression suite that runs on every change but never stops anything is a smoke alarm with the wires cut. The piece that gives it teeth is the release gate: a rule that says a change may ship only if it clears an agreed bar on the metrics that matter. The gate is what converts an evaluation number from something you glance at into a decision that blocks — and designing it well (which metrics, what threshold, agreed by whom, when) is what keeps a continuously-changing system from degrading one "small" merge at a time.

What you will understand by the end

  • What a release gate is and why a suite without one doesn't protect you.
  • Which metrics to gate on — and why correctness, not validity, must be among them.
  • How to set a threshold that blocks regressions without blocking all progress.
  • Why the gate must be agreed in advance, not negotiated at merge time.

A gate is a suite with authority

A release gate is a pre-committed rule: this change ships only if the evaluation clears these thresholds. It runs the regression suite (and any other required checks) automatically on the change, and a failure blocks the release rather than merely warning. That authority is the whole point — without it, evaluation is advisory, and advisory checks get overridden by whoever is in a hurry.

Key idea

A release gate turns evaluation from information into a decision: pass and ship, fail and stop. A number that can be ignored protects nothing over the long run of many changes; a gate that blocks is what actually keeps quality from eroding merge by merge.

Gate on the metrics that matter — correctness included

What you gate on determines what the gate protects. The cardinal rule from Part 4 applies with full force: a validity check ("100% schema-valid") is not a quality gate — it can pass while correctness collapses. So a real gate includes a semantic correctness threshold on the held-out set, and typically:

  • Correctness on held-out data — the primary bar, measured with a meaning-level scoring method.
  • No regression on the regression suite — previously-passing cases must still pass.
  • Guardrail metrics — things that must not get worse even if the headline improves (latency, cost, a critical segment's accuracy).
Watch out

Gating on validity or an aggregate alone is how a change ships that lifts the mean while breaking critical cases. Add guardrail metrics the change must not worsen and a regression-suite pass, so "the average went up" can't wave through a change that regressed the cases you care about most. A gate is only as good as the metrics it checks.

Set the threshold with uncertainty in mind

A gate threshold has to block genuine regressions without blocking every change over statistical noise. Two forces:

  • Set it too tight and normal sampling variation fails good changes — a 1-point dip that's within the error bars shouldn't block a release.
  • Set it too loose and real degradation slips through.

The usual shape is a no-regression bar (don't drop below the current champion by more than the noise) plus a must-not-worsen rule on guardrail metrics — informed by the confidence intervals so the gate reacts to real changes, not noise.

Observed evidence

This project frames its result as a gate, not a ranking: two systems reached 95.5% and the question was which configurations clear the bar for this task. That's a release-gate mindset — a pre-agreed threshold on held-out correctness that a configuration must pass to be the default. The open models improving from 3B to 7B but staying below the gate is exactly the gate doing its job: measured, not argued. See which systems clear the bar →

Agree it in advance

A gate negotiated at merge time isn't a gate — it's a debate the deadline usually wins. The threshold, the metrics, and who can grant an exception must be decided before the change is on the table, so the rule constrains the moment instead of bending to it. Exceptions should be rare, explicit, and logged, not a quiet override.

Mental model

A release gate is a pre-agreed rule that blocks a change unless it clears thresholds on the metrics that matter — correctness on held-out data, no regression on the suite, and guardrail metrics that mustn't worsen. It gives evaluation authority (block, not warn), reacts to real changes not noise, and is agreed in advance so the deadline can't renegotiate it.

Common mistakes

  • A suite with no gate. Checks that only warn get ignored; without blocking authority quality erodes merge by merge.
  • Gating on validity or aggregate alone. Both can pass while correctness or critical cases regress; gate on semantic correctness plus guardrail metrics.
  • A threshold set without uncertainty. Too tight blocks good changes on noise; too loose passes real regressions.
  • Negotiating the gate at merge time. A bar decided under deadline pressure isn't a bar; agree it in advance.

Practical guidance

  • Make the gate block, not warn, and run it automatically on every change.
  • Gate on held-out correctness, a regression-suite pass, and guardrail metrics that must not worsen — never validity alone.
  • Set thresholds against confidence intervals so the gate reacts to real regressions, not sampling noise.
  • Pre-agree the metrics, thresholds, and exception process; log the rare exceptions instead of quietly overriding.

Summary

  • A release gate gives the regression suite authority: a change ships only if it clears the bar.
  • Gate on correctness (held-out), no regression, and guardrail metrics — validity alone is not a quality gate.
  • Set the threshold with uncertainty in mind, and agree it in advance so deadlines can't renegotiate it.
  • This project's "which systems clear the bar" framing is a release gate in action.

Knowledge check

A team gates releases on "100% schema validity and no crashes." A change ships that passes both but drops semantic accuracy from 92% to 78%. What was wrong with the gate?

It gated on structural health (validity, no crashes) but not on semantic correctness — and those are independent axes, so a change can keep 100% valid JSON while its answers become far more often wrong. The gate needed a held-out correctness threshold (with a meaning-level scoring method) as its primary bar, plus a regression-suite pass. Validity is a syntax check; a quality gate must measure meaning, or it waves through exactly this kind of degradation.

Why must a gate threshold account for confidence intervals rather than blocking on any drop?

Because every eval number is an estimate with error bars, so a small drop (say 1 point on 88 examples) is often just sampling noise, not real degradation — blocking on it would fail good changes and grind iteration to a halt. The gate should react to changes larger than the combined uncertainty (a real regression), typically as a "don't drop below the champion by more than the noise" rule. Ignoring uncertainty makes the gate either too twitchy (blocks noise) or, if compensated by loosening, too permissive.

Related chapters