Part 9 — Continuous Evaluation

Champion–challenger testing

Comparing and evolving·Evaluation·6 min read

How a production system evolves without lurching: keep a reigning champion configuration, test each new challenger against it on the same evaluation, and promote only when the challenger beats the champion by more than the noise. Disciplined, reversible, and honest about when there's no real winner.

A production system needs to keep improving and stay stable — two goals that fight unless you have a discipline for changing the thing users depend on. Champion–challenger is that discipline: there is always a reigning champion (the configuration currently in production), and any proposed change is a challenger that must beat the champion on the same evaluation before it takes the throne. It's how you evolve a live system deliberately — promoting real improvements, rejecting ties and regressions, and always keeping a known-good fallback.

What you will understand by the end

  • What champion and challenger mean, and why the framing keeps a live system stable.
  • The promotion rule: a challenger must beat the champion beyond the noise.
  • Why "no significant difference" means keep the champion.
  • How this combines controlled comparison, gates, and reversibility.

Always a champion, always a challenger

The model is simple and powerful: production is always running one blessed configuration, the champion. Improvement happens by proposing a challenger — a new prompt, model, or policy — and running it head-to-head against the champion. The champion doesn't change until a challenger earns the promotion by winning the comparison. This gives you a system that can evolve continuously while never being in an unvetted state — there's always a current best, and a known-good thing to fall back to.

Key idea

Champion–challenger makes improvement monotonic and reversible: you only ever replace the champion with something that measurably beat it, and you always keep the previous champion as a fallback. The system moves forward in vetted steps instead of lurching between whatever was merged last.

The promotion rule

A challenger is promoted only if it beats the champion by more than the noise on the shared evaluation. This folds together the last two chapters:

  • It's a controlled comparison — champion and challenger differ by the change under test and are scored on the same current evaluation.
  • The win must clear the uncertainty — a within-error-bar lead is not a win.
  • The challenger must also pass the release gate: no regression on the suite, guardrail metrics not worse.

Only when all of that holds does the challenger become the new champion.

Watch out

When the challenger's advantage is within the noise, the right move is to keep the champion — not to promote the shiny new thing "because it's at least as good." A tie is not a reason to change the system users depend on: promoting on noise churns production for no measured gain and forfeits the stability the champion represents. Default to the incumbent unless the challenger truly wins.

Offline first, then online

The comparison usually runs in two stages. Offline champion-challenger on the held-out set is cheap and reproducible — it filters out challengers that don't clear the bar without any production risk. A challenger that wins offline is then confirmed with an online experiment on real traffic before it's fully promoted, because an offline win still has to hold up in production. Offline decides which challengers are worth an online test; online decides whether the win is real for users.

Observed evidence

This project's six-system comparison is champion-challenger in a snapshot: configurations are measured head-to-head on the same 88-task evaluation to decide which should be the default. The result — two systems at 95.5%, the rest below — is exactly a promotion decision: a challenger only earns the default slot by clearing the incumbent beyond the noise, and the two tied leaders show that "no significant difference" means you choose on other grounds, not by flipping a coin. See the head-to-head that decides the default →

Mental model

Production always runs a champion; every change is a challenger that must beat it on the same evaluation, beyond the noise, and pass the gate, to be promoted. Ties keep the champion. Filter challengers offline, confirm winners online, and always keep the old champion as a fallback — improvement in vetted, reversible steps.

Common mistakes

  • Promoting on a within-noise lead. A tie isn't a win; churning production for an unmeasured gain forfeits stability.
  • No fallback. Replacing the champion without keeping the previous one removes your safe revert.
  • Skipping the offline filter. Testing every challenger online is slow and risky; filter cheaply offline first.
  • Promoting on offline alone. An offline win must be confirmed on real traffic before full rollout.

Practical guidance

  • Keep a named champion in production and treat every change as a challenger that must earn promotion.
  • Promote only on a win that exceeds the noise and passes the gate; on a tie, keep the champion.
  • Filter challengers offline, then confirm winners online before full rollout.
  • Always retain the previous champion as an instant fallback for a fast, safe revert.

Summary

  • Champion–challenger keeps a live system stable while it improves: production always runs a vetted champion.
  • A challenger is promoted only if it beats the champion beyond the noise and passes the gate; ties keep the champion.
  • Filter challengers offline, confirm winners online, and keep the old champion as a fallback.
  • This project's head-to-head-for-the-default framing is champion-challenger in a snapshot.

Knowledge check

A challenger scores 90.4% against the champion's 90.1% on the held-out set, clearly passing the gate. Promote it?

No — a 0.3-point lead is far inside the confidence band, so it's a tie, not a win. The champion-challenger rule is to promote only when the challenger beats the champion by more than the noise; a within-error-bar difference is no measured improvement, and swapping the production configuration for it churns the system users depend on for zero real gain while giving up the incumbent's proven stability. Keep the champion; if you want the challenger's other benefits (cost, simplicity), decide on those grounds, not the noise-level accuracy gap.

Why run champion-challenger offline first instead of just A/B testing every challenger in production?

Because online testing is slow, risky (bad challengers hit real users), and confounded, so running every proposed change through it doesn't scale and exposes users to unvetted configs. The offline held-out comparison is cheap, reproducible, and safe — it cheaply filters out the challengers that don't even clear the bar, so only the ones that win offline earn an online confirmation. Offline decides which challengers are worth a production test; online decides whether the win holds for users.

Related chapters