Offline, online and human evaluation

The same measurement can be taken in three very different settings, and each answers a different question. Offline evaluation runs a fixed dataset in a lab; online evaluation watches real traffic in production; human evaluation asks people to judge quality directly. They are not competitors — they are stages of a loop, and knowing which one a number came from is as important as the number itself.

What you will understand by the end

• What offline, online, and human evaluation each measure.
• The core trade each makes: reproducibility vs realism vs judgment.
• When to reach for each, and why they form a loop rather than a ranking.
• Where this project sits, and where the others appear later in the book.

The three modes

• Offline — run your fixed system against a fixed dataset with known answers, score automatically. Fast, cheap, perfectly reproducible, and safe to run before anything ships. Its weakness: the dataset is a model of reality, so it only reflects production to the extent it was built to.
• Online — measure the system on real traffic in production: A/B tests, and signals like task success, user edits, thumbs, retries, or downstream conversions. Maximally realistic, because it is literally what users experience. Its weakness: slow, risky (bad variants hit real users), confounded by everything else changing, and often lacking a clean "correct" label.
• Human — put a person in the loop to judge outputs a rule can't score: is this summary faithful, is this answer helpful, is this tone right. The gold standard for subjective quality. Its weakness: expensive, slow, and variable — humans disagree, so you need guidelines and multiple raters.

   OFFLINE            ONLINE               HUMAN
   fixed dataset      real traffic          people judge
   reproducible       realistic             judgment
   pre-deploy gate    production truth       subjective quality
   cheap & fast       slow & risky          expensive & slow

They form a loop, not a ranking

The modes are not rivals to rank "most real to least real." They chain:

   offline (gate before deploy) ─▶ online (measure on real traffic) ─▶ human (judge the hard cases)
        ▲                                                                        │
        └──────────── production failures become new offline test cases ─────────┘

You gate releases offline (cheap, reproducible, no user risk), confirm the win online (real behaviour), use humans to judge what rules can't and to adjudicate disagreements — and then feed the failures you find back into the offline set so the gate gets stronger. Turning production incidents into offline test cases is one of the highest-leverage habits in the field (revisited in continuous evaluation).

Where this book sits

This project's evaluation is offline: six fixed systems on a fixed 88-task dataset, scored by a deterministic rule, run before any deployment. That is deliberate — offline is where you compare configurations reproducibly and gate a decision. Online evaluation (A/B tests, production signals) is the subject of Part 9, and human judgment — including the LLM-as-judge stand-in for it — appears in continuous evaluation. Recognising which mode a number came from is the habit; the rest of the book fills in each.

Mental model

Offline = reproducible lab measurement on a fixed dataset (gate here). Online = realistic measurement on live traffic (confirm here). Human = judgment for what rules can't score (adjudicate here). They loop: gate offline, confirm online, judge with humans, feed failures back offline.

Common mistakes

• Arguing which mode is "real." They measure different things; the answer is "use all three at the right moment."
• Shipping on an offline number alone. A lab win must be confirmed on real traffic; the dataset is a proxy.
• Using humans for what a rule can score. Reserve expensive human judgment for genuinely subjective calls, not exact-match checks.
• Never closing the loop. Production failures that don't become offline test cases will recur; the loop is what makes the gate improve.

Practical guidance

• Gate releases offline — it's the only mode cheap and reproducible enough to run on every change without user risk.
• Confirm wins online with an A/B test and pre-chosen success signals before you trust them.
• Spend human evaluation on subjective quality and on adjudicating disagreements; write rater guidelines and use more than one rater.
• Always label a number with its mode — "94% offline on the held-out set" says far more than "94%."

Summary

• Offline (fixed dataset, reproducible, pre-deploy), online (real traffic, realistic, risky), and human (judgment, gold for subjective, costly) each measure something different.
• The trade is reproducibility vs realism vs judgment; no mode has all three.
• They form a loop: gate offline, confirm online, judge with humans, feed failures back.
• This project is offline; online and human evaluation arrive in Parts 8–9.

Knowledge check

A change gains 4 points offline but the online A/B shows no improvement in task success. Give the most likely explanation and what to do.

The offline dataset probably isn't fully representative of production — it rewarded something that doesn't matter to real users, or missed the cases where they differ. That gap is information: investigate which real inputs the offline set under-represents and add them (turn the discrepancy into new offline cases), so the offline gate better predicts online behaviour next time. Don't just declare the online result wrong.

Why not evaluate everything with human raters, since human judgment is the "gold standard"?

Because it's expensive, slow, and variable — humans disagree, so you need guidelines and multiple raters, and it doesn't scale to every change. For anything a deterministic rule can score (exact match, does-it-parse, does-it-execute) a rule is faster, cheaper, and perfectly reproducible. Reserve human judgment for genuinely subjective quality that rules can't capture.

Related chapters

• What an evaluation measures — the measurement these modes are three settings of
• Online experiments — online evaluation in depth
• LLM-as-judge — scaling human-style judgment with a model
• Turning incidents into test cases — closing the loop back to offline