Part 4 — Evaluation Data and Dataset Design

Sample size and uncertainty

Coverage and uncertainty·Evaluation·7 min read

A number from a finite dataset is an estimate, not a fact — it comes with error bars whose width depends on how many examples you measured. This chapter is about reading accuracy as a range, knowing when a difference between two systems is real, and resisting the urge to over-interpret a few examples.

Every accuracy you report is computed from a finite sample, so it is an estimate with uncertainty attached — a range, not a point. Ignore that and you'll chase differences that are just noise, declare winners that aren't, and trust small numbers you shouldn't. This closing chapter of Part 5 is about reading a score with its error bars on, so your comparisons survive contact with statistics.

What you will understand by the end

  • Why a measured accuracy is an estimate with a confidence interval.
  • How sample size sets the width of the error bars.
  • When a difference between two systems is real versus within-noise.
  • How to size a dataset for the decision you need to make.

An accuracy is a range, not a point

Measure a system on 50 examples and get 90%, and the honest statement isn't "the system is 90% accurate" — it's "our best estimate is 90%, and the true value is plausibly somewhere in a band around it." That band is the confidence interval, and it exists because a different 50 examples would have given a slightly different number. The score you report is one draw from a distribution.

Key idea

A measured accuracy is an estimate, and every estimate carries a confidence interval. The right mental model is "90% ± a few points," not "90%." Reporting a bare point number hides the uncertainty and invites everyone to over-read it — especially small gaps that fall entirely inside the band.

Sample size sets the error bars

The width of the interval shrinks as the sample grows — roughly with the square root of the number of examples, so quadrupling the dataset halves the error bars. The practical consequences:

  • Small sets have wide bands. On a handful of examples, the interval can be ±15 points or more — wide enough that two very different-looking scores are statistically indistinguishable.
  • Diminishing returns. Because of the square-root relationship, going from 25 to 100 examples helps a lot; going from 1,000 to 4,000 helps much less. There's a point where more data barely narrows the band.
  • Per-slice bands are wider. A 20% slice of a 100-example set is 20 examples; its error bars are far wider than the overall number's — so per-segment claims need more caution.
   n = 20     ├────────── 90% ──────────┤     wide band: ±~13 pts
   n = 100    ├──── 90% ────┤                 ±~6 pts
   n = 500    ├─ 90% ─┤                        ±~3 pts
              (bands shrink ~ 1/√n — quadruple n to halve the band)

Is the difference real?

The most common statistical error in LLM evaluation is treating a small gap as a result. If system A scores 91% and B scores 89% on 88 examples, that 2-point gap is almost certainly inside the noise — a different sample could easily flip it. Before you act on "A beats B," check whether the gap is larger than the combined uncertainty of the two estimates. If it isn't, you haven't found a winner; you've found two systems you can't yet distinguish.

Watch out

"System A (95.5%) edged out System B (94.3%)" on a small set is usually a story about sampling luck, not quality. On ~88 examples a one-or-two-point gap sits well within the error bars, so ranking by it is ranking noise. Either gather more data to shrink the bands, or report the two as a tie — and never build a decision on a difference smaller than its uncertainty.

Observed evidence

This project scores on 88 tasks, and two systems tied at 95.5%. That tie is the honest reading: on 88 examples the confidence band is several points wide, so systems within a few points of each other are statistically indistinguishable — calling one "best" over the other would be reading sampling noise as signal. The right conclusion is "these two lead; the rest trail," not a precise ranking of the leaders. See the leaders and the spread →

Size the dataset for the decision

How many examples you need depends on the difference you must detect. To tell 90% from 70% (a 20-point gap), a small set suffices — the bands don't overlap. To tell 95% from 93% (a 2-point gap), you need a much larger set to shrink the bands below the gap. Decide the smallest difference that would change your decision, then size the dataset so its error bars are narrower than that.

Mental model

A score is an estimate with a confidence interval whose width shrinks like 1/√n. Read accuracy as "90% ± a few points," treat gaps smaller than the combined uncertainty as ties, and size the dataset so its error bars are narrower than the smallest difference you need to act on.

Common mistakes

  • Reporting a point number. "90%" hides the band; "90% ± 6" tells the truth and stops over-reading.
  • Ranking by within-noise gaps. A 1–2 point gap on a small set is sampling luck, not a result.
  • Trusting tiny slices. A 20-example segment has wide bands; per-slice claims need far more caution than the overall number.
  • Adding data past the point of return. Beyond a certain size the bands barely move; spend the effort on coverage instead.

Practical guidance

  • Report accuracy with a confidence interval, and compare systems by whether their intervals overlap, not by raw point gaps.
  • Treat differences smaller than the combined uncertainty as ties — gather more data or accept the tie, don't rank noise.
  • Size the dataset to the smallest decision-relevant difference: big gaps need few examples, small gaps need many.
  • Be extra cautious with per-slice numbers; their bands are much wider than the aggregate's.

Summary

  • A measured accuracy is an estimate with a confidence interval; read it as a range, not a point.
  • The band shrinks like 1/√n — small sets have wide bands, and there are diminishing returns.
  • A difference smaller than the combined uncertainty is a tie; ranking by it is ranking noise.
  • Size the dataset to the smallest difference you must detect — this project's two systems "tied at 95.5%" is exactly that lesson on 88 tasks.

Knowledge check

On a 60-example held-out set, config A scores 88% and config B scores 85%. A teammate wants to ship A. What's the statistical caution?

On 60 examples the confidence band is roughly ±8–9 points, so a 3-point gap sits well inside the combined uncertainty of the two estimates — a different 60 examples could easily reverse it. That's a tie, not a win for A. Either gather substantially more data to shrink the bands below the gap, or choose between A and B on other grounds (latency, cost, simplicity); don't ship A on a difference smaller than its own error bars.

You need to detect a 2-point accuracy difference reliably. Why won't 100 examples do, and what's the fix?

On 100 examples the confidence band is roughly ±5–6 points — far wider than the 2-point difference you're trying to detect — so the two systems' intervals overlap heavily and the gap is indistinguishable from noise. Because the band shrinks like 1/√n, detecting a 2-point gap needs many more examples (thousands, not hundreds) to push the error bars below 2 points. The fix is to size the dataset to the difference you must detect — or accept that at 100 examples you can only resolve large gaps.

Related chapters