Part 4 — Evaluation Data and Dataset Design

Holdout contamination

Splits and contamination·Evaluation·6 min read

A hidden set is only honest until something about it leaks into the system. This chapter is about the many quiet ways that happens — repeated peeking, examples sneaking into training data, benchmark leakage into the base model — and how each silently inflates a number nobody realises is wrong.

The previous chapter said: never tune on the hidden set. This one is about all the ways the hidden set gets tuned on without anyone deciding to — a slow leak rather than an act. Contamination is dangerous precisely because it's invisible: the number still looks honest, the process still looks disciplined, and the score is quietly, unmeasurably too high. Knowing the leak paths is the only defence.

What you will understand by the end

  • What "contamination" means and why it's usually accidental, not cheating.
  • The main leak paths: repeated peeking, training-data leakage, and benchmark leakage into the base model.
  • Why contamination inflates scores invisibly — the number looks fine.
  • How to detect it and keep hidden sets honest over time.

Contamination is a leak, not a crime

Contamination is any way the hidden set influences the system it's supposed to judge — usually without a decision to do so. Three common paths:

  • Repeated peeking. You run the hidden set, tweak, run it again, tweak again. No single step is "tuning on the test set," but across dozens of iterations you've slowly fit the system to that specific sample. The set decays into a development set one glance at a time.
  • Training-data leakage. A held-out example (or a near-duplicate) ends up in the training or fine-tuning data. Now the system has literally seen the answer; scoring it measures memorisation. This is easy to do with scraped or overlapping datasets.
  • Benchmark leakage into the base model. For public benchmarks, the test questions are often already in the model's pre-training data (the web was scraped). The model "knows" the answers for reasons that won't generalise — which is why a stellar public-benchmark score can evaporate on your private data.
Key idea

Contamination doesn't announce itself. A contaminated hidden set produces a number that looks exactly as honest as a clean one — same process, same dashboard — but is inflated by an amount you can't see. That invisibility is what makes it the most dangerous failure in evaluation: you trust the number precisely because nothing looks wrong.

Why it's so corrosive

Every other evaluation error announces itself eventually — a fuzzy contract causes visible disagreement, a bad scoring method fails obvious cases. Contamination doesn't. The gap only appears in production, long after the decision was made on an inflated number, and even then it's hard to attribute. It quietly breaks the one thing the hidden set exists to provide: an honest estimate of unseen performance.

Watch out

A model that aces a public benchmark tells you little about your task, partly because those questions may already be in its training data. Treat public-benchmark scores as a weak prior for choosing a base model, and get your honest number from a private hidden set the model has never seen. A model can't have memorised data you created and kept off the internet.

Detecting and preventing it

  • Keep hidden sets private and fresh. Data you author and never publish can't be in a model's training set. If a hidden set has been peeked at many times, retire and refresh it — treat repeated exposure as gradual contamination.
  • De-duplicate across splits and against training data. Check that no held-out example (or near-duplicate) appears in development or in any fine-tuning set.
  • Watch for the tell. A suspiciously large gap between public-benchmark performance and your private-set performance is a classic contamination signature.
  • Budget your peeks. Decide in advance how many times the hidden set will be run, and stick to it; every extra look spends some of its honesty.
Observed evidence

This project's held-out (60) and adversarial (8) tasks are hand-built for this specific schema and domain, not drawn from any public benchmark — so the base models cannot have memorised them, and the 95.5% is a measurement of the task, not of leaked test data. Authoring your own private evaluation set is the simplest, strongest guard against contamination. See the private, hand-built splits →

Mental model

Contamination is the hidden set leaking into the system — by repeated peeking, by examples in training data, or by benchmark questions already in the base model. It inflates the score invisibly, so the number looks honest while being wrong. Private, fresh, de-duplicated, peek-budgeted hidden sets are the defence.

Common mistakes

  • Peeking at the hidden set repeatedly. Slow tuning that decays it into a development set.
  • Trusting public-benchmark scores for your task. The questions may be in the model's training data; get your number from private data.
  • Not de-duplicating splits. A held-out example that also sits in training or dev silently measures memorisation.
  • Never refreshing an over-used hidden set. After enough exposure it's contaminated; retire and rebuild it.

Practical guidance

  • Author your hidden set privately and keep it off the internet — the strongest guard against training-data and benchmark leakage.
  • De-duplicate aggressively across dev/validation/hidden and against any training set, including near-duplicates.
  • Budget peeks at the hidden set and refresh it once it's been over-exposed.
  • Read a big public-vs-private gap as a contamination warning, not a mystery.

Summary

  • Contamination is any leak of the hidden set into the system — repeated peeking, training-data leakage, or benchmark leakage into the base model.
  • It inflates scores invisibly: the number looks honest while being wrong, and the gap only shows in production.
  • Defend with private, fresh, de-duplicated, peek-budgeted hidden sets.
  • This project's hand-built private splits are its guard — the base models can't have seen them.

Knowledge check

A vendor's model tops every public benchmark, but on your private hidden set it lands mid-pack. Name the most likely benign explanation and the concerning one.

Benign: your task differs from the benchmarks, so general capability doesn't transfer — normal and expected. Concerning: benchmark contamination — those public questions were in the model's training data, so its high public scores partly reflect memorisation that doesn't generalise, while your private set (which it never saw) reveals true task performance. Either way, your private number is the trustworthy one; the public gap is exactly why you keep a private hidden set.

Why does running your hidden set 50 times over a project slowly ruin it, even if you never explicitly "tune on it"?

Because each run lets its results influence your next decision — which tweak to keep, which config to try — so across 50 iterations the system gradually fits the quirks of that specific sample, exactly like overfitting a development set, just slower and less visibly. The set's honesty erodes with every peek. Budget the number of runs and refresh the set once it's been heavily exposed.

Related chapters