AI Glossary Term

What does “Benchmark Leakage” mean?

Definition

When benchmark test examples or close variants are present in training data, inflating reported performance.

Browse the full AI Glossary Explore all free AI guides

How Benchmark Leakage works

Example

A coding benchmark's problems and solutions are posted on public sites. A model trained on a web scrape may have seen them, so a high score may reflect memorization rather than ability.

Why it matters

Leakage makes comparisons unreliable. Fresh, private, or regularly updated tests help check whether a score is real.

Common questions

How is it detected?

By searching training data for test items, or checking whether a model performs worse on newly written, similar questions.

Is it always deliberate?

No. It often happens by accident through large web scrapes.

How can it be avoided?

Keep test sets private, filter training data against known benchmarks, and use new test items regularly.

BenchmarkA standardized test or dataset used to measure and compare model performance.Data AugmentationTechniques that create modified training examples to improve model generalization.OverfittingWhen a model memorizes training data and performs poorly on unseen inputs.Algorithmic TransparencyHow clearly an AI system's logic, data sources, and limitations are documented and understandable.Continual LearningTraining approaches that let a model keep learning from new data without forgetting prior knowledge.Data DriftA shift in real-world input data over time that can degrade model performance.

Learn more in our free guides

AI Benchmarks Benchmark Contamination SWE-bench and Coding Agent Benchmarks IFEval and Instruction-Following Benchmarks

Test yourself

AI Benchmarks Quiz Benchmark Contamination Quiz SWE-bench and Coding Agent Benchmarks Quiz