AI Glossary Term
What does “Benchmark” mean?
A standardized test or dataset used to measure and compare model performance.
How Benchmark works
Example
A benchmark like MMLU gives many models the same multiple-choice questions across school and professional subjects. Because the questions and scoring are fixed, researchers can compare one model against another or a model against its own earlier version.
Why it matters
Benchmarks make progress measurable, but a high score only reflects the specific test. Scores can be inflated by leaked test data or by tuning to the benchmark.
Common questions
Does a high benchmark score mean a model is better for my task?
Not necessarily. Benchmarks measure specific skills, and your task may differ. Testing on your own examples is often more informative.
Why do benchmarks become outdated?
Models improve until scores saturate near the maximum, and test questions may leak into training data, so new harder benchmarks are created.
Who creates benchmarks?
Academic groups, companies, and open communities. Each may have its own goals, so it helps to check how a benchmark was built.