AI Glossary Term

What does “MMLU” mean?

Definition

A benchmark testing language models across 57 academic and professional subjects using multiple-choice questions.

Browse the full AI Glossary Explore all free AI guides

How MMLU works

Example

Each question has four choices, such as a college chemistry or legal question. A model picks A, B, C, or D, and its overall score is the share it answers correctly.

Why it matters

MMLU became a standard way to compare general knowledge across models. Top models now score so high that it has limited ability to separate them, and some questions contain errors.

Common questions

Who created it?

Dan Hendrycks and colleagues published it in 2020.

What does it stand for?

Massive Multitask Language Understanding.

Does a high score mean expertise?

No. It tests multiple-choice recall and reasoning, not real-world professional skill.

LLM-as-JudgeUsing a language model to score or compare outputs from other models during evaluation.GSM8KA benchmark of grade-school math word problems used to evaluate step-by-step reasoning in language models.Memory (Agent Memory)Stored context an AI agent uses across steps or sessions to improve continuity.Multimodal ModelA model that can process or generate multiple data types such as text, image, and audio.Model RegistryA central catalog for versioning, approving, and tracking models across environments.AGI (Artificial General Intelligence)A hypothetical AI system that can perform most intellectual tasks at a human level across many domains.