AI Glossary Term

What does “HumanEval” mean?

Definition

A benchmark of Python programming problems used to measure code-generation correctness via unit tests.

Browse the full AI GlossaryExplore all free AI guides

How HumanEval works

Diagram for HumanEval, steps in order: Function signature and description; then Model writes the code; then Run hidden unit tests; then Pass or fail per problem.Function signature and descriptionModel writes the codeRun hidden unit testsPass or fail per problem

Example

A problem gives a function name and a docstring such as 'return the sum of even numbers in a list'. The model writes the body, and tests check it against several inputs.

Why it matters

HumanEval offered a simple way to measure whether generated code works. Its 164 problems are small, so strong models have largely saturated it.

Common questions

Who introduced it?

OpenAI researchers in 2021, alongside the Codex model.

How many problems?

164 hand-written Python programming problems.

Limits?

The problems are short and well known, so it says little about real software projects.

Related terms

BenchmarkA standardized test or dataset used to measure and compare model performance.GSM8KA benchmark of grade-school math word problems used to evaluate step-by-step reasoning in language models.Evaluation SetA held-out dataset used to measure model quality after training.Test-Time ComputeAdditional inference computation used during response generation to improve quality or reasoning.Pass@kA code-evaluation metric measuring the chance that at least one of k generated samples passes the tests.AI GovernancePolicies, standards, and oversight mechanisms that guide how AI is developed and used in society.

See also

MMLUFactualityLLM-as-JudgeCitationsBi-EncoderWatermarkingCross-EncoderMid-training