AI Glossary Term
What does “Pass@k” mean?
Definition
A code-evaluation metric measuring the chance that at least one of k generated samples passes the tests.
How Pass@k works
Example
For each programming problem a model writes 10 attempts. If at least one passes all tests, the problem counts as solved for pass@10. Pass@1 asks whether a single attempt passes.
Why it matters
It reflects how code assistants are used: people can try again. It also rewards generating many varied attempts, so the value of k is important context.
Common questions
Pass@1 vs pass@10?
Pass@1 requires the first try to work; pass@10 allows ten tries, so it is usually higher.
How is it estimated?
By generating more than k samples and using an unbiased estimator, as described in the 2021 Codex paper.
What does it not measure?
Code quality, readability, or security beyond what the tests check.