Pass@k Metric Explained

Pass@k grew out of program synthesis research, where Kulal and colleagues used it in 2019 for pseudocode-to-code tasks, and it became standard with OpenAI's 2021 Codex paper, which introduced HumanEval, a set of 164 hand-written Python programming problems. For each problem the model writes a function, and the function counts as correct only if it passes hidden unit tests. Pass@k asks: if you drew k samples, what is the chance that at least one passes? The benchmark score is that probability averaged over all problems. The naive method is to generate exactly k samples per problem and check whether any pass, but that estimate is very noisy. The Codex paper instead generates n samples, where n is larger than k (for example 200), counts the c correct ones, and computes the probability that a random subset of k samples contains at least one correct sample: 1 minus C(n minus c, k) divided by C(n, k). This estimator is unbiased and far less variable. The shortcut of plugging c/n into 1 minus (1 minus p) to the power k is biased. Sampling temperature matters. Pass@1 is usually best at low temperature, while pass@100 benefits from higher temperature because more diverse samples raise the chance that one works. Papers often pick a different temperature for each k. Agent benchmarks added a complementary metric. The tau-bench paper from Sierra (2024) defined pass^k, the probability that all k independent trials succeed. It measures reliability rather than capability. Common misreadings include comparing one model's pass@10 with another's pass@1, forgetting that pass@k assumes something already knows which sample is correct, and treating greedy-decoding pass@1 as identical to sampled pass@1. Weak unit tests inflate every pass@k number, which is why EvalPlus added many more tests to HumanEval and saw scores drop.