Pass@k Metric Explained Quiz

Answers and explanations appear as you take the quiz.

1. What does pass@k measure for a single problem?
   • The fraction of k samples that pass the tests
   • The probability that at least one of k sampled attempts passes the tests
   • The probability that the first attempt passes after k retries of fine-tuning
   • The number of tests passed out of k tests
2. HumanEval, the benchmark introduced with the Codex paper, contains what?
   • 2,294 GitHub issues from Python projects
   • 1,000 competitive programming problems in C++
   • 164 hand-written Python programming problems with unit tests
   • 500 human-verified bug reports
3. Why does the Codex paper generate n samples with n larger than k?
   • To compute an unbiased, lower-variance estimate of pass@k
   • To let the model learn from its failures between samples
   • Because the tests require at least 200 inputs
   • To make the model's outputs deterministic
4. With n samples and c correct, which expression is the unbiased pass@k estimator?
   • c divided by n
   • 1 minus (1 minus c/n) to the power k
   • C(c, k) divided by C(n, k)
   • 1 minus C(n minus c, k) divided by C(n, k)
5. What does pass^k, defined in the tau-bench paper, measure?
   • The chance that at least one of k trials succeeds
   • The chance that all k independent trials succeed
   • The average number of steps across k trials
   • The score after k rounds of self-correction
6. How does sampling temperature typically interact with k?
   • Temperature has no effect on pass@k
   • Low temperature is best for every k
   • Higher temperature tends to help pass@k at large k because samples are more diverse
   • Higher temperature always improves pass@1
7. Why can a high pass@100 score overstate real-world usefulness?
   • It assumes something already knows which of the samples is correct
   • It only counts the first sample
   • It ignores the unit tests
   • It is always computed with greedy decoding
8. If n = 20 samples include c = 18 correct ones, what is pass@5?
   • 0.9
   • About 0.5
   • It cannot be computed because n is too small
   • Exactly 1