Ubuyobozi bwa tekiniki

Pass @ k Ibipimo Byasobanuwe

Pass@k is the probability that at least one of k sampled attempts from a model solves a problem, usually checked by running hidden unit tests on generated code.

Umva iki gitaboAI-byavuzwe · ~3 min

Kuriyi page4 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. The Future of Pass@k Metric Explained
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

It matters because code and agent benchmarks report it constantly, and a pass@10 or pass@100 score can look far better than what a user gets from a single try.

Kwibira cyane

Pass@k grew out of program synthesis research, where Kulal and colleagues used it in 2019 for pseudocode-to-code tasks, and it became standard with OpenAI's 2021 Codex paper, which introduced HumanEval, a set of 164 hand-written Python programming problems. For each problem the model writes a function, and the function counts as correct only if it passes hidden unit tests. Pass@k asks: if you drew k samples, what is the chance that at least one passes? The benchmark score is that probability averaged over all problems. The naive method is to generate exactly k samples per problem and check whether any pass, but that estimate is very noisy. The Codex paper instead generates n samples, where n is larger than k (for example 200), counts the c correct ones, and computes the probability that a random subset of k samples contains at least one correct sample: 1 minus C(n minus c, k) divided by C(n, k). This estimator is unbiased and far less variable. The shortcut of plugging c/n into 1 minus (1 minus p) to the power k is biased. Sampling temperature matters. Pass@1 is usually best at low temperature, while pass@100 benefits from higher temperature because more diverse samples raise the chance that one works. Papers often pick a different temperature for each k. Agent benchmarks added a complementary metric. The tau-bench paper from Sierra (2024) defined pass^k, the probability that all k independent trials succeed. It measures reliability rather than capability. Common misreadings include comparing one model's pass@10 with another's pass@1, forgetting that pass@k assumes something already knows which sample is correct, and treating greedy-decoding pass@1 as identical to sampled pass@1. Weak unit tests inflate every pass@k number, which is why EvalPlus added many more tests to HumanEval and saw scores drop.

Gushyira mu bikorwa Isi

A team evaluating a code model on HumanEval samples 200 completions per problem, counts how many pass the tests, and reports pass@1, pass@10 and pass@100 from that one set of samples using the unbiased estimator.

An editor autocomplete that shows one suggestion should be judged by pass@1, while a tool that generates five candidates and runs your test suite to pick a working one is closer to a pass@5 situation.

A customer-support agent that succeeds on 75 percent of trials looks strong on pass@k, but if trials are independent its pass^4 score (all four trials succeed) is only about 0.32, which shows how unreliable it would feel across repeated use.

A competitive programming system generates many candidate programs, filters them against the example tests, and submits a small number; its reported solve rate depends on how many submissions were allowed, which is a pass@k-style budget.

The Future of Pass@k Metric Explained

As models move from single completions to multi-step agents, reliability measures such as pass^k and run-to-run consistency are getting more attention alongside pass@k, because users experience one attempt, not the best of many. Benchmark authors are also investing in stronger test suites, since weak tests let incorrect code pass and inflate every metric. Careful reports should state k, sample count, temperature and whether any selection step was used, but practice varies, so readers should keep checking those details rather than assuming them.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Pass@k Metric Explained quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Pass@k Metric Explained?

Pass@k is the probability that at least one of k sampled attempts from a model solves a problem, usually checked by running hidden unit tests on generated code. It matters because code and agent benchmarks report it constantly, and a pass@10 or pass@100 score can look far better than what a user gets from a single try.

Niki pass @ k igipimo cyikibazo kimwe?

Pass@k is the chance that at least one of k attempts succeeds. The benchmark score averages that probability over all problems.

HumanEval, igipimo cyatangijwe nimpapuro Codex, gikubiyemo iki?

HumanEval has 164 hand-written Python problems, each checked by hidden unit tests.

Kuki impapuro Codex zitanga n ingero hamwe n nini kuruta k?

Generating exactly k samples gives a noisy estimate. Using n samples and counting c correct ones allows an unbiased estimate with much less variance.

Hamwe n ingero na c ikosora, niyihe mvugo niyerekana itabogamye pass @ k igereranya?

It is one minus the probability that a random k-subset contains only incorrect samples. C(c, k)/C(n, k) is the pass^k estimator, and the power-k shortcut is biased.

Niki pass ^ k, isobanurwa mu mpapuro za tau-intebe?

Pass^k requires every trial to succeed, so it measures reliability, which matters for agents serving many users.