Gids-gekoppelde quiz · Middelmatig Niveau

Pass@k Metriek uitgelegd quiz

Test de pass@k-definitie, de zuivere schatter uit n steekproeven, de pass^k-betrouwbaarheid en veelvoorkomende manieren waarop pass@k-scores verkeerd worden geïnterpreteerd of opgeblazen.

Gerelateerde gidspadenPass bij K-metrisch
Vraag 1 van 8

Wat meet pass@k voor een enkel probleem?

Bekijk elke vraag8 vragen

Antwoorden en uitleg verschijnen terwijl je de quiz doet.

  1. Wat meet pass@k voor een enkel probleem?
    • The fraction of k samples that pass the tests
    • The probability that at least one of k sampled attempts passes the tests
    • The probability that the first attempt passes after k retries of fine-tuning
    • The number of tests passed out of k tests
  2. Wat bevat HumanEval, de benchmark geïntroduceerd met het Codex-papier?
    • 2,294 GitHub issues from Python projects
    • 1,000 competitive programming problems in C++
    • 164 hand-written Python programming problems with unit tests
    • 500 human-verified bug reports
  3. Waarom genereert het Codex-papier n monsters met n groter dan k?
    • To compute an unbiased, lower-variance estimate of pass@k
    • To let the model learn from its failures between samples
    • Because the tests require at least 200 inputs
    • To make the model's outputs deterministic
  4. Met n monsters en c correct, welke uitdrukking is de zuivere pass@k-schatter?
    • c divided by n
    • 1 minus (1 minus c/n) to the power k
    • C(c, k) divided by C(n, k)
    • 1 minus C(n minus c, k) divided by C(n, k)
  5. Wat meet pass^k, gedefinieerd in het tau-bench-papier?
    • The chance that at least one of k trials succeeds
    • The chance that all k independent trials succeed
    • The average number of steps across k trials
    • The score after k rounds of self-correction
  6. How does sampling temperature typically interact with k?
    • Temperature has no effect on pass@k
    • Low temperature is best for every k
    • Higher temperature tends to help pass@k at large k because samples are more diverse
    • Higher temperature always improves pass@1
  7. Why can a high pass@100 score overstate real-world usefulness?
    • It assumes something already knows which of the samples is correct
    • It only counts the first sample
    • It ignores the unit tests
    • It is always computed with greedy decoding
  8. If n = 20 samples include c = 18 correct ones, what is pass@5?
    • 0,9
    • About 0.5
    • It cannot be computed because n is too small
    • Precies 1