Quiz collegato alla guida · Medio Livello

Quiz sulla spiegazione delle metriche Pass@k

Verifica la definizione di pass@k, lo stimatore imparziale da n campioni, l'affidabilità di pass^k e i modi comuni in cui i punteggi pass@k vengono interpretati erroneamente o gonfiati.

Percorsi guida correlatiPassa alla metrica K
Domanda 1 di 8

Cosa misura pass@k per un singolo problema?

Anteprima di ogni domanda8 domande

Le risposte e le spiegazioni vengono visualizzate mentre rispondi al quiz.

  1. Cosa misura pass@k per un singolo problema?
    • The fraction of k samples that pass the tests
    • The probability that at least one of k sampled attempts passes the tests
    • The probability that the first attempt passes after k retries of fine-tuning
    • The number of tests passed out of k tests
  2. HumanEval, il benchmark introdotto con il documento Codex, cosa contiene?
    • 2,294 GitHub issues from Python projects
    • 1,000 competitive programming problems in C++
    • 164 hand-written Python programming problems with unit tests
    • 500 human-verified bug reports
  3. Perché la carta Codex genera n campioni con n maggiore di k?
    • To compute an unbiased, lower-variance estimate of pass@k
    • To let the model learn from its failures between samples
    • Because the tests require at least 200 inputs
    • To make the model's outputs deterministic
  4. Con n campioni e c corretto, quale espressione è lo stimatore imparziale pass@k?
    • c divided by n
    • 1 minus (1 minus c/n) to the power k
    • C(c, k) divided by C(n, k)
    • 1 minus C(n minus c, k) divided by C(n, k)
  5. Cosa misura pass^k, definito nel documento tau-bench?
    • The chance that at least one of k trials succeeds
    • The chance that all k independent trials succeed
    • The average number of steps across k trials
    • The score after k rounds of self-correction
  6. How does sampling temperature typically interact with k?
    • Temperature has no effect on pass@k
    • Low temperature is best for every k
    • Higher temperature tends to help pass@k at large k because samples are more diverse
    • Higher temperature always improves pass@1
  7. Why can a high pass@100 score overstate real-world usefulness?
    • It assumes something already knows which of the samples is correct
    • It only counts the first sample
    • It ignores the unit tests
    • It is always computed with greedy decoding
  8. If n = 20 samples include c = 18 correct ones, what is pass@5?
    • 0.9
    • About 0.5
    • It cannot be computed because n is too small
    • Esattamente 1