Cuestionario vinculado a una guía · Medio Nivel

Prueba explicativa de la métrica Pass@k

Prueba la definición de pass@k, el estimador insesgado de n muestras, la confiabilidad de pass@k y las formas comunes en que las puntuaciones de pass@k se malinterpretan o se inflan.

Rutas de guía relacionadasPasa en la métrica K
Pregunta 1 de 8

¿Qué mide pass@k para un solo problema?

Vista previa de cada pregunta8 preguntas

Las respuestas y explicaciones aparecen a medida que realiza el cuestionario.

  1. ¿Qué mide pass@k para un solo problema?
    • The fraction of k samples that pass the tests
    • The probability that at least one of k sampled attempts passes the tests
    • The probability that the first attempt passes after k retries of fine-tuning
    • The number of tests passed out of k tests
  2. HumanEval, el benchmark introducido con el artículo Codex, ¿qué contiene?
    • 2,294 GitHub issues from Python projects
    • 1,000 competitive programming problems in C++
    • 164 hand-written Python programming problems with unit tests
    • 500 human-verified bug reports
  3. ¿Por qué el artículo Codex genera n muestras con n mayor que k?
    • To compute an unbiased, lower-variance estimate of pass@k
    • To let the model learn from its failures between samples
    • Because the tests require at least 200 inputs
    • To make the model's outputs deterministic
  4. Con n muestras y c correctas, ¿cuál es la expresión del estimador insesgado de pass@k?
    • c divided by n
    • 1 minus (1 minus c/n) to the power k
    • C(c, k) divided by C(n, k)
    • 1 minus C(n minus c, k) divided by C(n, k)
  5. ¿Qué mide pass^k, definido en el artículo de tau-bench?
    • The chance that at least one of k trials succeeds
    • The chance that all k independent trials succeed
    • The average number of steps across k trials
    • The score after k rounds of self-correction
  6. How does sampling temperature typically interact with k?
    • Temperature has no effect on pass@k
    • Low temperature is best for every k
    • Higher temperature tends to help pass@k at large k because samples are more diverse
    • Higher temperature always improves pass@1
  7. Why can a high pass@100 score overstate real-world usefulness?
    • It assumes something already knows which of the samples is correct
    • It only counts the first sample
    • It ignores the unit tests
    • It is always computed with greedy decoding
  8. If n = 20 samples include c = 18 correct ones, what is pass@5?
    • 0,9
    • About 0.5
    • It cannot be computed because n is too small
    • exactamente 1