Kuiz berkaitan panduan · Sederhana Tahap

Kuiz Penjelasan Metrik Lulus@k

Menguji takrif pass@k, penganggar tidak berat sebelah daripada n sampel, kebolehpercayaan lulus^k dan cara biasa skor pass@k mendapat salah baca atau dinaikkan.

Laluan panduan yang berkaitanLulus Di K Metrik
soalan 1 daripada 8

Apakah ukuran pass@k untuk satu masalah?

Pratonton setiap soalan8 soalan

Jawapan dan penjelasan muncul semasa anda mengambil kuiz.

  1. Apakah ukuran pass@k untuk satu masalah?
    • The fraction of k samples that pass the tests
    • The probability that at least one of k sampled attempts passes the tests
    • The probability that the first attempt passes after k retries of fine-tuning
    • The number of tests passed out of k tests
  2. HumanEval, penanda aras yang diperkenalkan dengan kertas Codex, mengandungi apa?
    • 2,294 GitHub issues from Python projects
    • 1,000 competitive programming problems in C++
    • 164 hand-written Python programming problems with unit tests
    • 500 human-verified bug reports
  3. Mengapakah kertas Codex menjana n sampel dengan n lebih besar daripada k?
    • To compute an unbiased, lower-variance estimate of pass@k
    • To let the model learn from its failures between samples
    • Because the tests require at least 200 inputs
    • To make the model's outputs deterministic
  4. Dengan n sampel dan c betul, ungkapan yang manakah merupakan penganggar pas@k tidak berat sebelah?
    • c divided by n
    • 1 minus (1 minus c/n) to the power k
    • C(c, k) divided by C(n, k)
    • 1 minus C(n minus c, k) divided by C(n, k)
  5. Apakah yang diukur oleh pas^k, yang ditakrifkan dalam kertas bangku tau?
    • The chance that at least one of k trials succeeds
    • The chance that all k independent trials succeed
    • The average number of steps across k trials
    • The score after k rounds of self-correction
  6. How does sampling temperature typically interact with k?
    • Temperature has no effect on pass@k
    • Low temperature is best for every k
    • Higher temperature tends to help pass@k at large k because samples are more diverse
    • Higher temperature always improves pass@1
  7. Why can a high pass@100 score overstate real-world usefulness?
    • It assumes something already knows which of the samples is correct
    • It only counts the first sample
    • It ignores the unit tests
    • It is always computed with greedy decoding
  8. If n = 20 samples include c = 18 correct ones, what is pass@5?
    • 0.9
    • About 0.5
    • It cannot be computed because n is too small
    • Tepat 1