Guide-yakabatana mibvunzo · Pakati Nhanho

Pass@k Metric Yakatsanangurwa Mibvunzo

Inoedza tsananguro yekupasa @ k, muyedzero asina kwaakarerekera kubva kun samples, pfuura ^k kuvimbika, uye nzira dzinozivikanwa dzekupasa@k zvibodzwa kuverengerwa zvisizvo kana kuwedzera.

Nzira dzekutungamira dzinoenderanaPfuura PaK Metric
Mubvunzo 1 ye 8

Ko pass@k inoyera chii kune rimwe dambudziko?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Ko pass@k inoyera chii kune rimwe dambudziko?
    • Chikamu che k samples dzinopasa bvunzo
    • Mukana wekuti imwe chete ye k sampled yekuedza ipfuure bvunzo
    • Iko mukana wekuti kuedza kwekutanga kunopfuura mushure me k edzazve kugadzirisa-zvakanaka
    • Huwandu hwebvunzo dzakapasa kunze kwek bvunzo
  2. HumanEval, bhenji rakaunzwa nebepa reCodex, rine chii?
    • 2,294 GitHub nyaya kubva kuPython mapurojekiti
    • 1,000 yemakwikwi ekuronga matambudziko muC ++
    • 164 yakanyorwa nemaoko Python matambudziko ekuronga nemayuniti bvunzo
    • 500 vanhu-yakasimbiswa bug mishumo
  3. Sei bepa reCodex richigadzira n samples ine n yakakura kudarika k?
    • Kuverengera fungidziro isina kurerekera, yakaderera-mutsauko we pass@k
    • Kuita kuti modhi idzidze kubva mukukundikana kwayo pakati pemasampuli
    • Nekuti bvunzo dzinoda anokwana mazana maviri ekuisa
    • Kuita kuti zvinobuda zvemodhi zvive deterministic
  4. Ne n samples uye c chaiyo, ndechipi chirevo chinomiririra pass@k estimator?
    • c akapatsanurwa na n
    • 1 kubvisa (1 minus c/n) kusimba k
    • C(c, k) yakakamurwa naC(n,k)
    • 1 kubvisa C(n kubvisa c, k) yakakamurwa naC(n, k)
  5. Chii chinonzi pass^k, chinotsanangurwa mubepa retau-bhenji, chiyero?
    • Mukana wekuti kana imwe yemayedzo k inobudirira
    • Mukana wekuti yese k miyedzo yakazvimirira inobudirira
    • Avhareji yenhamba yematanho pane k miedzo
    • Zvibodzwa mushure me k kutenderera kwekuzvigadzirisa
  6. Sampling tembiricha inowanzobatana sei ne k?
    • Tembiricha haina zvazvinoita pa pass@k
    • Tembiricha yakaderera yakanakira chero k
    • Kupisa kwepamusoro kunobatsira pass@k pak hombe k nekuti masampuli akawanda akasiyana
    • Tembiricha yepamusoro inogara ichivandudza pass@1
  7. Sei yakakwirira pass@100 mamakisi overstate chaiyo-nyika inobatsira?
    • Inotora chimwe chinhu chinotoziva kuti ndeipi yemasampuli ndeyechokwadi
    • Inongoverenga muenzaniso wekutanga
    • Iyo inofuratira bvunzo dzeyuniti
    • Inogara ichiverengerwa nekukara decoding
  8. Kana n = 20 samples inosanganisira c = 18 chaiyo, pass@5 chii?
    • 0.9
    • Zvinenge 0.5
    • Haikwanise kuverengerwa nekuti n idiki
    • Ndizvozvo 1