Nhungamiro yehunyanzvi

Pass@k Metric Yakatsanangurwa

Pass@k ndiyo mukana wekuti ingangoita imwe ye k sampled kuyedza kubva ku modhi inogadzirisa dambudziko, kazhinji inotariswa nekumhanyisa yakavanzika unit bvunzo pane yakagadzirwa kodhi.

Teerera kugwaro iriAI-yakarondedzerwa · ~3 min

Pa peji ino4 min verenga
  1. Pfupiso
  2. Kudzika Kwakadzika
  3. Kuitwa muNyika yeChokwadi
  4. Ramangwana rePass@k Metric Yakatsanangurwa
  5. Kuzvishandisa
  6. Ramba Uchiongorora
  7. Mibvunzo inowanzo bvunzwa

Pfupiso

Izvo zvine basa nekuti kodhi uye mabhenji mabhenji anozvitaura nguva dzose, uye pass@10 kana pass@100 mamakisi anogona kutaridzika zvirinani pane izvo mushandisi anowana kubva mukuedza kumwe chete.

Kudzika Kwakadzika

Pass@k yakakura kubva mukutsvagisa kwechirongwa, uko Kulal nevamwe vaaishanda navo vakaishandisa muna 2019 kuita pseudocode-to-code mabasa, uye yakave yakajairwa neOpenAI's 2021 Codex bepa, iyo yakaunza HumanEval, seti ye164 yakanyorwa nemaoko ZXQZAIU1QXZ matambudziko ekuronga. Padambudziko rega rega modhi inonyora basa, uye basa rinoverengera sechakarurama chete kana ikapfuura yakavanzika unit bvunzo. Pass@k anobvunza: kana wakadhirowa k samples, ndeupi mukana wekuti imwe inopfuura? Chiyero chebhenji ndechekuti mukana wakaenzana pamusoro pematambudziko ese. Nzira yekusaziva ndeyekugadzira chaizvo k samples padambudziko uye tarisa kana paine pfuura, asi fungidziro iyi ine ruzha. Bepa reCodex pachinzvimbo rinoburitsa n samples, apo n ihombe pane k (semuyenzaniso 200), inoverenga c echokwadi, uye inokokorodza mukana wekuti chidimbu chechikamu chek samples chine kanenge sampuli imwe chaiyo: 1 kubvisa C(n kubvisa c, k) yakakamurwa naC(n, k). Iyi fungidziro haina rusaruro uye ishoma kuchinjika. Nzira yekudimbudzira c/n mu 1 minus (1 minus p) kune simba k yakarerekera. Sampling tembiricha inokosha. Pass@1 inowanzo nyanya kuderera tembiricha, nepo pass@100 inobatsira kubva kune yakakwira tembiricha nekuti akawanda akasiyana sampuli anosimudza mukana wekuti munhu ashande. Mapepa anowanzo tora tembiricha yakasiyana kune yega k. Agent mabhenji akawedzera metric inowirirana. Iyo tau-bhenji bepa kubva kuSierra (2024) yakatsanangura pass^k, mukana wekuti yese k miedzo yakazvimirira inobudirira. Inoyera kuvimbika pane kugona. Kusaverengeka kwakajairika kunosanganisira kuenzanisa pass@10 yeimwe modhi neyomumwe pass@1, kukanganwa kuti pass@k inotora chimwe chinhu chatoziva kuti ndeipi sampuro yakarurama, uye kubata kukara-decoding pass@1 zvakafanana nesampled pass@1. Weak unit test inopinza yese pass@k nhamba, ndosaka EvalPlus yakawedzera mimwe bvunzo kuHumanEval uye yakaona zvibodzwa zvichidonha.

Kuitwa muNyika yeChokwadi

Chikwata chinoongorora modhi yekodhi paHumanEval sampuli mazana maviri ekupedzisa padambudziko, inoverenga kuti vangani vakapasa bvunzo, uye mishumo pass@1, pass@10 uye pass@100 kubva kune iyo imwe seti yemasampuli uchishandisa iyo isina kurerekera estimator.

Mupepeti anopedza otomatiki anoratidza zano rimwe chete rinofanirwa kutongwa nepass@1, nepo chishandiso chinoburitsa vashanu vakwikwidzi uye chinomhanyisa bvunzo yako sutu yekutora inoshanda iri padyo nepasi @ 5 mamiriro.

Mutengi-rutsigiro mumiririri anobudirira pamakumi manomwe neshanu muzana emiedzo anotaridzika akasimba papass@k, asi kana zviyedzo zvikazvimiririra kupasa kwayo^4 zvibodzwa (miyedzo mina yose inobudirira) inongoita 0.32 chete, inoratidza kusavimbika kwayaizonzwa pakushandiswa kwakadzokororwa.

Iyo yemakwikwi programming system inogadzira mapurogiramu akawanda evamiriri, anoasefa achipesana nemuenzaniso bvunzo, uye anoendesa nhamba diki; chiyero chayo chekugadzirisa chinoenderana nekuti mangani ekutumira akabvumidzwa, inova pass@k-style bhajeti.

Ramangwana rePass@k Metric Yakatsanangurwa

Sezvo mamodheru anofamba kubva pakupedzisa kumwe kuenda kune akawanda-nhanho vamiririri, kuvimbika matanho akadai pass^k uye run-to-run consistency ari kuwedzera kutariswa padivi pass@k, nekuti vashandisi vanosangana nekuedza kumwe chete, kwete kwakanakisa kwevazhinji. Vanyori veBenchmark varikudyarawo mune yakasimba bvunzo masutu, sezvo bvunzo dzisina kusimba dzinorega kodhi isiriyo ipfuure uye inflate metric yese. Mishumo yakangwarira inofanira kutaura k, sampuli kuverenga, tembiricha uye kana danho rekusarudza rakashandiswa, asi tsika inosiyana, saka vaverengi vanofanirwa kuramba vachiongorora izvo zvinhu pane kuzvifungira.

Kuzvishandisa

Nei zvichikosha

  • Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
  • Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
  • Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.

Chenjera

  • Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
  • Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
  • Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.

Matanho anotevera

  1. Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
  2. Benchmark pasi pechokwadi mutoro uye data mamiriro.
  3. Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
  4. Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.

Ramba Uchiongorora

Tsamba yemahara

Tora iyo yemazuva ese AI pfupiso

Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.

Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.

Zviedze iwe pachako

Take the Pass@k Metric Explained quiz

Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.

Tanga mibvunzo

Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo

Mibvunzo inowanzo bvunzwa

Chii chinonzi Pass@k Metric Inotsanangurwa?

Pass@k ndiyo mukana wekuti ingangoita imwe ye k sampled kuyedza kubva ku modhi inogadzirisa dambudziko, kazhinji inotariswa nekumhanyisa yakavanzika unit bvunzo pane yakagadzirwa kodhi. Izvo zvine basa nekuti kodhi uye mabhenji mabhenji anozvitaura nguva dzose, uye pass@10 kana pass@100 mamakisi anogona kutaridzika zvirinani pane izvo mushandisi anowana kubva mukuedza kumwe chete.

Ko pass@k inoyera chii kune rimwe dambudziko?

Pass@k mukana wekuti kana imwe yekuedza k ibudirire. Iyo benchmark mamakisi maavhareji mukana iwoyo pamusoro pematambudziko ese.

HumanEval, bhenji rakaunzwa nebepa reCodex, rine chii?

HumanEval ine 164 yakanyorwa nemaoko Python matambudziko, imwe neimwe yakatariswa neyakavanzika unit bvunzo.

Sei bepa reCodex richigadzira n samples ine n yakakura kudarika k?

Kugadzira chaizvo k samples kunopa fungidziro ine ruzha. Kushandisa n samples uye kuverenga c chaiko kunobvumira fungidziro isina kurerekera ine misiyano mishoma.

Ne n samples uye c chaiyo, ndechipi chirevo chinomiririra pass@k estimator?

Ndiyo imwe chete kubvisa mukana wekuti chingori k-subset inongove nemasamples asiri iwo. C(c, k)/C(n, k) ndiyo pass^k estimator, uye simba-k nzira yekudimbudzira yakarerekera.

Chii chinonzi pass^k, chinotsanangurwa mubepa retau-bhenji, chiyero?

Pass^k inoda kuti muyedzo wese ubudirire, saka inoyera kuvimbika, izvo zvine basa kune vamiririri vanoshandira vashandisi vazhinji.