InoteveraGaidhi rinotevera
RAG Evaluation Metrics uye RAGAS
Tekinoroji
Nhungamiro yehunyanzvi
Pass@k ndiyo mukana wekuti ingangoita imwe ye k sampled kuyedza kubva ku modhi inogadzirisa dambudziko, kazhinji inotariswa nekumhanyisa yakavanzika unit bvunzo pane yakagadzirwa kodhi.
Teerera kugwaro iriAI-yakarondedzerwa · ~3 min
Izvo zvine basa nekuti kodhi uye mabhenji mabhenji anozvitaura nguva dzose, uye pass@10 kana pass@100 mamakisi anogona kutaridzika zvirinani pane izvo mushandisi anowana kubva mukuedza kumwe chete.
Pass@k yakakura kubva mukutsvagisa kwechirongwa, uko Kulal nevamwe vaaishanda navo vakaishandisa muna 2019 kuita pseudocode-to-code mabasa, uye yakave yakajairwa neOpenAI's 2021 Codex bepa, iyo yakaunza HumanEval, seti ye164 yakanyorwa nemaoko ZXQZAIU1QXZ matambudziko ekuronga. Padambudziko rega rega modhi inonyora basa, uye basa rinoverengera sechakarurama chete kana ikapfuura yakavanzika unit bvunzo. Pass@k anobvunza: kana wakadhirowa k samples, ndeupi mukana wekuti imwe inopfuura? Chiyero chebhenji ndechekuti mukana wakaenzana pamusoro pematambudziko ese. Nzira yekusaziva ndeyekugadzira chaizvo k samples padambudziko uye tarisa kana paine pfuura, asi fungidziro iyi ine ruzha. Bepa reCodex pachinzvimbo rinoburitsa n samples, apo n ihombe pane k (semuyenzaniso 200), inoverenga c echokwadi, uye inokokorodza mukana wekuti chidimbu chechikamu chek samples chine kanenge sampuli imwe chaiyo: 1 kubvisa C(n kubvisa c, k) yakakamurwa naC(n, k). Iyi fungidziro haina rusaruro uye ishoma kuchinjika. Nzira yekudimbudzira c/n mu 1 minus (1 minus p) kune simba k yakarerekera. Sampling tembiricha inokosha. Pass@1 inowanzo nyanya kuderera tembiricha, nepo pass@100 inobatsira kubva kune yakakwira tembiricha nekuti akawanda akasiyana sampuli anosimudza mukana wekuti munhu ashande. Mapepa anowanzo tora tembiricha yakasiyana kune yega k. Agent mabhenji akawedzera metric inowirirana. Iyo tau-bhenji bepa kubva kuSierra (2024) yakatsanangura pass^k, mukana wekuti yese k miedzo yakazvimirira inobudirira. Inoyera kuvimbika pane kugona. Kusaverengeka kwakajairika kunosanganisira kuenzanisa pass@10 yeimwe modhi neyomumwe pass@1, kukanganwa kuti pass@k inotora chimwe chinhu chatoziva kuti ndeipi sampuro yakarurama, uye kubata kukara-decoding pass@1 zvakafanana nesampled pass@1. Weak unit test inopinza yese pass@k nhamba, ndosaka EvalPlus yakawedzera mimwe bvunzo kuHumanEval uye yakaona zvibodzwa zvichidonha.
Chikwata chinoongorora modhi yekodhi paHumanEval sampuli mazana maviri ekupedzisa padambudziko, inoverenga kuti vangani vakapasa bvunzo, uye mishumo pass@1, pass@10 uye pass@100 kubva kune iyo imwe seti yemasampuli uchishandisa iyo isina kurerekera estimator.
Mupepeti anopedza otomatiki anoratidza zano rimwe chete rinofanirwa kutongwa nepass@1, nepo chishandiso chinoburitsa vashanu vakwikwidzi uye chinomhanyisa bvunzo yako sutu yekutora inoshanda iri padyo nepasi @ 5 mamiriro.
Mutengi-rutsigiro mumiririri anobudirira pamakumi manomwe neshanu muzana emiedzo anotaridzika akasimba papass@k, asi kana zviyedzo zvikazvimiririra kupasa kwayo^4 zvibodzwa (miyedzo mina yose inobudirira) inongoita 0.32 chete, inoratidza kusavimbika kwayaizonzwa pakushandiswa kwakadzokororwa.
Iyo yemakwikwi programming system inogadzira mapurogiramu akawanda evamiriri, anoasefa achipesana nemuenzaniso bvunzo, uye anoendesa nhamba diki; chiyero chayo chekugadzirisa chinoenderana nekuti mangani ekutumira akabvumidzwa, inova pass@k-style bhajeti.
Sezvo mamodheru anofamba kubva pakupedzisa kumwe kuenda kune akawanda-nhanho vamiririri, kuvimbika matanho akadai pass^k uye run-to-run consistency ari kuwedzera kutariswa padivi pass@k, nekuti vashandisi vanosangana nekuedza kumwe chete, kwete kwakanakisa kwevazhinji. Vanyori veBenchmark varikudyarawo mune yakasimba bvunzo masutu, sezvo bvunzo dzisina kusimba dzinorega kodhi isiriyo ipfuure uye inflate metric yese. Mishumo yakangwarira inofanira kutaura k, sampuli kuverenga, tembiricha uye kana danho rekusarudza rakashandiswa, asi tsika inosiyana, saka vaverengi vanofanirwa kuramba vachiongorora izvo zvinhu pane kuzvifungira.
Tsamba yemahara
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Pass@k ndiyo mukana wekuti ingangoita imwe ye k sampled kuyedza kubva ku modhi inogadzirisa dambudziko, kazhinji inotariswa nekumhanyisa yakavanzika unit bvunzo pane yakagadzirwa kodhi. Izvo zvine basa nekuti kodhi uye mabhenji mabhenji anozvitaura nguva dzose, uye pass@10 kana pass@100 mamakisi anogona kutaridzika zvirinani pane izvo mushandisi anowana kubva mukuedza kumwe chete.
Pass@k mukana wekuti kana imwe yekuedza k ibudirire. Iyo benchmark mamakisi maavhareji mukana iwoyo pamusoro pematambudziko ese.
HumanEval ine 164 yakanyorwa nemaoko Python matambudziko, imwe neimwe yakatariswa neyakavanzika unit bvunzo.
Kugadzira chaizvo k samples kunopa fungidziro ine ruzha. Kushandisa n samples uye kuverenga c chaiko kunobvumira fungidziro isina kurerekera ine misiyano mishoma.
Ndiyo imwe chete kubvisa mukana wekuti chingori k-subset inongove nemasamples asiri iwo. C(c, k)/C(n, k) ndiyo pass^k estimator, uye simba-k nzira yekudimbudzira yakarerekera.
Pass^k inoda kuti muyedzo wese ubudirire, saka inoyera kuvimbika, izvo zvine basa kune vamiririri vanoshandira vashandisi vazhinji.
Ramba uchidzidza
Mamwe madhairekitori akasarudzirwa nyaya iyi
InoteveraGaidhi rinotevera
RAG Evaluation Metrics uye RAGAS
Tekinoroji