InoteveraGaidhi rinotevera
IFEval and Instruction-Following Benchmarks
Tekinoroji
Nhungamiro yehunyanzvi
SWE-bhenji chiratidzo chinoedza kana AI system inogona kugadzirisa chaiyo GitHub nyaya nekugadzirisa Python codebase kuitira kuti bvunzo dzeprojekiti dzipfuure.
Teerera kugwaro iriAI-yakarondedzerwa · ~3 min
Izvo zvine basa nekuti ndeimwe yematanho akanyanya kutaurwa evamiriri vekodhi, uye zvibodzwa zvawo zviri nyore kuverenga-pamusoro usingazive kuti mabasa, maajenti scaffolds uye cheki yebvunzo zvinoshanda sei.
SWE-bhenji rakaunzwa naCarlos Jimenez nevamwe vaaishanda navo kuPrinceton muna 2023 uye rakatsikiswa paICLR 2024. Rine 2,294 zviitiko zvebasa zvakatorwa kubva kune gumi nemaviri anoshandiswa zvakanyanya Python repositories, kusanganisira Django, scikit-dzidza, sympy, mattlib uye requestli. Imwe neimwe muenzaniso inobatanidza chaiyo GitHub nyaya nechikumbiro chekudhonza chakagadzirisa. Iyo sisitimu iri pasi pekuyedzwa inogamuchira iyo nyaya uye repository sezvayaive isati yagadziriswa, uye inofanirwa kuburitsa chigamba. Kugova kunoshandisa bvunzo kubva kune yekutanga kudhonza chikumbiro. FAIL_TO_PASS bvunzo dzakundikana pamberi pekugadzirisa kwevanhu uye vakapfuura mushure, saka vanotarisa kuti nyaya yagadziriswa. PASS_TO_PASS bvunzo dzakapfuura pamberi uye shure, saka vanotarisa kuti hapana chimwe chakatyoka. Chiitiko chinoverengera sekugadziriswa chete kana zvese izvi zvikapfuura, uye nhamba yemusoro ndiyo muzana yakagadziriswa. Pakuvhurwa, iwo emhando yepamusoro akagadzirisa mashoma muzana emabasa. Zvikamu zviviri zvakakosha. SWE-bench Lite ine mazana matatu zviitiko zvakasarudzwa kuve zvakachipa uye zvakanyanya kuzvimiririra. SWE-bhenji Yakasimbiswa, yakaburitswa muna 2024 neOpenAI nevanyori veSWE-bhenji, ine mazana mashanu ezviitiko izvo mainjiniya evanhu akaongorora kuti abvise tsananguro yenyaya isina kujeka kana bvunzo dzinoramba mhinduro dzinoshanda. Verified ikozvino ndiyo inonyanya kutaurwa kupatsanurwa. Zvibodzwa zvinoratidza modhi pamwe nechikafu chayo, chirongwa chinoibvumira kuongorora mafaera, kumhanya mirairo uye kugadzirisa kodhi. SWE-mumiririri kubva kuPrinceton uye iyo Agentless pombi, iyo inogara, kugadzirisa uye kusimbisa pasina yakavhurika-yakapera mumiriri loop, mienzaniso inozivikanwa. Iyo imwechete modhi inogona kukora zvakasiyana zvakanyanya mune dzakasiyana scaffolds. Kusaverengeka kwakajairika: iyo yekutanga seti ndeye Python-chete uye inovhara gumi nemaviri repositori; kupasa bvunzo hakuvimbisi inochengetedzwa kodhi; marepositori uye zvigadziriso zviri pachena, saka kusvibiswa kunogoneka; uye mhedzisiro yekuedza kwakawanda haina kufananidzwa neyeye-yekuedza imwe chete. Mabhenji ane hukama akadai seSWE-bhenji Multilingual, SWE-Lancer uye Terminal-Bench inoongorora mimwe mitauro, mabasa ega uye basa rekuraira-mutsara.
Basa rinopa mumiririri nyaya yeDjango inotsanangura maitiro asina kunaka pamwe nekutarisa repository pane chibvumirano chisati chagadziriswa; mumiririri anofanira kunyora chigamba, icho chinozotongwa nemiedzo iyo yekutanga kugadziriswa kwevanhu yakapfuura.
Mutengesi anoshuma SWE-bhenji Verified mamakisi ichishandisa yayo mumiririri harness uye akati wandei kuedza pabasa; mutengi anofanira kusimbisa kana iri mhedzisiro yekuedza kumwe chete asati aienzanisa nenhamba yemumwe mutengesi.
Chikwata cheinjiniya chinovaka yemukati SWE-bhenji-maitiro akaiswa kubva kune ayo akavanzika repositori uye yakagadzirisa matikiti ekutarisa kana mumiriri anomira zvakanaka pachena anoshanda pane yavo kodhi.
Vatsvagiri vanoshandisa SWE-bhenji Multimodal, uko nyaya dzinobva kuJavaScript kumberi-kumagumo mapurojekiti uye kazhinji dzinosanganisira zviratidziro zvezvipembenene zvinoonekwa, kuyedza hunyanzvi iyo Python-chete yepakutanga haigone kuyera.
Sezvibodzwa zvepamusoro paSWE-bhenji Yakasimbiswa inosimuka, bhenji rinopatsanura masisitimu anotungamira zvishoma zvakajeka, uye kutarisa kuri kuenda kumaseti matsva ebasa ane nyaya nyowani, mitauro yakawanda yehurongwa, basa rakareba-nhanho uye kushuma kwemutengo uye kuvimbika. Kusvibiswa kunetsa kunogara nekuti tsime repositori ndeyeruzhinji, izvo zvinofarira mabhenji anowedzera mabasa achangoburwa. Kuzvikwata zvinosarudza maturusi, mabhodhi evatungamiri vevoruzhinji inzvimbo yekutanga; vamiririri vekuyedza pamacodebases avo uye matikiti anoramba ari chiratidzo chakavimbika.
Tsamba yemahara
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
SWE-bhenji chiratidzo chinoedza kana AI system inogona kugadzirisa chaiyo GitHub nyaya nekugadzirisa Python codebase kuitira kuti bvunzo dzeprojekiti dzipfuure. Izvo zvine basa nekuti ndeimwe yematanho akanyanya kutaurwa evamiriri vekodhi, uye zvibodzwa zvawo zviri nyore kuverenga-pamusoro usingazive kuti mabasa, maajenti scaffolds uye cheki yebvunzo zvinoshanda sei.
Chiitiko chega chega chinobatanidza nyaya chaiyo ine repository snapshot, uye sisitimu inofanirwa kugadzira kodhi chigamba chinoigadzirisa.
Vanosimbisa kuti nyaya yacho yakanyatsogadziriswa, nekuti ivo vanongopfuura kana kugadzirisa kwavepo.
Ivo vakapfuura pamberi uye mushure mekugadzirisa kwemunhu, saka vanochengetedza kusadzokororwa kunounzwa nechigamba chemumiririri.
Verified yakaburitswa muna 2024 neOpenAI nevanyori veSWE-bhenji, kusefa mabasa akange asina kutsanangurwa kana kuti aive nemiedzo yairamba mhinduro dzinoshanda.
Lite ndeye 300-chiitiko subset yakasarudzwa kuve yakachipa uye inozvimiririra.
Ramba uchidzidza
Mamwe madhairekitori akasarudzirwa nyaya iyi
InoteveraGaidhi rinotevera
IFEval and Instruction-Following Benchmarks
Tekinoroji