Nhungamiro yehunyanzvi

SWE-bhenji uye Coding Agent Benchmarks

SWE-bhenji chiratidzo chinoedza kana AI system inogona kugadzirisa chaiyo GitHub nyaya nekugadzirisa Python codebase kuitira kuti bvunzo dzeprojekiti dzipfuure.

Teerera kugwaro iriAI-yakarondedzerwa · ~3 min

Pa peji ino4 min verenga
  1. Pfupiso
  2. Kudzika Kwakadzika
  3. Kuitwa muNyika yeChokwadi
  4. Ramangwana reSWE-bhenji uye Coding Agent Benchmarks
  5. Kuzvishandisa
  6. Ramba Uchiongorora
  7. Mibvunzo inowanzo bvunzwa

Pfupiso

Izvo zvine basa nekuti ndeimwe yematanho akanyanya kutaurwa evamiriri vekodhi, uye zvibodzwa zvawo zviri nyore kuverenga-pamusoro usingazive kuti mabasa, maajenti scaffolds uye cheki yebvunzo zvinoshanda sei.

Kudzika Kwakadzika

SWE-bhenji rakaunzwa naCarlos Jimenez nevamwe vaaishanda navo kuPrinceton muna 2023 uye rakatsikiswa paICLR 2024. Rine 2,294 zviitiko zvebasa zvakatorwa kubva kune gumi nemaviri anoshandiswa zvakanyanya Python repositories, kusanganisira Django, scikit-dzidza, sympy, mattlib uye requestli. Imwe neimwe muenzaniso inobatanidza chaiyo GitHub nyaya nechikumbiro chekudhonza chakagadzirisa. Iyo sisitimu iri pasi pekuyedzwa inogamuchira iyo nyaya uye repository sezvayaive isati yagadziriswa, uye inofanirwa kuburitsa chigamba. Kugova kunoshandisa bvunzo kubva kune yekutanga kudhonza chikumbiro. FAIL_TO_PASS bvunzo dzakundikana pamberi pekugadzirisa kwevanhu uye vakapfuura mushure, saka vanotarisa kuti nyaya yagadziriswa. PASS_TO_PASS bvunzo dzakapfuura pamberi uye shure, saka vanotarisa kuti hapana chimwe chakatyoka. Chiitiko chinoverengera sekugadziriswa chete kana zvese izvi zvikapfuura, uye nhamba yemusoro ndiyo muzana yakagadziriswa. Pakuvhurwa, iwo emhando yepamusoro akagadzirisa mashoma muzana emabasa. Zvikamu zviviri zvakakosha. SWE-bench Lite ine mazana matatu zviitiko zvakasarudzwa kuve zvakachipa uye zvakanyanya kuzvimiririra. SWE-bhenji Yakasimbiswa, yakaburitswa muna 2024 neOpenAI nevanyori veSWE-bhenji, ine mazana mashanu ezviitiko izvo mainjiniya evanhu akaongorora kuti abvise tsananguro yenyaya isina kujeka kana bvunzo dzinoramba mhinduro dzinoshanda. Verified ikozvino ndiyo inonyanya kutaurwa kupatsanurwa. Zvibodzwa zvinoratidza modhi pamwe nechikafu chayo, chirongwa chinoibvumira kuongorora mafaera, kumhanya mirairo uye kugadzirisa kodhi. SWE-mumiririri kubva kuPrinceton uye iyo Agentless pombi, iyo inogara, kugadzirisa uye kusimbisa pasina yakavhurika-yakapera mumiriri loop, mienzaniso inozivikanwa. Iyo imwechete modhi inogona kukora zvakasiyana zvakanyanya mune dzakasiyana scaffolds. Kusaverengeka kwakajairika: iyo yekutanga seti ndeye Python-chete uye inovhara gumi nemaviri repositori; kupasa bvunzo hakuvimbisi inochengetedzwa kodhi; marepositori uye zvigadziriso zviri pachena, saka kusvibiswa kunogoneka; uye mhedzisiro yekuedza kwakawanda haina kufananidzwa neyeye-yekuedza imwe chete. Mabhenji ane hukama akadai seSWE-bhenji Multilingual, SWE-Lancer uye Terminal-Bench inoongorora mimwe mitauro, mabasa ega uye basa rekuraira-mutsara.

Kuitwa muNyika yeChokwadi

Basa rinopa mumiririri nyaya yeDjango inotsanangura maitiro asina kunaka pamwe nekutarisa repository pane chibvumirano chisati chagadziriswa; mumiririri anofanira kunyora chigamba, icho chinozotongwa nemiedzo iyo yekutanga kugadziriswa kwevanhu yakapfuura.

Mutengesi anoshuma SWE-bhenji Verified mamakisi ichishandisa yayo mumiririri harness uye akati wandei kuedza pabasa; mutengi anofanira kusimbisa kana iri mhedzisiro yekuedza kumwe chete asati aienzanisa nenhamba yemumwe mutengesi.

Chikwata cheinjiniya chinovaka yemukati SWE-bhenji-maitiro akaiswa kubva kune ayo akavanzika repositori uye yakagadzirisa matikiti ekutarisa kana mumiriri anomira zvakanaka pachena anoshanda pane yavo kodhi.

Vatsvagiri vanoshandisa SWE-bhenji Multimodal, uko nyaya dzinobva kuJavaScript kumberi-kumagumo mapurojekiti uye kazhinji dzinosanganisira zviratidziro zvezvipembenene zvinoonekwa, kuyedza hunyanzvi iyo Python-chete yepakutanga haigone kuyera.

Ramangwana reSWE-bhenji uye Coding Agent Benchmarks

Sezvibodzwa zvepamusoro paSWE-bhenji Yakasimbiswa inosimuka, bhenji rinopatsanura masisitimu anotungamira zvishoma zvakajeka, uye kutarisa kuri kuenda kumaseti matsva ebasa ane nyaya nyowani, mitauro yakawanda yehurongwa, basa rakareba-nhanho uye kushuma kwemutengo uye kuvimbika. Kusvibiswa kunetsa kunogara nekuti tsime repositori ndeyeruzhinji, izvo zvinofarira mabhenji anowedzera mabasa achangoburwa. Kuzvikwata zvinosarudza maturusi, mabhodhi evatungamiri vevoruzhinji inzvimbo yekutanga; vamiririri vekuyedza pamacodebases avo uye matikiti anoramba ari chiratidzo chakavimbika.

Kuzvishandisa

Nei zvichikosha

  • Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
  • Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
  • Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.

Chenjera

  • Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
  • Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
  • Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.

Matanho anotevera

  1. Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
  2. Benchmark pasi pechokwadi mutoro uye data mamiriro.
  3. Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
  4. Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.

Ramba Uchiongorora

Tsamba yemahara

Tora iyo yemazuva ese AI pfupiso

Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.

Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.

Zviedze iwe pachako

Take the SWE-bench and Coding Agent Benchmarks quiz

Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.

Tanga mibvunzo

Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo

Mibvunzo inowanzo bvunzwa

Chii chinonzi SWE-bhenji uye Coding Agent Benchmarks?

SWE-bhenji chiratidzo chinoedza kana AI system inogona kugadzirisa chaiyo GitHub nyaya nekugadzirisa Python codebase kuitira kuti bvunzo dzeprojekiti dzipfuure. Izvo zvine basa nekuti ndeimwe yematanho akanyanya kutaurwa evamiriri vekodhi, uye zvibodzwa zvawo zviri nyore kuverenga-pamusoro usingazive kuti mabasa, maajenti scaffolds uye cheki yebvunzo zvinoshanda sei.

Chii chinokumbirwa neSWE-bhenji kuti sisitimu iite?

Chiitiko chega chega chinobatanidza nyaya chaiyo ine repository snapshot, uye sisitimu inofanirwa kugadzira kodhi chigamba chinoigadzirisa.

FAIL_TO_PASS bvunzo chii?

Vanosimbisa kuti nyaya yacho yakanyatsogadziriswa, nekuti ivo vanongopfuura kana kugadzirisa kwavepo.

Ndeipi basa rePASS_TO_PASS bvunzo?

Ivo vakapfuura pamberi uye mushure mekugadzirisa kwemunhu, saka vanochengetedza kusadzokororwa kunounzwa nechigamba chemumiririri.

Chii chinonzi SWE-bhenji Yakasimbiswa?

Verified yakaburitswa muna 2024 neOpenAI nevanyori veSWE-bhenji, kusefa mabasa akange asina kutsanangurwa kana kuti aive nemiedzo yairamba mhinduro dzinoshanda.

Mangani ezviitiko ari muSWE-bhenji Lite?

Lite ndeye 300-chiitiko subset yakasarudzwa kuve yakachipa uye inozvimiririra.