Nhungamiro yehunyanzvi
SWE-bhenji uye Coding Agent Benchmarks
SWE-bhenji chiratidzo chinoedza kana AI system inogona kugadzirisa chaiyo GitHub nyaya nekugadzirisa Python codebase kuitira kuti bvunzo dzeprojekiti dzipfuure.
Pa peji ino4 min verenga
Pfupiso
Izvo zvine basa nekuti ndeimwe yematanho akanyanya kutaurwa evamiriri vekodhi, uye zvibodzwa zvawo zviri nyore kuverenga-pamusoro usingazive kuti mabasa, maajenti scaffolds uye cheki yebvunzo zvinoshanda sei.
Kudzika Kwakadzika
SWE-bhenji rakaunzwa naCarlos Jimenez nevamwe vaaishanda navo kuPrinceton muna 2023 uye rakatsikiswa paICLR 2024. Rine 2,294 zviitiko zvebasa zvakatorwa kubva kune gumi nemaviri anoshandiswa zvakanyanya Python repositories, kusanganisira Django, scikit-dzidza, sympy, mattlib uye requestli. Imwe neimwe muenzaniso inobatanidza chaiyo GitHub nyaya nechikumbiro chekudhonza chakagadzirisa. Iyo sisitimu iri pasi pekuyedzwa inogamuchira iyo nyaya uye repository sezvayaive isati yagadziriswa, uye inofanirwa kuburitsa chigamba. Kugova kunoshandisa bvunzo kubva kune yekutanga kudhonza chikumbiro. FAIL_TO_PASS bvunzo dzakundikana pamberi pekugadzirisa kwevanhu uye vakapfuura mushure, saka vanotarisa kuti nyaya yagadziriswa. PASS_TO_PASS bvunzo dzakapfuura pamberi uye shure, saka vanotarisa kuti hapana chimwe chakatyoka. Chiitiko chinoverengera sekugadziriswa chete kana zvese izvi zvikapfuura, uye nhamba yemusoro ndiyo muzana yakagadziriswa. Pakuvhurwa, iwo emhando yepamusoro akagadzirisa mashoma muzana emabasa. Zvikamu zviviri zvakakosha. SWE-bench Lite ine mazana matatu zviitiko zvakasarudzwa kuve zvakachipa uye zvakanyanya kuzvimiririra. SWE-bhenji Yakasimbiswa, yakaburitswa muna 2024 neOpenAI nevanyori veSWE-bhenji, ine mazana mashanu ezviitiko izvo mainjiniya evanhu akaongorora kuti abvise tsananguro yenyaya isina kujeka kana bvunzo dzinoramba mhinduro dzinoshanda. Verified ikozvino ndiyo inonyanya kutaurwa kupatsanurwa. Zvibodzwa zvinoratidza modhi pamwe nechikafu chayo, chirongwa chinoibvumira kuongorora mafaera, kumhanya mirairo uye kugadzirisa kodhi. SWE-mumiririri kubva kuPrinceton uye iyo Agentless pombi, iyo inogara, kugadzirisa uye kusimbisa pasina yakavhurika-yakapera mumiriri loop, mienzaniso inozivikanwa. Iyo imwechete modhi inogona kukora zvakasiyana zvakanyanya mune dzakasiyana scaffolds. Kusaverengeka kwakajairika: iyo yekutanga seti ndeye Python-chete uye inovhara gumi nemaviri repositori; kupasa bvunzo hakuvimbisi inochengetedzwa kodhi; marepositori uye zvigadziriso zviri pachena, saka kusvibiswa kunogoneka; uye mhedzisiro yekuedza kwakawanda haina kufananidzwa neyeye-yekuedza imwe chete. Mabhenji ane hukama akadai seSWE-bhenji Multilingual, SWE-Lancer uye Terminal-Bench inoongorora mimwe mitauro, mabasa ega uye basa rekuraira-mutsara.
Strategic Impact
Mutengo uye bhajeti
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
Sarudzo dzakajeka
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
Kudzora kwemhando yepamusoro
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Ramangwana reSWE-bhenji uye Coding Agent Benchmarks
Sezvibodzwa zvepamusoro paSWE-bhenji Yakasimbiswa inosimuka, bhenji rinopatsanura masisitimu anotungamira zvishoma zvakajeka, uye kutarisa kuri kuenda kumaseti matsva ebasa ane nyaya nyowani, mitauro yakawanda yehurongwa, basa rakareba-nhanho uye kushuma kwemutengo uye kuvimbika. Kusvibiswa kunetsa kunogara nekuti tsime repositori ndeyeruzhinji, izvo zvinofarira mabhenji anowedzera mabasa achangoburwa. Kuzvikwata zvinosarudza maturusi, mabhodhi evatungamiri vevoruzhinji inzvimbo yekutanga; vamiririri vekuyedza pamacodebases avo uye matikiti anoramba ari chiratidzo chakavimbika.
Real-World Implementation
Basa rinopa mumiririri nyaya yeDjango inotsanangura maitiro asina kunaka pamwe nekutarisa repository pane chibvumirano chisati chagadziriswa; mumiririri anofanira kunyora chigamba, icho chinozotongwa nemiedzo iyo yekutanga kugadziriswa kwevanhu yakapfuura.
Mutengesi anoshuma SWE-bhenji Verified mamakisi ichishandisa yayo mumiririri harness uye akati wandei kuedza pabasa; mutengi anofanira kusimbisa kana iri mhedzisiro yekuedza kumwe chete asati aienzanisa nenhamba yemumwe mutengesi.
Chikwata cheinjiniya chinovaka yemukati SWE-bhenji-maitiro akaiswa kubva kune ayo akavanzika repositori uye yakagadzirisa matikiti ekutarisa kana mumiriri anomira zvakanaka pachena anoshanda pane yavo kodhi.
Vatsvagiri vanoshandisa SWE-bhenji Multimodal, uko nyaya dzinobva kuJavaScript kumberi-kumagumo mapurojekiti uye kazhinji dzinosanganisira zviratidziro zvezvipembenene zvinoonekwa, kuyedza hunyanzvi iyo Python-chete yepakutanga haigone kuyera.
Njodzi & Guardrails
Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Implementation Roadmap
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
Benchmark pasi pechokwadi mutoro uye data mamiriro.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SWE-bench and Coding Agent Benchmarks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Mibvunzo inowanzo bvunzwa
Chii chinonzi SWE-bhenji uye Coding Agent Benchmarks?
SWE-bhenji chiratidzo chinoedza kana AI system inogona kugadzirisa chaiyo GitHub nyaya nekugadzirisa Python codebase kuitira kuti bvunzo dzeprojekiti dzipfuure. Izvo zvine basa nekuti ndeimwe yematanho akanyanya kutaurwa evamiriri vekodhi, uye zvibodzwa zvawo zviri nyore kuverenga-pamusoro usingazive kuti mabasa, maajenti scaffolds uye cheki yebvunzo zvinoshanda sei.
Chii chinokumbirwa neSWE-bhenji kuti sisitimu iite?
Chiitiko chega chega chinobatanidza nyaya chaiyo ine repository snapshot, uye sisitimu inofanirwa kugadzira kodhi chigamba chinoigadzirisa.
FAIL_TO_PASS bvunzo chii?
Vanosimbisa kuti nyaya yacho yakanyatsogadziriswa, nekuti ivo vanongopfuura kana kugadzirisa kwavepo.
Ndeipi basa rePASS_TO_PASS bvunzo?
Ivo vakapfuura pamberi uye mushure mekugadzirisa kwemunhu, saka vanochengetedza kusadzokororwa kunounzwa nechigamba chemumiririri.
Chii chinonzi SWE-bhenji Yakasimbiswa?
Verified yakaburitswa muna 2024 neOpenAI nevanyori veSWE-bhenji, kusefa mabasa akange asina kutsanangurwa kana kuti aive nemiedzo yairamba mhinduro dzinoshanda.
Mangani ezviitiko ari muSWE-bhenji Lite?
Lite ndeye 300-chiitiko subset yakasarudzwa kuve yakachipa uye inozvimiririra.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi