HejuruUbuyobozi bukurikira
IFEval hamwe nubuyobozi-Gukurikiza Ibipimo
Tekiniki
Ubuyobozi bwa tekiniki
SWE-intebe ni igipimo gipima niba sisitemu ya AI ishobora gukemura ibibazo nyabyo bya GitHub uhindura codebase ya Python kugirango ibizamini byumushinga byatsinde.
Umva iki gitaboAI-byavuzwe · ~3 min
Nibyingenzi kuko nimwe mungero zavuzwe cyane kubakozi ba code, kandi amanota yayo yoroshye kuyasoma cyane utazi uko imirimo, scaffolds hamwe na cheque yikizamini ikora.
Intebe ya SWE yatangijwe na Carlos Jimenez na bagenzi be i Princeton mu 2023 kandi isohoka kuri ICLR 2024. Irimo ingero 2,294 zashizwe mu bubiko bwa Python bukoreshwa cyane, harimo Django, scikit-kwiga, impuhwe, matplotlib n'ibisabwa. Buri rugero ruhuza ikibazo nyacyo cya GitHub hamwe no gukurura icyifuzo cyakosoye. Sisitemu iri kugeragezwa yakira ikibazo cyibisobanuro hamwe nububiko nkuko byari bimeze mbere yo gukosorwa, kandi bigomba kubyara patch. Gutanga amanota akoresha ibizamini bivuye kubisabwa gukurura. Ibizamini bya FAIL_TO_PASS byananiranye mbere yo gukosora abantu bikanyura nyuma yabyo, bityo bagenzura ko ikibazo cyakemutse. Ibizamini bya PASS_TO_PASS byatsinze mbere na nyuma, bityo bareba ko ntakindi cyacitse. Urugero rubarwa nkuko byakemuwe gusa niba ibyo byose byatsinzwe, numutwe wumutwe nijanisha ryakemuwe. Mugutangiza, icyitegererezo cyiza cyakemuye bike ku ijana byimirimo. Ibice bibiri byingenzi. SWE-intebe Lite ifite ingero 300 zahisemo kuba zihendutse kandi zonyine. Intebe ya SWE Yagenzuwe, yasohowe mu 2024 na OpenAI hamwe n’abanditsi ba SWE-intebe, ifite inshuro 500 abashakashatsi b’abantu bagenzuye kugira ngo bakureho ibisobanuro bidasobanutse cyangwa ibizamini byanga ibisubizo bifatika. Kugenzurwa ubu nibisanzwe byavuzwe gutandukana. Amanota yerekana icyitegererezo wongeyeho scafold yayo, porogaramu ireka ikareba amadosiye, ikoresha amategeko kandi igahindura code. SWE-umukozi wo muri Princeton hamwe numuyoboro wa Agentless, uhindura, ugasana kandi ukemezwa udafunguye-abakozi bafunguye, ni ingero zizwi. Icyitegererezo kimwe gishobora gutsinda amanota atandukanye muburyo butandukanye. Gusoma nabi: ibice byumwimerere ni Python-gusa kandi ikubiyemo ububiko icumi; gutsinda ibizamini ntabwo byemeza kode ishobora kugumaho; ububiko no gukosora ni rusange, bityo kwanduza birashoboka; n'ibisubizo byinshi-byo kugerageza ntabwo bigereranywa nimwe-igerageza. Ibipimo bifitanye isano nka SWE-intebe Indimi nyinshi, SWE-Lancer na Terminal-Bench ikora izindi ndimi, imirimo yigenga hamwe nakazi kayobora.
Igikorwa giha umukozi ikibazo cya Django gisobanura imyitwarire itari yo hiyongereyeho no kugenzura ububiko bwiyemeje mbere yo gukosora; umukozi agomba kwandika agapapuro, hanyuma agasuzumwa nibizamini ko umwimerere wambere wumuntu wakoze pass.
Umucuruzi atanga raporo ya SWE-intebe Yagenzuwe amanota ukoresheje ibikoresho byayo bwite kandi igerageza inshuro nyinshi kuri buri gikorwa; umuguzi agomba kwemeza niba ari igisubizo kimwe cyo kugerageza mbere yo kugereranya numero yundi mucuruzi.
Itsinda ryubwubatsi ryubaka imbere ya SWE-intebe-yuburyo yashyizwe mububiko bwayo bwite kandi ikemura amatike yo kugenzura niba umukozi uhagaze neza kumugaragaro akora kode yabo.
Abashakashatsi bakoresha SWE-intebe ya Multimodal, aho ibibazo biva muri JavaScript umushinga wimbere kandi akenshi usanga ushiramo amashusho yibiboneka, kugirango ugerageze ubuhanga Python yumwimerere gusa ntishobora gupima.
Nkuko amanota yo hejuru kuri SWE-intebe Yagenzuwe kuzamuka, igipimo gitandukanya sisitemu iyobora neza neza, kandi kwitondera kugana kumurimo mushya hamwe nibibazo bishya, indimi nyinshi zo gutangiza gahunda, imirimo myinshi yintambwe ndende no gutanga raporo yibiciro kandi byizewe. Kwanduza ni impungenge zirambye kuko ububiko bwinkomoko ni rusange, butanga ibipimo byongera imirimo ya vuba. Ku makipe ahitamo ibikoresho, imiyoborere rusange ni intangiriro; ibizamini byo kugerageza kuri codebase zabo hamwe namatike bikomeza kuba ibimenyetso byizewe.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
SWE-intebe ni igipimo gipima niba sisitemu ya AI ishobora gukemura ibibazo nyabyo bya GitHub uhindura codebase ya Python kugirango ibizamini byumushinga byatsinde. Nibyingenzi kuko nimwe mungero zavuzwe cyane kubakozi ba code, kandi amanota yayo yoroshye kuyasoma cyane utazi uko imirimo, scaffolds hamwe na cheque yikizamini ikora.
Buri rugero ruhuza ikibazo nyacyo hamwe nububiko bwifoto, kandi sisitemu igomba kubyara kode ikosora.
Bemeza ko ikibazo gikemutse mubyukuri, kubera ko banyura mugihe gikosowe kiri.
Banyuze mbere na nyuma yo gukosora abantu, bityo birinda gusubira inyuma kwatangijwe na patch yumukozi.
Verified yarekuwe muri 2024 na OpenAI hamwe nabanditsi ba SWE-intebe, bashungura imirimo itarasobanuwe neza cyangwa ifite ibizamini byanga ibisubizo bifatika.
Lite ni 300-yintangarugero ya sisitemu yahisemo kubahendutse kandi irigenga.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
IFEval hamwe nubuyobozi-Gukurikiza Ibipimo
Tekiniki