Ubuyobozi bwa tekiniki

Intebe ya SWE hamwe na Coding Agent Ibipimo

SWE-intebe ni igipimo gipima niba sisitemu ya AI ishobora gukemura ibibazo nyabyo bya GitHub uhindura codebase ya Python kugirango ibizamini byumushinga byatsinde.

Umva iki gitaboAI-byavuzwe · ~3 min

Kuriyi page4 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Kazoza ka SWE-intebe hamwe na Coding Agent Ibipimo
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Nibyingenzi kuko nimwe mungero zavuzwe cyane kubakozi ba code, kandi amanota yayo yoroshye kuyasoma cyane utazi uko imirimo, scaffolds hamwe na cheque yikizamini ikora.

Kwibira cyane

Intebe ya SWE yatangijwe na Carlos Jimenez na bagenzi be i Princeton mu 2023 kandi isohoka kuri ICLR 2024. Irimo ingero 2,294 zashizwe mu bubiko bwa Python bukoreshwa cyane, harimo Django, scikit-kwiga, impuhwe, matplotlib n'ibisabwa. Buri rugero ruhuza ikibazo nyacyo cya GitHub hamwe no gukurura icyifuzo cyakosoye. Sisitemu iri kugeragezwa yakira ikibazo cyibisobanuro hamwe nububiko nkuko byari bimeze mbere yo gukosorwa, kandi bigomba kubyara patch. Gutanga amanota akoresha ibizamini bivuye kubisabwa gukurura. Ibizamini bya FAIL_TO_PASS byananiranye mbere yo gukosora abantu bikanyura nyuma yabyo, bityo bagenzura ko ikibazo cyakemutse. Ibizamini bya PASS_TO_PASS byatsinze mbere na nyuma, bityo bareba ko ntakindi cyacitse. Urugero rubarwa nkuko byakemuwe gusa niba ibyo byose byatsinzwe, numutwe wumutwe nijanisha ryakemuwe. Mugutangiza, icyitegererezo cyiza cyakemuye bike ku ijana byimirimo. Ibice bibiri byingenzi. SWE-intebe Lite ifite ingero 300 zahisemo kuba zihendutse kandi zonyine. Intebe ya SWE Yagenzuwe, yasohowe mu 2024 na OpenAI hamwe n’abanditsi ba SWE-intebe, ifite inshuro 500 abashakashatsi b’abantu bagenzuye kugira ngo bakureho ibisobanuro bidasobanutse cyangwa ibizamini byanga ibisubizo bifatika. Kugenzurwa ubu nibisanzwe byavuzwe gutandukana. Amanota yerekana icyitegererezo wongeyeho scafold yayo, porogaramu ireka ikareba amadosiye, ikoresha amategeko kandi igahindura code. SWE-umukozi wo muri Princeton hamwe numuyoboro wa Agentless, uhindura, ugasana kandi ukemezwa udafunguye-abakozi bafunguye, ni ingero zizwi. Icyitegererezo kimwe gishobora gutsinda amanota atandukanye muburyo butandukanye. Gusoma nabi: ibice byumwimerere ni Python-gusa kandi ikubiyemo ububiko icumi; gutsinda ibizamini ntabwo byemeza kode ishobora kugumaho; ububiko no gukosora ni rusange, bityo kwanduza birashoboka; n'ibisubizo byinshi-byo kugerageza ntabwo bigereranywa nimwe-igerageza. Ibipimo bifitanye isano nka SWE-intebe Indimi nyinshi, SWE-Lancer na Terminal-Bench ikora izindi ndimi, imirimo yigenga hamwe nakazi kayobora.

Gushyira mu bikorwa Isi

Igikorwa giha umukozi ikibazo cya Django gisobanura imyitwarire itari yo hiyongereyeho no kugenzura ububiko bwiyemeje mbere yo gukosora; umukozi agomba kwandika agapapuro, hanyuma agasuzumwa nibizamini ko umwimerere wambere wumuntu wakoze pass.

Umucuruzi atanga raporo ya SWE-intebe Yagenzuwe amanota ukoresheje ibikoresho byayo bwite kandi igerageza inshuro nyinshi kuri buri gikorwa; umuguzi agomba kwemeza niba ari igisubizo kimwe cyo kugerageza mbere yo kugereranya numero yundi mucuruzi.

Itsinda ryubwubatsi ryubaka imbere ya SWE-intebe-yuburyo yashyizwe mububiko bwayo bwite kandi ikemura amatike yo kugenzura niba umukozi uhagaze neza kumugaragaro akora kode yabo.

Abashakashatsi bakoresha SWE-intebe ya Multimodal, aho ibibazo biva muri JavaScript umushinga wimbere kandi akenshi usanga ushiramo amashusho yibiboneka, kugirango ugerageze ubuhanga Python yumwimerere gusa ntishobora gupima.

Kazoza ka SWE-intebe hamwe na Coding Agent Ibipimo

Nkuko amanota yo hejuru kuri SWE-intebe Yagenzuwe kuzamuka, igipimo gitandukanya sisitemu iyobora neza neza, kandi kwitondera kugana kumurimo mushya hamwe nibibazo bishya, indimi nyinshi zo gutangiza gahunda, imirimo myinshi yintambwe ndende no gutanga raporo yibiciro kandi byizewe. Kwanduza ni impungenge zirambye kuko ububiko bwinkomoko ni rusange, butanga ibipimo byongera imirimo ya vuba. Ku makipe ahitamo ibikoresho, imiyoborere rusange ni intangiriro; ibizamini byo kugerageza kuri codebase zabo hamwe namatike bikomeza kuba ibimenyetso byizewe.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the SWE-bench and Coding Agent Benchmarks quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

Niki SWE-intebe hamwe na Coding Agent Ibipimo?

SWE-intebe ni igipimo gipima niba sisitemu ya AI ishobora gukemura ibibazo nyabyo bya GitHub uhindura codebase ya Python kugirango ibizamini byumushinga byatsinde. Nibyingenzi kuko nimwe mungero zavuzwe cyane kubakozi ba code, kandi amanota yayo yoroshye kuyasoma cyane utazi uko imirimo, scaffolds hamwe na cheque yikizamini ikora.

Niki SWE-intebe isaba sisitemu gukora?

Buri rugero ruhuza ikibazo nyacyo hamwe nububiko bwifoto, kandi sisitemu igomba kubyara kode ikosora.

Ni ibihe bizamini bya FAIL_TO_PASS?

Bemeza ko ikibazo gikemutse mubyukuri, kubera ko banyura mugihe gikosowe kiri.

Ni uruhe ruhare rw'ibizamini bya PASS_TO_PASS?

Banyuze mbere na nyuma yo gukosora abantu, bityo birinda gusubira inyuma kwatangijwe na patch yumukozi.

Intebe ya SWE Yagenzuwe Niki?

Verified yarekuwe muri 2024 na OpenAI hamwe nabanditsi ba SWE-intebe, bashungura imirimo itarasobanuwe neza cyangwa ifite ibizamini byanga ibisubizo bifatika.

Ni kangahe ziri muri SWE-intebe Lite?

Lite ni 300-yintangarugero ya sisitemu yahisemo kubahendutse kandi irigenga.