Xiga xigaHagaha xiga
IFEval iyo Tilmaamaha-Raac Tilmaamaha
Farsamo
HAGAHA Farsamada
SWE-bench iyo Astaamaha Wakiilka Codaynta
SWE-bench waa bartilmaameedka tijaabiya in nidaamka AI uu xallin karo arrimaha dhabta ah ee GitHub isagoo tafatiraaya codebase Python si imtixaannada mashruuca u gaar ah u gudbaan.
Dhageyso hagahanAI-waa laga sheekeeyay · ~3 min
Boggaan4 daqiiqo akhri
Dulmar
Waa muhiim sababtoo ah waa mid ka mid ah cabbiraadaha ugu badan ee la soo xigtay ee wakiilada koodh-samaynta, iyo buundooyinkeeda way fududahay in si xad dhaaf ah loo akhriyo iyada oo aan la garanaynin sida hawlaha, wakiilada qashinka iyo hubinta imtixaanadu u shaqeeyaan.
quusid qoto dheer
SWE-bench was introduced by Carlos Jimenez and colleagues at Princeton in 2023 and published at ICLR 2024. It contains 2,294 task instances drawn from 12 widely used Python repositories, including Django, scikit-learn, sympy, matplotlib and requests. Each instance pairs a real GitHub issue with the pull request that fixed it. The system under test receives the issue text and the repository as it was before the fix, and must produce a patch. Scoring uses the tests from the original pull request. FAIL_TO_PASS tests failed before the human fix and passed after it, so they check that the issue is solved. PASS_TO_PASS tests passed both before and after, so they check that nothing else broke. An instance counts as resolved only if all of these pass, and the headline number is the percentage resolved. At launch, the best models resolved only a few percent of tasks. Two subsets matter. SWE-bench Lite has 300 instances chosen to be cheaper and more self-contained. SWE-bench Verified, released in 2024 by OpenAI with the SWE-bench authors, has 500 instances that human engineers screened to remove vague issue descriptions or tests that would reject valid solutions. Verified is now the most commonly quoted split. Scores reflect a model plus its scaffold, the program that lets it browse files, run commands and edit code. SWE-agent from Princeton and the Agentless pipeline, which localizes, repairs and validates without an open-ended agent loop, are well-known examples. The same model can score very differently in different scaffolds. Common misreadings: the original set is Python-only and covers a dozen repositories; passing tests does not guarantee maintainable code; the repositories and fixes are public, so contamination is possible; and multi-attempt results are not comparable to single-attempt ones. Related benchmarks such as SWE-bench Multilingual, SWE-Lancer and Terminal-Bench probe other languages, freelance tasks and command-line work.
Dhaqangelinta Adduunka-dhabta ah
Hawshu waxay wakiilka u dhiibtaa arrin Django ah oo sifaynaysa hab-dhaqan khaldan iyo hubinta kaydka ballanta ka hor inta aan la hagaajin; Wakiilku waa inuu qoraa balastar, ka dibna lagu qiimeeyay imtixaanada in hagaajinta asalka ah ee bini'aadamku sameeyay.
Iibiyuhu waxa uu soo sheegaa dhibcaha la xaqiijiyay ee SWE-bench isagoo isticmaalaya qalabkiisa wakiilkiisa iyo isku dayo badan oo hawl kasta ah; iibsaduhu waa inuu xaqiijiyaa inay tahay natiijo hal-isku-day ah ka hor inta aan la barbar dhigin lambarka iibiyaha kale.
Kooxda injineeradu waxay dhistaa qaabka SWE-banch-ka ee gudaha ee laga soo bilaabo kaydadkeeda gaarka ah iyo tigidhada la xaliyay si loo hubiyo in wakiilka si fiican u darajeeyay si cad uu dhab ahaantii uga shaqeeyo koodka.
Cilmi-baarayaashu waxay isticmaalaan Multimodal-ka SWE-bench, halkaas oo arrimuhu ka yimaadaan mashaariicda hore ee JavaScript oo inta badan ay ku jiraan sawir-qaadista cayayaanka muuqaalka ah, si loo tijaabiyo xirfadaha Python-kaliya asalka ah ma cabbiri karo.
Mustaqbalka SWE-bench iyo Astaamaha Wakiilka Codaynta
Sida buundooyinka ugu sarreeya ee SWE-bench ee la xaqiijiyay kor u kaca, bartilmaameedku wuxuu u kala saaraa nidaamyada hogaaminta si aan cad ahayn, feejignaantuna waxay u socotaa hawlo cusub oo arrimo cusub leh, luqado badan oo barnaamij ah, shaqo tilaabo badan leh oo dheer iyo ka warbixinta qiimaha iyo isku halaynta. Wasakhdu waa walaac waara sababtoo ah meelaha laga helo waa dad caam ah, kuwaas oo door bida jaangooyooyinka ku daraya hawlihii dhawaa. Kooxuhu waxay dooranayaan aaladaha, sabuuradaha hogaaminta dadwaynuhu waa bar bilow; wakiilada tijaabinta ee saldhigyadooda codebase iyo tigidhada ayaa weli ah calaamada ugu kalsoonaan karo.
Ku dhaqangelinta
Maxay muhiim u tahay
- Qiimaha iyo miisaaniyada. Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
- Go'aamo cad. Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
- Xakamaynta tayada. Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Iska ilaali
- Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
- Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
- Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Tallaabooyinka xiga
- Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
- Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
- La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
- U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Warqad bilaash ah
Hel warbixinta maalinlaha ah ee AI
Saddex sheeko oo la xaqiijiyay oo AI ah maalin kasta oo shaqo ah, oo lagu qoray Ingiriis fudud. Bilaash had iyo jeer, aan xayaysiis lahayn.
Hal iimayl maalin kasta oo shaqo ah. Ka bax hal gujin. Marna ma iibinno ama ma wadaagno ciwaankaaga.
Is tijaabi
Qaado kediska SWE-bench iyo Coding Agent Benchmarks
Fikrad degdeg ah oo ku saabsan jawaab kasta, iyo shahaado la wadaajin karo oo leh aqoonsi la xaqiijin karo markaad ka gudubto koorso.
Taageer waxbarashada AI bilaashka ah. AI Understanding waa 501(c)(3) aan faa'iido doon ahayn — xayaysiis la'aan, albaab lacag ah ma jiro, had iyo jeer. Samee deeq
Su'aalaha soo noqnoqda
Waa maxay bartilmaameedyada SWE-bench iyo wakiilka coding?
SWE-bench waa bartilmaameedka tijaabiya in nidaamka AI uu xallin karo arrimaha dhabta ah ee GitHub isagoo tafatiraaya codebase Python si imtixaannada mashruuca u gaar ah u gudbaan. Waa muhiim sababtoo ah waa mid ka mid ah cabbiraadaha ugu badan ee la soo xigtay ee wakiilada koodh-samaynta, iyo buundooyinkeeda way fududahay in si xad dhaaf ah loo akhriyo iyada oo aan la garanaynin sida hawlaha, wakiilada qashinka iyo hubinta imtixaanadu u shaqeeyaan.
Waa maxay shaqada SWE-bench waxay weydiisataa nidaamka inuu sameeyo?
Tusaale kastaa waxa uu ku lammaaniyaa arrin dhab ah oo leh sawir-qaadis, nidaamkuna waa in uu abuuraa balastar kood ah oo hagaajiya.
Waa maxay FAIL_TO_PASS imtixaanada?
Waxay xaqiijinayaan in arrinta dhab ahaantii la xaliyay, maadaama ay dhaafaan kaliya marka hagaajintu meesha taal.
Waa maxay doorka PASS_TO_PASS imtixaanada?
Waxay dhaafeen kahor iyo ka dib hagaajinta bini'aadamka, si ay uga ilaaliyaan dib-u-celinta ay soo bandhigtay balastar wakiilka.
Waa maxay SWE-banch la Xaqiijiyay?
Xaqiijiyay waxaa sii daayay 2024 by OpenAI oo ay la socdaan qorayaasha SWE-bench, shaandhaynta hawlaha aan la cayimin ama laga helay imtixaano diidaya xalal sax ah.
Immisa xaaladood ayaa ku jirta SWE-bench Lite?
Lite waa qayb-hoosaadyo 300-marsal ah oo loo doortay inay ka raqiisan yihiin oo ay ka kooban yihiin.
Sii wad waxbarashada
Tilmaamaha la xidhiidha
Tilmaamayaal badan ayaa loo doortay mawduucan