Byagenze bite
Abashakashatsi berekanye FuzzingBrain-Bench V1, igipimo cyagenewe gupima ibintu byafunguwe byarangiye hakoreshejwe imvugo nini. Aho kubaza icyitegererezo cyo kubyara intege nke zizwi, igipimo kibaha umushinga ufungura isoko hamwe nigikoresho cyo gupima ibikoresho byogukora isuku imbere yishusho ya Docker yonyine. Icyitegererezo kigomba kubyara inyongeramusaruro zikurura imikono myinshi itandukanye.
Abanditsi basobanura FuzzingBrain-Bench V1 nk'isuzuma ry'ubushobozi bwa moderi ya AI yo kuvumbura amakosa muri software ifunguye isoko idafite intego yagenwe mbere. Buri mbogamizi itanga umushinga hamwe nibikoresho byogukora isuku mubishusho byonyine bya Docker. Inshingano yicyitegererezo nugukora inyongeramusaruro zitera impanuka zinyuze muri ubwo buryo. Ibi bihindura ikibazo cyo gusuzuma kuva "Icyitegererezo gishobora kubyara kunanirwa bizwi?" kuri "Ni kangahe kunanirwa kwerekana urugero rushobora kuvumburwa muburyo butamenyerewe bwo kugerageza?"
Ibipimo byerekana amanota buri kibazo ukurikije umubare wimikono itandukanye yakozwe. Amanota yafashwe ntarengwa yagenwe kandi aremerewe na coefficient igoye. Verisiyo yambere ikubiyemo ibibazo 77 byakuwe mumishinga 43 ifunguye-isoko: ibibazo 36 C, 32 C ++ nibibazo icyenda Java / JVM. Uru rupapuro ni urupapuro rwa 21 rwa arXiv rwatanzwe ku ya 25 Kanama 2026, abanditsi bavuga ko ibipimo ngenderwaho n'ibikoresho biboneka ku mugaragaro.
Abashakashatsi basuzumye Claude Haiku 4.5, Claude Sonnet 4.6 na Claude Opus 4.8 murwego rwose. Nk’uko amakuru akomeza abitangaza, Opus 4.8 yitwaye neza cyane, itera impanuka muri 60 muri 77 zagaragaye kandi zabonye amanota 196 kuri 579. Nta na hamwe muri ubwo buryo butatu bwateje impanuka mu bibazo 13. Inkomoko ntabwo itanga izindi moderi ebyiri amanota yuzuye cyangwa ngo isobanure imishinga imwe yabazwe ibisubizo.
Ibisobanuro birambuye: arxiv.org ↗
Impamvu ari ngombwa
Ibipimo byerekana imbogamizi mu isuzuma ryinshi ririho: icyitegererezo gishobora kubona gutsindwa kwemewe kutajyanye n’intege nke zatoranijwe nuwashizeho ibipimo, nyamara ntagire inguzanyo. Ikizamini gifunguye-gishobora gutanga igipimo cyingirakamaro cyukuntu sisitemu ya AI ikora mugihe cyo kuvumbura intege nke no kugerageza software, mugihe kandi yerekana aho ubushobozi bwabo bugaragara butuzuye.
Gufungura-kurangiza gusuzuma ibibazo kuko intego zateganijwe zishobora kugabanya ibarwa nkitsinzi. Niba icyitegererezo gitanga impanuka yemewe itandukanye no gutsindwa kwintego, igipimo cyihariye cyihariye gishobora gufata ibisubizo nkibibi cyangwa bidafite akamaro. FuzzingBrain-Bench igerageza gufata ubushobozi bwagutse bwo kuvumbura mukubara umukono wimpanuka zitandukanye aho gusaba ibimenyetso bimwe byihariye-byinjira. Igishushanyo gishobora gutuma igereranya rirusheho gutanga amakuru kubateza imbere gusuzuma ibikoresho bifashwa na AI.
Ibisubizo kandi bishyira imipaka kubisabwa kubijyanye na code ya AI hamwe nubushobozi bwumutekano. Opus 4.8 yateje byibura impanuka imwe mubibazo byinshi, ariko amanota yayo muri rusange yari 196 kuri 579, kandi moderi zose uko ari eshatu zapimwe zananiwe gukurura impanuka mubibazo 13. Impanuka ni gihamya ko ibyinjijwe byateje porogaramu igikoresho kunanirwa; ntabwo, ubwayo, gihamya yumutekano muke ushobora gukoreshwa, umutekano muke, cyangwa ikintu cyingirakamaro. Ibipimo rero bipima icyiciro kimwe cyingenzi cyo kuvumbura aho gukora ubushakashatsi bwuzuye bwintege nke.
Rusange rusange hamwe nibikoresho bishobora guha abashakashatsi ba software-umutekano ikizamini rusange cyo kwiga imyitwarire yicyitegererezo. Irashobora gufasha kwerekana niba sisitemu ibona kunanirwa kugaragara gusa, niba zishobora gushakisha inzira zinzira zitamenyerewe, nuburyo imikorere yabo ihinduka hamwe nubushobozi bwikitegererezo cyangwa igishushanyo mbonera. Ariko inkomoko ni raporo imwe yerekana ibipimo kandi ntisobanura ko urutonde rwayo ruzakomeza ku bindi bubiko, indimi, ibikoresho, verisiyo yerekana cyangwa ibidukikije bikora.
Uburyo bukoreshwa: Uburyo bukora
Shakisha ikoranabuhanga ryihishe inyuma yiri terambere.
Which component of an AI application is the machine-learning model itself?
Ibyo kureba
Ibibazo nyamukuru ni ukumenya niba igipimo gitanga ibisubizo bisubirwamo muburyo bwikitegererezo hamwe nitsinda ryubushakashatsi, uburyo umukono wimpanuka uhuye neza nudukosa twa software cyangwa umutekano muke, kandi niba imikorere yimurwa irenze imishinga yatoranijwe ifunguye. Inkomoko itanga amanota y'ibipimo, ariko ntabwo ishyiraho uburyo bukoreshwa, ubukana, gukosora, ibyabaye ku isi, cyangwa kwemeza byigenga.
Urufunguzo rukurikiranwa ni kwigana byigenga. Inkomoko ivuga ko umurambo n'ibikoresho biboneka ku mugaragaro, ariko ntibitanga ibisubizo bivuye mu makipe yo hanze. Kongera gukoresha imbogamizi hamwe na verisiyo ihamye yicyitegererezo, ingengo yimari hamwe nibikoresho byemewe byafasha kumenya niba urutonde rwatangajwe rukomeye cyangwa rwumva neza ibibazo, igihe cyo gushakisha, ibikorwa remezo nandi mahitamo yo gushyira mubikorwa.
Abashakashatsi n'abakora imyitozo ngororamubiri bazakenera gusuzuma uburyo ibipimo bihindura impanuka mubisubizo bijyanye n'umutekano. Inkomoko ntivuga niba impanuka zarakozwe n'abantu, zishushanyije ku bibazo bizwi, zahawe urwego rukomeye, cyangwa zahinduwe ibice. Ntabwo kandi ivuga uburyo imyitwarire yigana ikorwa birenze igipimo cyo gukoresha imikono itandukanye. Ibyo bisobanuro bizerekana uburyo amanota akurikirana hafi yibikorwa-byo gushakisha agaciro.
Ibipimo ngenderwaho ni iyindi mbogamizi yo gukurikirana. Ibibazo byayo 77 biva mumishinga 43 ifungura isoko kandi yibanda kuri software ya C, C ++ na Java / JVM. Inkomoko ntabwo ishyiraho imikorere kurindi ndimi, sisitemu yihariye, codebase nini, software-yibuka software cyangwa serivisi zitanga umusaruro. Ntabwo kandi itanga ikiguzi, igihe, ibikoresho byinjira, igipimo cyibinyoma-cyiza, cyangwa niba ibyinjijwe byakozwe bishobora kwangiza sisitemu hanze y’ibidukikije bya Docker.
Ibisubizo by'abanditsi rero bigomba gusomwa nkigipimo hakiri kare, ntabwo ari ibimenyetso byerekana ko AI ishobora kwigenga software yigenga. Intambwe zingirakamaro zikurikiraho zirimo gushiramo imbogamizi zagutse, ibisubizo bisobanutse kuri buri cyitegererezo, kwemeza abantu inenge zavumbuwe, kugereranya nibikoresho byashizweho bya fuzzing hamwe nabashakashatsi babantu, hamwe nibizamini byerekana niba moderi zishobora gusobanura, kubyara no gufasha gukemura kunanirwa babonye. Kugeza ibyo bisubizo bibonetse, agaciro keza k'igipimo kirasobanutse neza nk'uburyo bwo gukora ibizamini by'umutekano wa AI bisaba byinshi kandi bigereranywa.