Byagenze bite
Abashakashatsi berekanye RestoreBench, igipimo cyo gusuzuma niba sisitemu ya AI ishobora gusuzuma no gukemura ibibazo bituruka ku mashanyarazi adashobora guhura. Ibipimo ngenderwaho bisuzuma ibishushanyo bitatu bya sisitemu - kuganira, abakozi bonyine hamwe na sisitemu nyinshi-zikoresha amashanyarazi abiri, hamwe na 46 kuri gride. Buri rubanza rusaba ibikorwa bimwe cyangwa byinshi byo gukosora, ukurikije impapuro.
Uru rupapuro rwashyikirijwe arXiv ku ya 31 Kanama, rutangiza RestoreBench nk'igipimo ngenderwaho cy’indimi nini-y’intangarugero ikora ku manza zidahuye. Abanditsi basobanura umurimo nkugusaba ubuhanga bwubuhanga, kugerageza no gufata ibyemezo mumwanya muto wibikorwa. Mubipimo, sisitemu ya AI igomba gusobanura ibisubizo bigereranijwe kandi igahitamo guhitamo ibikorwa byo gukosora bigamije kugarura ubumwe. Inkomoko irerekana ibi nkibisobanuro bitagenzuwe byabakozi ba LLM, aho kuba sisitemu yoherejwe cyangwa ibicuruzwa byuzuye byuzuye.
RestoreBench igereranya ibyubatswe bitatu: chatbot, umukozi umwe na sisitemu ya agent-benshi. Ibisobanuro ntibisobanura ishyirwa mubikorwa ryimbere yubwubatsi, vuga imiterere yururimi rwageragejwe cyangwa gusobanura uburyo sisitemu yakira ibitekerezo bivuye mubidukikije. Ivuga ko igipimo cyerekana ibidukikije byigana, umwanya wo kwitegereza, umwanya wibikorwa hamwe nugupima ibipimo. Ibyo bisobanuro bigamije gukora ubushakashatsi bwororoka no guha abashakashatsi urufatiro rusange rwo guteza imbere no kugereranya sisitemu yo guteganya ingufu-sisitemu.
Isuzuma ririmo amashanyarazi abiri na 46 kuri buri gride. Inkomoko ivuga ko buri rubanza rusaba byibuze igikorwa kimwe cyo gukosora kugirango ugarure ubufatanye, ariko ntirondora imanza, kwerekana imiterere ya gride, gusobanura ibikorwa bihari cyangwa gutanga imibare y'ibisubizo mumyandiko yatanzwe. Abanditsi bavuga kandi ko code ihari. Inkomoko rero ishyiraho irekurwa ryibipimo n’ibipimo byavuzwe, ariko ntibisobanura ko sisitemu ya AI yapimwe yakemuye neza imanza cyangwa ko igipimo cyerekana ibintu byose byagaragaye muri sisitemu y’amashanyarazi nyayo.
Ibisobanuro birambuye: arxiv.org ↗
Impamvu ari ngombwa
Igikorwa cyibanze ku isuzuma rya AI kumurimo wihariye wubwubatsi aho sisitemu igomba gusobanura ibisubizo bigereranijwe, igateganya kandi igakorera mubikorwa bibujijwe. Mu gusobanura ibidukikije, kwitegereza, ibikorwa no gupima ibipimo, abanditsi bavuga ko RestoreBench itanga ishingiro ryororoka ryo kugereranya sisitemu ya AI igenewe igenamigambi rya sisitemu.
Ubusobanuro bufatika buva muburyo bwicyemezo igipimo cyagenewe gupima. Igikorwa ntabwo kigarukira gusa mugutanga igisubizo cyinyandiko: umukozi agomba gusobanura ibisubizo bivuye mu kwigana, impamvu yo gutabara gushoboka no gukora muburyo bwateganijwe bwo guhitamo. Ibyo bituma igipimo kijyanye nicyiciro cyibikorwa bya AI aho akamaro k'icyitegererezo gaterwa nurukurikirane rw'ibyemezo n'ingaruka z'ibyo byemezo, aho gushingira ku gisubizo kimwe cyaciwe n'amagambo cyangwa ukuri.
Ikizamini cyororoka gishobora gufasha gutandukanya ibirego bijyanye nubushobozi bwa injeniyeri yubukorikori hamwe n’imyiyerekano yubatswe hafi y'urugero rumwe rwatoranijwe. Imiterere yavuzwe na RestoreBench iha abashakashatsi ibidukikije bisobanutse, kwitegereza, ibikorwa na metrics, mugihe imanza 92 zitanga ikizamini kinini kuruta imyigaragambyo imwe. Ugereranije chatbot, umukozi umwe hamwe nubushakashatsi bwinshi bushobora nanone gusobanura niba guhuza ibikorwa byongera kunoza ibisubizo kuriki gikorwa cyihariye. Izi ninyungu zishobora guterwa nigishushanyo mbonera; inkomoko ntabwo itanga ibimenyetso byerekana ko imyubakire imwe isumba izindi.
Ikiganiro gifite akamaro rusange kandi gifatika kuko igipimo cyibanda ku mikorere ya sisitemu yo gutegura no gukora, ahantu aho ibikorwa bitari byo bishobora kugira ingaruka zirenze inyandiko y’icyitegererezo. Muri icyo gihe, impapuro ni integuza kandi isoko yatanzwe nta kwemeza kwigenga, kohereza ibikorwa, kugereranya-injeniyeri cyangwa gusesengura umutekano. Ibipimo birashobora gutuma isuzuma rirushaho kuba indangagaciro uterekanye ko umukozi wa AI agomba kugenzura ibikorwa remezo nyabyo. Umusanzu wacyo ako kanya ni urwego rwo kugerageza icyo kibazo, ntabwo ari ibimenyetso byerekana ko kugarura ubwigenge byiteguye.
Uburyo bukoreshwa: Uburyo bukora
Shakisha ikoranabuhanga ryihishe inyuma yiri terambere.
An agent must create a draft calendar event for Tuesday at 2 p.m. Which evidence would establish the requested result?
Ibyo kureba
Inkomoko yatanzwe ntabwo itanga raporo igereranya, kumenya LLMs yasuzumwe, gusobanura ibikorwa byo gukosora birambuye cyangwa kwerekana ko sisitemu iyo ari yo yose yiteguye gukoreshwa. Ibibazo byingenzi bikurikiraho ni ukumenya niba abakozi bagarura guhuza kwizerwa, ni kangahe bahitamo ibikorwa bidafite umutekano cyangwa bidafite ingaruka, niba ibisubizo bihurira hejuru ya gride zombi kandi niba injeniyeri zishobora kugenzura ibitekerezo no gutabara.
Ikibazo cya mbere kidakemutse ni imikorere. Ibisobanuro bivuga ko igipimo gisuzuma LLMs nyinshi nubwubatsi, ariko ntabwo itanga amanota, igipimo cyo gutsinda, uburyo bwo gutsindwa cyangwa kugereranya. Impapuro zuzuye cyangwa kode iherekeza byakenera kwerekana inshuro nyinshi sisitemu igarura guhuza, ibikorwa bingana iki basaba kandi niba bashobora kumenya mugihe intervention yatanzwe idakorwa. Hatariho ibisubizo, isoko ntishobora gushyigikira umwanzuro werekana icyitegererezo cyangwa ubwubatsi bukora neza.
Ibipimo ngenderwaho nabyo bikeneye kugenzurwa. Imiyoboro ibiri gusa hamwe na 46 kuri gride yamenyekanye, kandi inkomoko ntisobanura uburyo bahagarariye, uburyo imanza zigoye cyangwa niba ibintu birimo ibintu bidasanzwe. Isuzuma ry'ejo hazaza rigomba gusuzuma niba ibyagezweho byimurirwa mubindi bikoresho bya gride nibikorwa. Byaba byiza kandi umenye niba umwanya wibikorwa ukuyemo ibikorwa bidashoboka kumubiri nuburyo igipimo gikemura ibibazo aho inzira zirenze imwe ikosora iboneka.
Hanyuma, ibibazo byo kohereza bikomeza gufungura. Inkomoko ntivuga niba umukozi uwo ari we wese yaba yarahujwe na sisitemu nzima-sisitemu ikora, niba umuntu agomba kwemeza ibikorwa byose cyangwa uburyo ibyemezo bya sisitemu bishobora kugenzurwa. Ntabwo kandi itanga imbaraga zo kwitegereza nabi, amakuru atuzuye cyangwa guhindura sisitemu. Abanditsi bavuga ko RestoreBench ishyigikira ubushakashatsi mu igenamigambi no mu bikorwa bigomba rero gusomwa nk'itangazo ryerekeye igipimo cyagenewe gukoreshwa, atari nk'ikimenyetso cyo kwizerwa mu mikorere.