Byagenze bite
Urupapuro rwa arXiv rutangiza TrustDABench, igipimo cyo gusuzuma niba imiterere yindimi nini ishobora gusesengura byimazeyo amakuru yubatswe nkimpapuro zerekana na dosiye ya CSV. Abanditsi bakoze ingero 2,340 zagenzuwe n'abantu kuva ku bwoko 19 bw'impinduka kandi basuzuma LLM umunani. Uru rupapuro ruvuga ko akenshi icyitegererezo cyatanze ibisubizo byumvikana ariko bidashyigikiwe, bikabura ibimenyetso bivuguruzanya kandi bigahindura imyanzuro yabo iyo imiterere yimeza cyangwa umubano wambukiranya imipaka byahinduwe.
Urupapuro rwibanze rwibanze ni uko isesengura-ryuzuye-ryubatswe-ryamakuru risaba ibirenze gukora urutonde rwibikorwa. Irasobanura kwizerwa hafi yinzira yemewe kuva ikibazo cyumukoresha kugeza kubimenyetso bifatika. Ibyo gushushanya biganisha ku bizamini bibiri: niba LLM yanze gusubiza cyangwa igasaba ibisobanuro mugihe nta nzira yemewe ihari, kandi niba ikomeza isesengura ryukuri mugihe ibimenyetso bimwe bigaragajwe muburyo butandukanye. Inkomoko irerekana ibi bizamini nkibipimo ngenderwaho byo kwizerwa no gukomera.
TrustDABench yubatswe mugutangirira kubimenyetso-inzira yo kureba no gukoresha 19 bakora perturbation. Abanditsi bavuga ko abo bakoraga bahise babinyuza mu gisekuru cya Agentic-LLM, bitanga ingero 2,340 abantu bagenzuye. Ibisobanuro ntabwo byerekana abakora 19 bose cyangwa ngo bisobanure protocole yo kugenzura birambuye. Irerekana ko igipimo cyagenewe kumenyekanisha impinduka zagenzuwe kubikorwa byubatswe aho gukusanya gusa ibibazo nibisubizo bisanzwe. Abanditsi basuzumye LLMs umunani bahagarariye, ariko abstract ntabwo igaragaza sisitemu zose uko ari umunani cyangwa ngo basobanure ibyifuzo byabo, ibikoresho byinjira cyangwa ibidukikije.
Umutwe wibisubizo ni intege nke mubikorwa byombi. Ibisubizo byiza byizewe byavuzwe ni impuzandengo ya MRS ya 24.21%, yagezweho na GPT-5.5. Ibisubizo byiza byavuzwe neza ni impuzandengo ya ASR ya 9,10%, yagezweho na Claude-Sonnet-5. Inkomoko ntisobanura amagambo ahinnye muri abstract, kubara kwabo nicyerekezo nyacyo bigomba kwemezwa mumpapuro yuzuye mbere yo kugereranya imibare nibindi bipimo. Abanditsi baranga kunanirwa nka gahunda: icyitegererezo ntigishobora kumenya ibimenyetso bivuguruzanya, akenshi bikomeza binyuze munzira zisesengurwa ariko zidashyigikiwe, kandi zigakomeza kumva impinduka zirimo imipaka yo kwitegereza cyangwa umubano wambukiranya imipaka.
Ibisobanuro birambuye: arxiv.org ↗
Impamvu ari ngombwa
LLMs ikoreshwa cyane mugusobanura amakuru yubucuruzi, siyanse nubuyobozi, aho igisubizo cyiza gishobora guhisha inzira yisesengura itemewe. TrustDABench yibanze ku kumenya niba icyitegererezo kimenya igihe ibimenyetso bidahagije kandi niba bigera ku mwanzuro umwe mugihe amakuru ahwanye ahagarariwe ukundi. Ibisubizo byatangajwe byerekana ko gutambutsa amakuru asanzwe yisesengura bidashobora kwerekana ibitekerezo byizewe kumeza.
Ikibazo gifatika ni ikinyuranyo hagati yigisubizo gishobora gutangwa nigisubizo gishyigikiwe namakuru. Icyitegererezo kirashobora gukurikira urutonde rwurupapuro cyangwa ibikorwa byameza mugihe utangirira kumurongo utari wo, wirengagije kwivuguruza cyangwa kurenga umubano ibimenyetso bidafite ishingiro. Muri ibyo bihe, kurangiza neza ntabwo byemeza umwanzuro wemewe. KwizeraDABench ningaruka kuko ireba iryo tandukaniro mu buryo butaziguye aho gufata igisubizo icyo ari cyo cyose gisomwa cyangwa cyatanzwe muburyo bwa gihamya nkibimenyetso byokwizerwa.
Ibi bifite akamaro aho abantu bakoresha LLMs kugirango bavuge muri make cyangwa basesengure amakuru yubatswe. Inkomoko ivuga amazina yurupapuro, dosiye ya CSV nandi makuru yubatswe, ariko ntabwo yanditse inyandiko yoherejwe mubikorwa byinganda cyangwa ngo yerekane ibyangiritse kwisi. Ibipimo byerekana intege nke nyamara byerekana ikibazo gifatika cyo gusubiramo: abakoresha barashobora gukenera kugenzura gusa umubare wanyuma cyangwa ibisobanuro gusa, ariko kandi nibisobanuro, imbibi zimbonerahamwe nubusabane icyitegererezo cyakoreshejwe. Ibyavuye mu mpapuro bishyigikira ubwo bwitonzi nk'umwanzuro w'ubushakashatsi, ntabwo ari igipimo cyagenzuwe cyigenga cya buri sisitemu yoherejwe.
Ibisubizo kandi birwanya icyitegererezo cyoroshye-cyerekana ibitekerezo. GPT-5.5 yageze ku manota akomeye yatangajwe yo kwizerwa, mu gihe Claude-Sonnet-5 yatsindiye amanota akomeye yatangajwe; abstract ntabwo ivuga ko sisitemu yari nziza kuri buri gikorwa cyangwa ko ibipimo bipima imyitwarire imwe. Icyitegererezo rero gishobora kugaragara gikomeye kurwego rumwe mugihe gisigaye cyoroshye kurundi. Inshingano yagutse, nkuko impapuro zibivuga, ni uko isesengura ryizewe ryakozwe-risaba ibimenyetso byombi byerekana imipaka no kwerekana ibitekerezo-bidahinduka, ntabwo ari ubushobozi bukomeye ku mirimo isanzwe.
Uburyo bukoreshwa: Uburyo bukora
Shakisha ikoranabuhanga ryihishe inyuma yiri terambere.
What is a common training objective for an autoregressive language model?
Ibyo kureba
Urupapuro rwerekana ibintu bibiri byihutirwa: kumenya neza imipaka yibimenyetso no gutekereza bikomeza guhagarara neza kumeza atandukanye. Inkomoko ivuga ko code hamwe namakuru bihari, ariko ntibisobanura niba igipimo cyerekana imibare nyayo yisi, uburyo moderi umunani zigereranya mubikorwa bitandukanye, cyangwa niba intege nke zavuzwe zikomeza nyuma yo kunoza intego. Gukurikirana akazi bigomba kugerageza imibare yo hanze, ibikorwa byakazi hamwe nimyitwarire yicyitegererezo nyuma ya mituweli.
Ikintu cya mbere ugomba kureba ni ukumenya niba ibipimo ngenderwaho kode namakuru bifasha kubyara byigenga. Inkomoko ivuga ko "Code & Data" irahari, ariko inyandiko yatanzwe ntabwo ikubiyemo umurongo wububiko, uruhushya rwo kugena ibipimo, imiterere yimirimo cyangwa ibisobanuro byerekana amanota. Ibyo bisobanuro bizagaragaza uburyo byoroshye abashakashatsi bashobora kugenzura ibyangiritse, kugenzura igenzura ryabantu no kumva icyo indangagaciro za MRS na ASR zipima. Kugeza igihe ayo makuru azasuzumwa, ijanisha ryatangajwe rigomba gufatwa nkibisabwa muri iyi progaramu imwe aho kugereranya ibigereranyo byakozwe.
Ikibazo cya kabiri nikibazo cyo hanze. TrustDABench ikubiyemo ingero zagenzuwe n'abantu, ariko ibivugwa ntibivuga uburyo izo ngero zisa neza n'urupapuro rwerekana imitegekere idahwitse, imiyoboro ihindagurika cyangwa isesengura ryakozwe hakoreshejwe ibikoresho byo hanze. Ntabwo kandi itangaza niba ibisubizo bitandukanijwe nubunini bwamakuru, indangarugero, urwego rudasobanutse cyangwa ubwoko bwimbonerahamwe. Gukurikirana isuzuma ryamakuru yigenga hamwe nakazi gafatika byafasha kumenya niba uburyo bwo gutsindwa bwatanzwe bwihariye kubipimo cyangwa bisanzwe mubikoreshwa bifatika.
Hanyuma, abashakashatsi n'ababitanga bagomba kugerageza kugabanya aho kongeramo amanota menshi. Uru rupapuro rugaragaza ibisobanuro, kwangwa, gutahura amakimbirane no gutuza kumiterere yimbonerahamwe nkimyitwarire yingenzi, ariko inkomoko ntabwo itangaza ko hari intervention ibateza imbere. Intambwe zingirakamaro zikurikira zirimo gusuzuma ibimenyetso byerekana neza ibimenyetso, kugenzura hagati, kugenzura ibintu bivuguruzanya no gusuzuma abantu kurwanya ibintu bimwe. Inkomoko nayo isiga ifunguye uburyo moderi yitwara mugihe amakuru yukuri adafite igisubizo, mugihe ibisobanuro byinshi bishoboka, cyangwa mugihe umukoresha ahatira sisitemu gukomeza nubwo yabuze inkunga.