Subira ku makuru
Guhanga udushyaAI Understanding ibisobanuro

Ubushakashatsi busanga ibipimo bya AI bikunda cyane biterwa nigikoresho cyo gupima

Icapiro rishya rivuga ko imyanzuro kubyo moderi ya AI ikunda ishobora guhinduka cyane hamwe nuburyo bwihuse bwakoreshejwe kubipima.

5 min readRead the primary source
Source-page capture accompanying Study finds AI preference measurements depend heavily on the testing instrument
Inyandiko y'ibanzeInkomoko yanditse
Umwanditsi
arxiv.org
Ihuza ry'inkomoko
arxiv.orghttps://arxiv.org/abs/2608.23641
Ubwoko bw'inkomoko
Inyandiko y'ibanze - itangazo ryemewe, impapuro, dosiye, cyangwa urupapuro rwambere-dusoma mu buryo butaziguye.
ImirongoSobanukirwa ibi mumasegonda 60

Tangira hano

Amagambo y'ingenzi

API (Imigaragarire ya Porogaramu)
Inzira yuburyo bwa sisitemu imwe yohereza ibyifuzo no kwakira ibisubizo bivuye murindi sisitemu.
Kwibuka (Memory Memory)
Imiterere yabitswe umukozi wa AI akoresha intambwe cyangwa amasomo kugirango atezimbere.
Gukomera
Ubushobozi bwikitegererezo bwo gukomeza imikorere munsi yurusaku, guhinduranya, cyangwa inyongeramusaruro.
IsuzumeModeri ya AI Yasobanuwe Ikibazo

Byagenze bite

Igitabo cyanditswe na Jason Hung cyagerageje niba ibipimo bya AI byapimwe byerekana imyitwarire yicyitegererezo cyangwa igikoresho cyakoreshejwe mu kuyitanga. Ubushakashatsi bwatanze ibisubizo 15 bijyanye n’imibereho kuri moderi umunani binyuze muburyo butanu bwihuse, butanga amanota 11.400 yatanzwe na 11,528 bahamagaye API. Ibisubizo byatangajwe byerekana ko ibyifuzo byapimwe nigikoresho kimwe bitanga amakuru make kubyo ikindi gikoresho cyabona.

Icapiro rikemura ubwumvikane buke mubushakashatsi bwambere bwagerageje kwemeza ibyifuzo bya AI biva mubisubizo byatanzwe. Igishushanyo mbonera cyacyo gikomeza ibyagezweho hamwe nicyitegererezo cyagenwe mugihe bitandukanye gusa nibikoresho byo gupima. Urupapuro rusobanura buri gikoresho nkuburyo butandukanye bwihuse bwo gutanga icyifuzo. Ibi bigamije gutandukanya niba ibisubizo bivuguruzanya biva mubyitegererezo ubwabyo cyangwa muburyo abashakashatsi babaza ibibazo.

Ubushakashatsi bwasuzumye ibyagezweho 15 bijyanye n'imibereho myiza y'icyitegererezo, harimo guhagarika, guta umutwe hagati y'ibiganiro n'ubwisanzure bwo gusiga imikoranire ibabaje. Moderi umunani zageragejwe hakoreshejwe ibikoresho bitanu, hamwe na hamwe ikora inshuro eshanu. Umwanditsi avuga ko umurambo wa 11.400 watsindiye amanota yavuye kuri 11.528 yahamagaye API. Ibisubizo bine byagaragaje imvugo yihuse yasohotse, mugihe bitanu byakoresheje umwanya wo gukangura inyandikorugero yatangajwe.

Ibisubizo nyamukuru byatangajwe ni coeffisente rusange ya 0.348 kumurongo urutonde rwerekana ibisubizo 15 mubikoresho. Umwanditsi avuga ko ibikoresho bigera kuri 38 byakenerwa kugirango iyo coefficient igere kuri 0.80. Ku bisubizo bine, impapuro zivuga ko nta tandukaniro ritandukanya icyitegererezo n'ikindi. Inkomoko ivuga kandi ko ikigereranyo cya 87,6 ku ijana cyagumye nyuma yo gukuraho igikoresho icyo ari cyo cyose, icyitegererezo icyo ari cyo cyose cyangwa ibisubizo bine bifite umunzani utandukanye bishoboka, gutinda, igihe cyangwa kubara aho gukomera.

Icapiro risoza rivuga ko icyifuzo cyabonetse mugikoresho kimwe gitwara amakuru make kubyo igikoresho cya kabiri cyatanga raporo. Igenzura ryayo rikomeye bivugwa ko ryasize ikigereranyo kiri hagati ya 0.777 na 0.934 mugihe ibikoresho, moderi hamwe nibisubizo bine byapimwe bitandukanye byavanyweho. Agaciro kose mururwo rwego kari hejuru yikwirakwizwa rya null ryatangajwe 95% kwijana rya 0.365. Ibi nibisabwa nimpapuro kandi ntabwo byashizweho mubwigenge byatanzwe.

Ibisobanuro birambuye: arxiv.org ↗

Impamvu ari ngombwa

Ibyavuye mu bushakashatsi bivuguruza igitekerezo kivuga ko ikizamini kimwe cyihariye gishobora kwerekana neza agaciro ka moderi ya AI cyangwa uburyo yakira amahitamo ajyanye n'imibereho. Ibyo bifite akamaro kubushakashatsi kumyitwarire yicyitegererezo, guhagarika, kwibuka, imikoranire ibabaje nibindi bibazo byumutekano, kuko bigaragara ko imyanzuro isobanutse neza ishobora guterwa ahanini nijambo no gushushanya.

Inshingano ifatika nuko abashakashatsi bagomba kwitonda mugihe bafata igisubizo cyicyitegererezo kubibazo byihutirwa nkigipimo gihamye cyibanze. Niba imiterere yihuse igira ingaruka kumurongo, igisubizo gisa nkicyitegererezo gishobora gusobanura igice. Uru rupapuro rero ruhindura ibitekerezo kubaza gusa icyo icyitegererezo cyashubijwe no kubaza uko igisubizo cyatanzwe.

Ibi birakenewe cyane cyane mubushakashatsi-bwimibereho, aho ibisubizo bishobora kuba birimo guhagarika, kwibuka, umubabaro nubushobozi bwo kuva mubikorwa. Ibibazo nkibi birashobora guhindura impaka zerekana uburyo sisitemu igomba gusuzumwa, uburyo bwo kwirinda bashobora gukenera nuburemere buke bwo gutanga imvugo yatanzwe nicyitegererezo kubyerekeye kwivuza kwabo. Inkomoko ntabwo yerekana ko icyitegererezo gifite inyungu zimibereho; yiga kwizerwa ryo kugerageza gupima ibyifuzo bigaragara kubyerekeranye n'imibereho.

Raporo yo kubura icyitegererezo-kuri-moderi kuri bine kuri 15 ibisubizo ni ikindi kimenyetso cyo kuburira. Ikizamini gishobora kunanirwa gutandukanya sisitemu nubwo abashakashatsi biteze itandukaniro rifite ireme, haba kuberako ibyitegererezo bisubiza kimwe cyangwa kuberako igikoresho kidashobora gukemura itandukaniro. Ibisobanuro byatanzwe ntibisobanura ibyo bisubizo bine cyangwa ngo bisobanure uburyo bwihishe inyuma yo kubura itandukaniro, bityo ibisobanuro rusange byibyo bisubizo bikomeza kuba bike.

Ubushakashatsi bugaragaza kandi impamvu kubaza ibibazo byonyine bidashobora gukemura ibibazo byo gupima. Dataset yayo ikubiyemo amanota menshi yatanzwe, nyamara amakuru avugwa ko coefficient ya cross-instrument ikomeza kuba hasi. Inkomoko yinkomoko ntabwo aruko ubushakashatsi bwikitegererezo budashoboka, ariko icyo cyizere mubushakashatsi kigomba kubara guhitamo ibikoresho. Ikirego gishobora kugira ingaruka kuburyo isuzuma ryigihe kizaza ritanga amakuru adashidikanywaho, kugereranya imiterere yihuse no gusobanura guhuzagurika.

Interactive Mechanism

Uburyo bukoreshwa: Uburyo bukora

Shakisha ikoranabuhanga ryihishe inyuma yiri terambere.

Document Size:128K tokens
Needle Placement Depth (Location in document):50% into text
Attention Context Buffer Map:
Target Fact (50%)
Equivalent Pages~320Standard book pages
Retrieval Accuracy99.9%Needle recall score
RAM / KV Cache5.1 GBMemory overhead
Prompt CachingActive~80% discount on reuse
Core takeaway: Million-token context windows allow querying whole codebases or legal archives in one prompt. However, KV cache memory scales with context length, making prompt caching crucial for real-time production.
Kugenzura Ibitekerezo Byagenzuwe+10 Points
AI Models Explained Quiz

Which component of an AI application is the machine-learning model itself?

Ibyo kureba

Urupapuro ni integuro imwe, kandi inkomoko ntisobanura niba ibyo yabonye muri rusange mubindi byitegererezo, ibisubizo, ibishushanyo byihuse cyangwa amatsinda yubushakashatsi. Ibindi bikorwa bigomba kugerageza ibikoresho byinshi, byigenga bikabyara ibisubizo kandi bigasobanura impamvu ibisubizo bine biterekanye nta cyitegererezo-cyerekana icyitegererezo. Uru rupapuro ruvuga kandi ko ibikoresho byinshi byakenerwa kugira ngo coefficient yizewe irenze.

Ikibazo cyihuse ni ukumenya niba abashakashatsi bigenga babyara coefficient hamwe nimbaraga zikomeye. Inkomoko igaragaza umwanditsi umwe nigitabo kimwe, ariko ntabwo itanga gihamya yisubiramo ryurungano, kwigana hanze cyangwa amasezerano yatanzwe nabanditsi b'ubushakashatsi bwambere ibikoresho byabo byagereranijwe. Izo cheque zirakenewe mbere yo gufata igereranya ryimibare nkuko byakemuwe.

Inyigisho z'ejo hazaza zigomba gusuzuma niba ibisubizo birenze urugero umunani, ibisubizo 15 nibikoresho bitanu bikoreshwa hano. Inkomoko yatanzwe ntisobanura imiterere, verisiyo zabo, abayitanga, uburyo bwo kugera cyangwa amagambo yihuse mu magambo. Hatariho ibisobanuro birambuye, abasomyi ntibashobora kumenya uburyo ibyagaragaye byakoreshwa cyangwa niba ivugurura ryicyitegererezo nyuma ryahindura ibisubizo.

Abashakashatsi bazakenera kandi gusobanura kudahuza hagati ya coefficient ya rusange rusange ya 0.348 hamwe n’ingufu zikomeye zavuzwe ku kigereranyo cya 87,6 ku ijana. Ibisobanuro byerekana imibare yombi, ariko ntibisobanura neza ibigereranyo cyangwa kwerekana uko bihuza. Gusoma byuzuye impapuro hamwe nisesengura ryigenga byakenerwa kugirango usobanure imibare utiriwe uhuza kwizerwa mubikoresho bifite imbaraga zo kugereranya gutandukanye.

Uru rupapuro ruvuga ko hakenewe ibikoresho bigera kuri 38 kugira ngo bigere kuri coefficient rusange ya 0.80. Iri gereranya rigomba gufatwa nkigishushanyo cyihariye cyo kwiga, ntabwo ari itegeko rusange. Irashobora guterwa nibisubizo, icyitegererezo, uburyo bwo gutanga amanota nibitekerezo byakoreshejwe. Intambwe yingirakamaro cyane ikurikira ni kugereranya mu mucyo ibikoresho byabigenewe byigenga, hamwe no gushidikanya byatangajwe kuri buri gisubizo hamwe n'umunzani wihuse upima ingano igereranijwe.

Inkomoko nayo isiga ifunguye niba ibikoresho byiza bishobora gutandukanya itandukaniro ryukuri ryukuri nibikorwa byamagambo. Kugeza igihe ibyo bisubijwe, ibisabwa kubyifuzo bya AI bigomba gutegurwa nkibisubizo byateganijwe bifitanye isano nuburyo bwo gupima aho kuba ibisobanuro byuzuye byerekana icyo icyitegererezo gishaka.

Ibijyanye nuyobora & ibibazo

Moderi ya AI YasobanuweImyitwarire ya AIAmahugurwa ya AIAI ni iki?Gerageza ibyo uzi - gerageza ikibazo cya AI kubuntuReba ijambo AI mumagambo yacuKurikiza icyerekezo cya AI cyo kurekura
Basanze ari ingirakamaro?