Byagenze bite
Abashakashatsi bakoze ubushakashatsi bwibipimo byifashishwa muburyo busanzwe mururimi runini. Bavuga ko gukuraho izo vectors byangiza cyane imyitozo mbere, nubwo bafite uruhare runini rwibipimo byose. Urupapuro rw'imyumvire ruvuga agaciro kabo cyane cyane kunoza uburyo bwiza kuruta ubushobozi bwo guhagararirwa mububiko bwa Pre-Norm.
Inkomoko ni verisiyo ivuguruye ya progaramu ya arXiv yatanzwe ku ya 26 Gicurasi kandi iheruka gusubirwamo ku ya 28 Kanama 2026. Abanditsi batandatu basuzuma ibice byapimwe, bikaba indangagaciro zishobora kwigwa zifatanije n’ibikorwa bisanzwe bigenwa na LLMs zigezweho. Urupapuro rwibanze kubintu bitanga ibipimo bike cyane ariko birahari mubwubatsi. Icyifuzo cyacyo nyamukuru nuko ibipimo bibarwa ari ubuyobozi bubi kumahugurwa yibigize.
Abanditsi batanze raporo yubushakashatsi: gukuraho ibipimo byangiza cyane LLM ibanziriza amahugurwa. Bahuza ibisubizo hamwe nisesengura ryuburyo bwa Pre-Norm yubatswe. Ukurikije impapuro, ibipimo byerekana ntabwo byongera imiterere yicyitegererezo muri ubwo bwubatsi. Ahubwo, batezimbere optimizasiyo binyuze mubyo abanditsi basobanura nkigikorwa cyo kwikuramo imbaraga zo kubanziriza ibishushanyo mbonera. Mu buryo bufatika, ibisobanuro byatanzwe bireba uburyo ivugurura ryamahugurwa ryitwara, ntabwo kwiyongera muburyo bwimikorere icyitegererezo gishobora kwerekana.
Ubushakashatsi butandukanya kandi ibisanzwe mubisanzwe byinjira-bisanzwe hamwe nibisohoka-bisanzwe kandi bigasesengura kubora muburyo butandukanye kuri buri. Abanditsi bavuga ko kugabanuka k'uburemere bifasha ibipimo byinjira mu bice byinjira-bisanzwe ariko bikabangiriza mu bisohoka-bisanzwe kuko ibibanza byombi bigira uruhare rutandukanye mugutezimbere no kwerekana. Iri tandukanyirizo riganisha ku mpinduka eshatu zasabwe: ishami ryihariye rya heterogeneité, ivugururwa ryashyizwe kumurongo ushushanyije, hamwe nubunini-icyerekezo reparameterisation. Buri kimwe gisobanurwa nkicyoroshye kandi cyuzuzanya.
Urupapuro ruhuza izo mpinduka muburyo bumwe-buke-ingamba. Igitekerezo kivuga ko ingamba zasuzumwe mu bushakashatsi bwakozwe mbere y’amahugurwa arimo urugero rwinshi n’uruvange rw’inzobere kuva kuri miliyari 0.12 kugeza kuri miliyari 2. Ibizamini bivugwa ko byibanze kuri optimizers hamwe na gahunda yo kwiga-igipimo kandi ikoresha ingengo yimari yinganda. Abanditsi bavuga ko uburyo bumwe bwagiye butera igihombo cyo hasi kurenza umurongo wateganijwe neza kandi werekana imyitwarire myiza yo gupima mugihe wongeyeho ibipimo bidafite ishingiro no kubara hejuru.
Ibisobanuro birambuye: arxiv.org ↗
Impamvu ari ngombwa
Akazi kerekana igice gito cyigiciro cyubwubatsi bwa LLM gishobora kugira ingaruka kumyitozo ihamye no gupima. Niba ibisubizo byatangajwe birenze ubushakashatsi bwimpapuro, abategura icyitegererezo barashobora kunguka neza cyangwa gutakaza imyitozo yo hasi binyuze mumihindagurikire mito aho kuba moderi nini cyangwa ibyuma byiyongera.
Akamaro gafatika kakazi nako yibanda kubikorwa byiza. Guhugura imvugo nini yindimi ntabwo byakozwe gusa kubara ibipimo, imibare, hamwe nibikoresho, ariko kandi nuburyo gradients hamwe nibishya bigenda byubaka. Ikintu cyongeramo ingano cyangwa kubara birashobora guhindura niba imyitozo igera kubisubizo byiza. Ibyavuye mu mpapuro, nibisubirwamo, byerekana ibipimo byerekana ahantu hashobora kuba ingirakamaro mu kuzamura amahugurwa hatabayeho kwagura icyitegererezo.
Itandukaniro riri hagati yo kwerekana no gutezimbere nabyo ni ingaruka. Abanditsi ntibavuga ko ibipimo byerekana ibipimo bireka Imbere-isanzwe igereranya ibikorwa byinshi. Igitekerezo cyabo nuko ibice bifasha inzira yo guhugura gukoresha ubwubatsi buriho neza. Itandukaniro rifite akamaro mugihe usobanura ibisubizo: nibimenyetso byinzira yanyuze mugihe cyo kwiga, ntabwo ari gihamya ko inyubako ntoya yubatswe ihita ikora icyitegererezo gishoboye mugihe cyo gufata umwanzuro.
Impinduka zasabwe zirashobora gukurura abategura icyitegererezo kuko inkomoko ibasobanura nkibintu bifite agaciro gake hamwe no kubara hejuru. Ihinduka rishobora kuba byoroshye kugerageza mumahugurwa kuruta inzira zisaba moderi nini, imibare mishya, cyangwa sisitemu yinyongera. Raporo yatangajwe kuri moderi yuzuye kandi ivanze-yinzobere ifite akamaro kuko irerekana ko abanditsi bagerageje ingamba muburyo bubiri bwagutse aho kuba imiterere imwe gusa.
Inyungu rusange ziracyateganijwe. Igihombo cyo hasi cyamahugurwa gishobora kwerekana uburyo bwiza bwo gukora neza, ariko isoko ntisobanura ko ibyitegererezo ari ukuri, umutekano, bihendutse gutanga, cyangwa byiza kubisabwa byihariye. Ntabwo kandi igereranya kuzigama ingufu, kugabanya imyitozo-igihe, cyangwa kuzigama ibyuma. Agaciro ako kanya k'impapuro rero ni icyerekezo cyubushakashatsi hamwe nibisabwa byerekana ibishushanyo mbonera, ntabwo byerekana umusaruro.
Uburyo bukoreshwa: Uburyo bukora
Shakisha ikoranabuhanga ryihishe inyuma yiri terambere.
Which component of an AI application is the machine-learning model itself?
Ibyo kureba
Ibibazo by'ingenzi ni ukumenya niba ivugururwa ryakozwe ryigana mu bwigenge, uko inyungu nini ari mu bikorwa, kandi niba zihindura imikorere myiza yo hasi. Ubushakashatsi bukubiyemo icyitegererezo kuva kuri 0.12B kugeza kuri 2B, bityo rero akamaro kayo kuri sisitemu nini nini iracyashyirwaho.
Ikibazo cya mbere cyo kureba nubunini nuburinganire bwinyungu zavuzwe. Inkomoko ivuga ko ingamba zihuriweho zigera ku gihombo cyo hasi kuruta kugereranywa neza, ariko ibivuyeho nta tandukanyirizo rishingiye ku mibare, itandukaniro ryiruka, cyangwa gusenyuka ukurikije urugero rw'icyitegererezo, optimizer, gahunda yo kwiga-igipimo, cyangwa gushyira ibisanzwe. Ibyo bisobanuro birakenewe kugirango hamenyekane niba iterambere ari rinini bihagije kugirango rikorwe mubikorwa cyangwa ni ingaruka zifatika zubushakashatsi.
Kwigana byigenga bizaba ingenzi kuko impapuro nigitabo cya arXiv aho kuba igisubizo cyurungano rwasubiwemo ibisubizo byatanzwe. Gusubiramo byingirakamaro byagerageza impinduka eshatu zitandukanye kandi hamwe, ukabigereranya nibyingenzi bigezweho, kandi ugasuzuma niba inyungu igaragara irokoka impinduka zivanze namakuru, gutangiza, ingengo yimari, no kuyishyira mubikorwa. Urupapuro rwinkomoko rwerekana amahuza ajyanye ningingo, ariko inyandiko yatanzwe ntabwo igaragaza kuboneka, kuzura, cyangwa gukoresha ibikoresho byororoka.
Igipimo cyibipimo ni indi mbogamizi. Ubushakashatsi bwatangajwe buri hagati ya 0.12B kugeza 2B, bufite akamaro kubushakashatsi bugenzurwa ariko ntabwo ubwabwo bwerekana ko ingaruka zimwe zifata imipaka nini cyane. Abashakashatsi bazakenera gusuzuma niba imyitwarire-igipimo cyimyitwarire ihinduka hamwe nuburebure, ubugari, uburebure bwikurikiranya, imvange-yinzobere igenda, cyangwa ubundi buryo bwo guhitamo. Inyandiko yimpapuro irashobora gutanga ubuyobozi, ariko abstract yonyine ntabwo ishyiraho imipaka yibisubizo.
Hanyuma, ingaruka zo hasi zikomeza kutamenyekana. Isuzumabumenyi rizaza rigomba gupima neza inshingano, kalibrasi, imbaraga, ikiguzi cyo kugereranya, hamwe nuburyo bwiza bwo guhugura aho gushingira gusa kubihombo byanyuma. Bizaba byiza kandi kureba niba ingamba ziteganijwe zifitanye isano no kubara, gutunganya neza, gukomeza imyitozo mbere, cyangwa amahugurwa yumutekano. Kugeza ubwo ibyo bibazo bisubijwe, umwanzuro ushigikiwe cyane nuko igice gito cyubwubatsi gishobora kugira ingaruka zidasanzwe muburyo LLMs itoza.