RMSNorm iyo Caadiyeynta Lakabka Kahor
RMSNorm waa lakabka caadiga ah ee khafiifka ah kaas oo dib u miisaamaya dhaqdhaqaaqa xididkooda celcelis ahaan labajibbaaran, iyo lakabka hore ee caadiga ah ee meelaha ka hormaraya sublayer kasta halkii uu ka dambayn lahaa.
Dulmar
Together they make deep transformers train stably without warmup tricks.
quusid qoto dheer
Heerka LayerNorm wuxuu kala gooyaa celceliska wuxuuna u qaybiyaa isbedbeddelka caadiga ah ee guud ahaan vector sifada, ka dibna wuxuu dabaqaa cabbir la bartay iyo isbeddel. RMSNorm, oo ay soo bandhigeen Zhang iyo Sennrich sanadka 2019, waxay hoos u dhigaysaa dhexdhexaadnimada iyo eexda gebi ahaanba: waxay si fudud u qaybisaa vector kasta xididka celceliska curiyeheeda waxayna ku dhufataa faa'iido muuqaal kasta oo la bartay. Tani waxay meesha ka saaraysaa hal tiro-koob iyo hawlgalo dhawr ah, iyada oo la jarayo xisaabinta qiyaas ahaan 10-50% lakabka caadiga ah iyada oo la mid ah saxnaanta. Si gooni ah, meelaynta 'Pre-LN' (caadi ka hor feejignaanta/MLP, oo leh waddo nadiif ah oo hareeraheeda ah) waxay ilaalinaysaa cabbirka jaangooyooyinka bilowga ah, marka moodooyinka sida GPT-3, LLAMA, iyo PaLM tareenka iyada oo aan la jabsanayn heerka-kululaynta heerka waxbarashada kaas oo beddelka Post-LN asalka u baahan yahay.
Aragtida Farsamada
Halbeegga x ee cabbirka d, RMSNorm wuxuu xisaabiyaa x_i * g_i / sqrt ((1/d) * wadarta (x_j^2) + epsilon), halkaasoo g uu yahay faa'iido la bartay. Ma jiro kala goyn dhexdhexaad ah iyo eex. Sababtoo ah qulqulka hadhaaga ah ee ku jira xannibaadda Pre-LN wuxuu dhaafayaa caadinimada, dariiqa aqoonsiga ayaa ah mid aan la taaban oo jaangooyooyinku waxay si toos ah uga qulqulayaan wax soo saarka ilaa gelinta, taas oo ah sababta xirmooyinka aadka u qoto dheer ay isugu yimaadaan.
Saamaynta Istiraatijiyadeed
Qiimaha iyo miisaaniyada
Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
Go'aamo cad
Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
Xakamaynta tayada
Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Mustaqbalka RMSNorm iyo Caadiga Lakabka Kahor
RMSNorm hadda waa ku-talogalka inta badan LLM-yada miisaanka-furan (LLaMA, Mistral, Qwen, Gemma), markaa filo inay ahaan doonto heerka. Cilmi-baadhistu waxay nadiifinaysaa cuntada: QK-caadiga waxay khusaysaa RMSNorm su'aalaha dareenka iyo furayaasha si loo horumariyo logit, iyo shaybaadhka qaarkood waxay isku daraan ka hor iyo ka dib ('sandwich' ama 'peri-LN') xasilloonida dheeraadka ah ee qiyaasta trillion-parameter. Kernels-ka qalabku waxa ay ku wadaan in ay dardargeliyaan hawlgalka.
Dhaqangelinta Adduunka-dhabta ah
LLAMA, Mistral, iyo Qwen dhamaantood waxay ku beddelaan LayerNorm RMSNorm si ay u xiiraan daahitaanka calaamad kasta
Pre-LN waxay u ogolaataa moodooyinka qaabka GPT inay tababaraan iyaga oo aan lahayn kulaylka heerka waxbarashada ee 2017 ka dambeeya LN u baahan yahay
Caadiyan QK waxay isticmaashaa RMSNorm su'aalaha fiiro gaar ah iyo furayaasha si ay u joojiyaan saqafka inay ku qarxaan moodooyinka waaweyn
Transformers mobilada iyo geesaha waxay qaataan RMSNorm sababtoo ah hoos u dhigista celceliska iyo eexda waxay yaraynaysaa socodka xusuusta
Khatarta & Dariiqyada Ilaalada
Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Qorshe Hawleedka Dhaqangelinta
Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the RMSNorm and Pre-Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Caadiyan lakabka
Su'aalaha soo noqnoqda
What is RMSNorm and Pre-Layer Normalization?
RMSNorm waa lakabka caadiga ah ee khafiifka ah kaas oo dib u miisaamaya dhaqdhaqaaqa xididkooda celcelis ahaan labajibbaaran, iyo lakabka hore ee caadiga ah ee meelaha ka hormaraya sublayer kasta halkii uu ka dambayn lahaa. Si wada jir ah waxay u sameeyaan transformers qoto dheer u tababaran si xasilan oo aan lahayn khiyaamo kulul.
Muxuu RMSNorm ka tagayaa marka la barbar dhigo heerka LayerNorm?
RMSNorm waxa uu ka boodaa xisaabinta iyo kala-goynta celceliska, isaga oo caadi ka dhigaya kaliya labajibbaaran xididka ee hawlgelinta.
Immisa ayay RMSNorm u qaybinaysaa vector kasta oo firfircooni?
RMSNorm waxay u qaybisaa sqrt ee celceliska canaasiirta labajibbaaran, i.e. xididka celceliska labajibbaaran, dabadeed waxay dabaqaysaa faa'iido la bartay.
Waa maxay faa'iidada ugu weyn ee caadiga ah ee lakabka hore ee caadiga ah ee lakabka ka dambeeya?
Dhigista heerka caadiga ah ee ka horraysiiye kasta oo leh dariiqa hadhaaga oo nadiif ah ayaa xaddidaya cabbirka jaranjarada, taasoo meesha ka saaraysa baahida kululaynta heerka waxbarashada.
Gudaha ka hor-LN-ka, jidkee buu lakabka caadiga ka dhigayaa si badheedh ah oo aan la taaban?
Pre-LN waxa ay caadi ka dhigtaa wax galinta sublayer laakiin jidka gaaban ee soo hadha ayaa dhaafa, isaga oo ilaalinaya jid sare oo nadiif ah.
Qoysaskee moodeelka furfuran ee casriga ah ayaa si caadi ah u isticmaala RMSNorm?
RMSNorm waxay noqotay heerka caadiga ah ee LLMA, Mistral, Qwen, Gemma iyo LLM-yadii u dambeeyay.