InoteveraGaidhi rinotevera
Layer Normalization
Tekinoroji
Nhungamiro yehunyanzvi
RMSNorm uye Pre-Layer Normalization
RMSNorm inzvimbo yakareruka yekujairisa iyo inodzoreredza activation nemidzi yavo inoreva square, uye pre-layer normalization nzvimbo dzinotsika pamberi pega yega sublayer kwete mushure.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
Together they make deep transformers train stably without warmup tricks.
Kudzika Kwakadzika
Standard LayerNorm inobvisa zvinoreva uye inokamura neyakajairwa kutsauka pane imwe vheta, yobva yaisa chikero chakadzidzwa uye shanduko. RMSNorm, yakaunzwa naZhang naSennrich muna 2019, inodonhedza kureva-pakati uye kusarura zvachose: inongopatsanura yega yega vector nemudzi inorevesa square yezvinhu zvayo uye inowanza neyakadzidziswa pane-chimwe chinhu kuwana. Izvi zvinobvisa imwe nhamba uye akati wandei mashandiro, kucheka komputa neinoda kusvika 10-50% mune yakajairwa layer uku ichifananidza kurongeka. Neparutivi, iyo 'Pre-LN' yekuisa (yakajairika pamberi pekutarisa / MLP, ine yakachena yakasara nzira yakaitenderedza) inochengeta gradient magnitudes yakasungwa pakutanga, saka modhi seGPT-3, LLaMA, uye PaLM chitima pasina kudzidza-chiyero chekudziya hacks yaidiwa yekutanga Post-LN transformer.
Kuitwa muNyika yeChokwadi
LLaMA, Mistral, uye Qwen vese vanotsiva LayerNorm neRMSNorm yekuveura inference latency pachiratidzo chega chega.
Pre-LN inobvumira maGPT-maitiro ekudzidzira pasina yekudzidzira-chiyero chekudziya icho 2017 Post-LN transformer yaidiwa.
QK-normalization inoshandisa RMSNorm pamibvunzo yekutarisisa uye makiyi ekumisa matanda kubva kuputika mumhando huru.
Nharembozha uye kumucheto kushandura kunotora RMSNorm nekuti kudonhedza zvinoreva uye kusarura kunoderedza ndangariro traffic
Ramangwana reRMSNorm uye Pre-Layer Normalization
RMSNorm ikozvino ndiyo yakasarudzika mune akawanda akavhurika-huremu LLMs (LLaMA, Mistral, Qwen, Gemma), saka tarisira kuti irambe yakajairwa. Tsvagiridzo iri kunatsa iyo resipi: QK-yakajairika inoshandisa RMSNorm kumibvunzo yekutarisisa uye makiyi ekudzoreredza kukura kwelogiti, uye mamwe maLab anosanganisa pre- uye post-zvakajairwa ('sandwich' kana 'peri-LN') kuti iwedzere kugadzikana patrilioni-paramita chikero. Hardware kernels dzinoramba dzichibatanidza kushanda kwekukurumidza.
Kuzvishandisa
Nei zvichikosha
- Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
- Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
- Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Chenjera
- Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
- Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
- Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Matanho anotevera
- Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
- Benchmark pasi pechokwadi mutoro uye data mamiriro.
- Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
- Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the RMSNorm and Pre-Layer Normalization quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
What is RMSNorm and Pre-Layer Normalization?
RMSNorm inzvimbo yakareruka yekujairisa iyo inodzoreredza activation nemidzi yavo inoreva square, uye pre-layer normalization nzvimbo dzinotsika pamberi pega yega sublayer kwete mushure. Pamwe chete vanoita zvakadzika ma transformer kudzidzisa zvakatsiga pasina warmup tricks.
Chii chinosiya RMSNorm ichienzaniswa neyakajairwa LayerNorm?
RMSNorm inosvetuka komputa uye kubvisa zvinorehwa, zvinongojairika chete nemudzi unoreva chikwere chezvinoitwa.
Nehuwandu hupi RMSNorm inogovanisa yega yega activation vector?
RMSNorm inokamura ne sqrt yezvinoreva zvinhu zvine squared, kureva mudzi unoreva sikweya, wobva waisa budiriro yakadzidzwa.
Ndeapi maitiro egradient anotsigira Pre-LN pakutanga mune zvakadzika Transformers?
Xiong et al. ongorora gradient maitiro uye ratidza Pre-LN gradients ari nani kuzvibata pakutanga; izvi hazvivimbisi kudziya-yemahara kudzidziswa kwese kuseta.
Mune Pre-LN block, ndeipi nzira iyo iyo normalization layer inosiya nemaune isina kubatwa?
Pre-LN inogadzirisa kupinza kune sublayer asi yasara nzira yekudimbudzira inoipfuura, ichichengetedza yakachena gradient mugwagwa mukuru.
Ndedzipi dzemazuva ano dzakavhurika-huremu modhi mhuri dzinoshandisa RMSNorm nekukasira?
RMSNorm yakava yakajairwa yakajairwa muLLAMA, Mistral, Qwen, Gemma uye neazvino maLLM.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi