RMSNorm na Pre-Layeri bisanzwe
RMSNorm ni urwego rusanzwe rusanzwe rusubiramo ibikorwa ukoresheje imizi yabyo bisobanura kare, hamwe na pre-layer bisanzwe bisanzwe bigenda mbere ya buri sublayer kuruta nyuma.
Incamake
Together they make deep transformers train stably without warmup tricks.
Kwibira cyane
Igipimo gisanzweNorm ikuramo ibisobanuro kandi igabanywa no gutandukana bisanzwe kurwego rwimiterere, hanyuma ikoresha igipimo cyize no guhinduranya. RMSNorm, yatangijwe na Zhang na Sennrich muri 2019, irekura hagati-yo kubogama no kubogama rwose: igabanya buri vector kumuzi igereranya kwaduka yibintu byayo kandi ikagwiza kubwinyungu yize kuri buri kintu. Ibi bivanaho imibare imwe nibikorwa byinshi, kugabanya compte hafi 10-50% murwego rusanzwe mugihe bihuye nukuri. Bitandukanye, gushyira 'Pre-LN' gushyira (bisanzwe mbere yo kwitabwaho / MLP, hamwe n'inzira isigaranye hafi yayo) ituma ubunini bwa gradient bugarukira mugutangira, bityo moderi nka GPT-3, LLaMA, na PaLM gari ya moshi itabanje kwiga-igipimo cyo gushyushya hack ya transfert yambere ya Post-LN yasabwaga.
Ubushishozi
Kuri vector x yikigereranyo d, RMSNorm ibara x_i * g_i / sqrt ((1 / d) * igiteranyo (x_j ^ 2) + epsilon), aho g ni inyigisho yunguka. Nta gukuramo gukuramo kandi nta kubogama. Kuberako imigezi isigaye muri pre-LN ihagarika ibisanzwe, inzira iranga iguma idakoraho kandi gradients zitemba ziva mubisohoka zinjira, niyo mpamvu ibice byimbitse bihurira.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza ka RMSNorm na Pre-Layeri bisanzwe
RMSNorm ubu isanzwe muburyo bwa LLMs (LLaMA, Mistral, Qwen, Gemma), bityo rero utegereze ko bizakomeza kuba bisanzwe. Ubushakashatsi burimo kunonosora resept: QK-norm ikoresha RMSNorm kubibazo byo kwitondera hamwe nurufunguzo rwo gukura logit, kandi laboratoire zimwe zihuza mbere na nyuma yibisanzwe ('sandwich' cyangwa 'peri-LN') kugirango habeho umutekano muke kuri tiriyari-ibipimo. Ibikoresho byibyuma bikomeza guhuza ibikorwa kugirango byihute.
Gushyira mu bikorwa Isi
LLaMA, Mistral, na Qwen byose bisimbuza LayerNorm na RMSNorm kugirango bogoshe ubukererwe kuri buri kimenyetso
Imbere-LN ireka imiterere-yuburyo bwa GPT itozwa nta gipimo cyo kwiga-gishyushye ko Transformer ya Post-LN 2017 yari ikeneye
QK-isanzwe ikoresha RMSNorm kubibazo byibanze hamwe nurufunguzo rwo guhagarika logi guturika muburyo bunini
Impinduramatwara igendanwa nu mpande zifata RMSNorm kuko guta bivuze no kubogama bigabanya urujya n'uruza
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the RMSNorm and Pre-Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Ubusanzwe
Ibibazo bikunze kubazwa
What is RMSNorm and Pre-Layer Normalization?
RMSNorm ni urwego rusanzwe rusanzwe rusubiramo ibikorwa ukoresheje imizi yabyo bisobanura kare, hamwe na pre-layer bisanzwe bisanzwe bigenda mbere ya buri sublayer kuruta nyuma. Hamwe na hamwe bakora impinduka zimbitse zitoza neza nta mayeri yo gushyushya.
Niki RMSNorm ikuraho ugereranije na Layeri isanzwe?
RMSNorm isiba kubara no gukuramo uburyo, busanzwe gusa kumuzi bivuze kwaduka yibikorwa.
Nuwuhe mubare RMSNorm igabanya buri gikorwa cyo gukora?
RMSNorm igabanijwe na sqrt yuburyo bwo kwaduka kwaduka, ni ukuvuga imizi isobanura kare, hanyuma ikoresha inyungu yize.
Ni izihe nyungu nyamukuru zo kubanziriza ibice bisanzwe hejuru yubusanzwe?
Gushyira ibisanzwe mbere ya buri sublayer hamwe ninzira isigaranye inzira isigara ihuza ubunini bwa gradient, bikuraho gukenera ubushyuhe-buke.
Mubice byabanjirije LN, ni ubuhe buryo busanzwe urwego rusanzwe rusiga nkana?
Imbere-LN isanzwe iyinjiza kuri sublayer ariko shortcut isigaye irayirenga, irinda umuhanda mwiza usukuye.
Niyihe miryango igezweho yuburemere yuburemere ikoresha RMSNorm kubisanzwe?
RMSNorm yabaye ibisanzwe bisanzwe muri LLaMA, Mistral, Qwen, Gemma na LLM ziherutse.