Ubuyobozi bwa tekiniki
RMSNorm na Pre-Layeri bisanzwe
RMSNorm ni urwego rusanzwe rusanzwe rusubiramo ibikorwa ukoresheje imizi yabyo bisobanura kare, hamwe na pre-layer bisanzwe bisanzwe bigenda mbere ya buri sublayer kuruta nyuma.
Umva iki gitaboAI-byavuzwe · ~2 min
Kuriyi page2 min soma
Incamake
Together they make deep transformers train stably without warmup tricks.
Kwibira cyane
Igipimo gisanzweNorm ikuramo ibisobanuro kandi igabanywa no gutandukana bisanzwe kurwego rwimiterere, hanyuma ikoresha igipimo cyize no guhinduranya. RMSNorm, yatangijwe na Zhang na Sennrich muri 2019, irekura hagati-yo kubogama no kubogama rwose: igabanya buri vector kumuzi igereranya kwaduka yibintu byayo kandi ikagwiza kubwinyungu yize kuri buri kintu. Ibi bivanaho imibare imwe nibikorwa byinshi, kugabanya compte hafi 10-50% murwego rusanzwe mugihe bihuye nukuri. Bitandukanye, gushyira 'Pre-LN' gushyira (bisanzwe mbere yo kwitabwaho / MLP, hamwe n'inzira isigaranye hafi yayo) ituma ubunini bwa gradient bugarukira mugutangira, bityo moderi nka GPT-3, LLaMA, na PaLM gari ya moshi itabanje kwiga-igipimo cyo gushyushya hack ya transfert yambere ya Post-LN yasabwaga.
Gushyira mu bikorwa Isi
LLaMA, Mistral, na Qwen byose bisimbuza LayerNorm na RMSNorm kugirango bogoshe ubukererwe kuri buri kimenyetso
Imbere-LN ireka imiterere-yuburyo bwa GPT itozwa nta gipimo cyo kwiga-gishyushye ko Transformer ya Post-LN 2017 yari ikeneye
QK-isanzwe ikoresha RMSNorm kubibazo byibanze hamwe nurufunguzo rwo guhagarika logi guturika muburyo bunini
Impinduramatwara igendanwa nu mpande zifata RMSNorm kuko guta bivuze no kubogama bigabanya urujya n'uruza
Kazoza ka RMSNorm na Pre-Layeri bisanzwe
RMSNorm ubu isanzwe muburyo bwa LLMs (LLaMA, Mistral, Qwen, Gemma), bityo rero utegereze ko bizakomeza kuba bisanzwe. Ubushakashatsi burimo kunonosora resept: QK-norm ikoresha RMSNorm kubibazo byo kwitondera hamwe nurufunguzo rwo gukura logit, kandi laboratoire zimwe zihuza mbere na nyuma yibisanzwe ('sandwich' cyangwa 'peri-LN') kugirango habeho umutekano muke kuri tiriyari-ibipimo. Ibikoresho byibyuma bikomeza guhuza ibikorwa kugirango byihute.
Kubishyira mubikorwa
Impamvu ari ngombwa
- Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
- Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
- Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Witondere
- Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
- Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
- Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Intambwe ikurikira
- Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
- Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
- Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
- Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Ibaruwa y'amakuru y'ubuntu
Ibuka amakuru ya buri munsi ya AI
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Take the RMSNorm and Pre-Layer Normalization quiz
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Ibibazo bikunze kubazwa
What is RMSNorm and Pre-Layer Normalization?
RMSNorm ni urwego rusanzwe rusanzwe rusubiramo ibikorwa ukoresheje imizi yabyo bisobanura kare, hamwe na pre-layer bisanzwe bisanzwe bigenda mbere ya buri sublayer kuruta nyuma. Hamwe na hamwe bakora impinduka zimbitse zitoza neza nta mayeri yo gushyushya.
Niki RMSNorm ikuraho ugereranije na Layeri isanzwe?
RMSNorm isiba kubara no gukuramo uburyo, busanzwe gusa kumuzi bivuze kwaduka yibikorwa.
Nuwuhe mubare RMSNorm igabanya buri gikorwa cyo gukora?
RMSNorm igabanijwe na sqrt yuburyo bwo kwaduka kwaduka, ni ukuvuga imizi isobanura kare, hanyuma ikoresha inyungu yize.
What gradient behavior does Pre-LN support at initialization in deep Transformers?
Xiong et al. analyze gradient behavior and show Pre-LN gradients are better behaved at initialization; this does not guarantee warmup-free training for every setup.
Mubice byabanjirije LN, ni ubuhe buryo busanzwe urwego rusanzwe rusiga nkana?
Imbere-LN isanzwe iyinjiza kuri sublayer ariko shortcut isigaye irayirenga, irinda umuhanda mwiza usukuye.
Niyihe miryango igezweho yuburemere yuburemere ikoresha RMSNorm kubisanzwe?
RMSNorm yabaye ibisanzwe bisanzwe muri LLaMA, Mistral, Qwen, Gemma na LLM ziherutse.
Komeza wige
Kuyobora
Abandi bayobozi batoranijwe kuriyi ngingo