Ubuyobozi bwa tekiniki

RMSNorm na Pre-Layeri bisanzwe

RMSNorm ni urwego rusanzwe rusanzwe rusubiramo ibikorwa ukoresheje imizi yabyo bisobanura kare, hamwe na pre-layer bisanzwe bisanzwe bigenda mbere ya buri sublayer kuruta nyuma.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Kazoza ka RMSNorm na Pre-Layeri bisanzwe
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Together they make deep transformers train stably without warmup tricks.

Kwibira cyane

Igipimo gisanzweNorm ikuramo ibisobanuro kandi igabanywa no gutandukana bisanzwe kurwego rwimiterere, hanyuma ikoresha igipimo cyize no guhinduranya. RMSNorm, yatangijwe na Zhang na Sennrich muri 2019, irekura hagati-yo kubogama no kubogama rwose: igabanya buri vector kumuzi igereranya kwaduka yibintu byayo kandi ikagwiza kubwinyungu yize kuri buri kintu. Ibi bivanaho imibare imwe nibikorwa byinshi, kugabanya compte hafi 10-50% murwego rusanzwe mugihe bihuye nukuri. Bitandukanye, gushyira 'Pre-LN' gushyira (bisanzwe mbere yo kwitabwaho / MLP, hamwe n'inzira isigaranye hafi yayo) ituma ubunini bwa gradient bugarukira mugutangira, bityo moderi nka GPT-3, LLaMA, na PaLM gari ya moshi itabanje kwiga-igipimo cyo gushyushya hack ya transfert yambere ya Post-LN yasabwaga.

Gushyira mu bikorwa Isi

LLaMA, Mistral, na Qwen byose bisimbuza LayerNorm na RMSNorm kugirango bogoshe ubukererwe kuri buri kimenyetso

Imbere-LN ireka imiterere-yuburyo bwa GPT itozwa nta gipimo cyo kwiga-gishyushye ko Transformer ya Post-LN 2017 yari ikeneye

QK-isanzwe ikoresha RMSNorm kubibazo byibanze hamwe nurufunguzo rwo guhagarika logi guturika muburyo bunini

Impinduramatwara igendanwa nu mpande zifata RMSNorm kuko guta bivuze no kubogama bigabanya urujya n'uruza

Kazoza ka RMSNorm na Pre-Layeri bisanzwe

RMSNorm ubu isanzwe muburyo bwa LLMs (LLaMA, Mistral, Qwen, Gemma), bityo rero utegereze ko bizakomeza kuba bisanzwe. Ubushakashatsi burimo kunonosora resept: QK-norm ikoresha RMSNorm kubibazo byo kwitondera hamwe nurufunguzo rwo gukura logit, kandi laboratoire zimwe zihuza mbere na nyuma yibisanzwe ('sandwich' cyangwa 'peri-LN') kugirango habeho umutekano muke kuri tiriyari-ibipimo. Ibikoresho byibyuma bikomeza guhuza ibikorwa kugirango byihute.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the RMSNorm and Pre-Layer Normalization quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is RMSNorm and Pre-Layer Normalization?

RMSNorm ni urwego rusanzwe rusanzwe rusubiramo ibikorwa ukoresheje imizi yabyo bisobanura kare, hamwe na pre-layer bisanzwe bisanzwe bigenda mbere ya buri sublayer kuruta nyuma. Hamwe na hamwe bakora impinduka zimbitse zitoza neza nta mayeri yo gushyushya.

Niki RMSNorm ikuraho ugereranije na Layeri isanzwe?

RMSNorm isiba kubara no gukuramo uburyo, busanzwe gusa kumuzi bivuze kwaduka yibikorwa.

Nuwuhe mubare RMSNorm igabanya buri gikorwa cyo gukora?

RMSNorm igabanijwe na sqrt yuburyo bwo kwaduka kwaduka, ni ukuvuga imizi isobanura kare, hanyuma ikoresha inyungu yize.

What gradient behavior does Pre-LN support at initialization in deep Transformers?

Xiong et al. analyze gradient behavior and show Pre-LN gradients are better behaved at initialization; this does not guarantee warmup-free training for every setup.

Mubice byabanjirije LN, ni ubuhe buryo busanzwe urwego rusanzwe rusiga nkana?

Imbere-LN isanzwe iyinjiza kuri sublayer ariko shortcut isigaye irayirenga, irinda umuhanda mwiza usukuye.

Niyihe miryango igezweho yuburemere yuburemere ikoresha RMSNorm kubisanzwe?

RMSNorm yabaye ibisanzwe bisanzwe muri LLaMA, Mistral, Qwen, Gemma na LLM ziherutse.