RMSNorm ati Iṣatunṣe Iṣe-Layer
RMSNorm jẹ fẹlẹfẹlẹ deede iwuwo fẹẹrẹ ti o ṣe atunṣe awọn iṣẹ ṣiṣe nipasẹ onigun mẹrin tumọ root wọn, ati awọn aaye isọdi-tẹlẹ-Layer ti o ṣe igbesẹ ṣaaju sublayer kọọkan ju lẹhin.
Akopọ
Together they make deep transformers train stably without warmup tricks.
Jin Dive
Standard LayerNorm yọkuro iwọntunwọnsi ati pin nipasẹ iyapa boṣewa kọja fekito ẹya kan, lẹhinna lo iwọn ti a kọ ati iyipada. RMSNorm, ti a ṣafihan nipasẹ Zhang ati Sennrich ni ọdun 2019, ṣubu aarin-itumọ ati aibikita patapata: o rọrun pin ipin kọọkan nipasẹ ipilẹ tumọ si onigun mẹrin ti awọn eroja rẹ ati isodipupo nipasẹ ere ti ẹya-ara ti kọ ẹkọ. Eyi yọkuro iṣiro kan ati ọpọlọpọ awọn iṣẹ ṣiṣe, gige iṣiro nipasẹ aijọju 10-50% ni ipele iwuwasi lakoko ti o baamu deede. Lọtọ, ipo 'Pre-LN' (iwuwasi ṣaaju akiyesi/MLP, pẹlu ọna aloku ti o mọ ni ayika rẹ) jẹ ki awọn iwọn gradient di opin ni ibẹrẹ, nitorinaa awọn awoṣe bii GPT-3, LLAMA, ati PaLM ọkọ oju irin laisi awọn hakii igbona-oṣuwọn ẹkọ ti atilẹba Post-LN transformer nilo.
Imọ-imọ-ẹrọ
Fun fekito x ti iwọn d, RMSNorm ṣe iṣiro x_i * g_i / sqrt ((1/d) * apao (x_j^2) + epsilon), nibiti g jẹ fekito ere ti ẹkọ. Ko si iyokuro itumo ko si si abosi. Nitori ṣiṣan ti o ku ni bulọọki Pre-LN kọja isọdọtun, ọna idanimọ duro laifọwọkan ati awọn gradients nṣan taara lati iṣelọpọ si titẹ sii, eyiti o jẹ idi ti awọn akopọ ti o jinlẹ pupọ pejọ.
Ipa Ilana
Iye owo ati isuna
Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
Awọn ipinnu diẹ sii
Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
Iṣakoso didara
Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ojo iwaju ti RMSNorm ati Iṣatunṣe Iṣe-Layer
RMSNorm ni bayi aiyipada ni ọpọlọpọ iwuwo-ìmọ LLMs (LLaMA, Mistral, Qwen, Gemma), nitorinaa nireti pe ki o wa ni idiwọn. Iwadi n ṣe atunṣe ohunelo naa: QK-norm kan RMSNorm si awọn ibeere akiyesi ati awọn bọtini lati ṣe idagbasoke idagbasoke logit, ati pe diẹ ninu awọn laabu ṣopọpọ ṣaaju ati lẹhin iwuwasi ('sandwich' tabi 'peri-LN') fun imuduro afikun ni iwọn trillion-parameter. Awọn ekuro ohun elo n tẹsiwaju lati dapọ iṣẹ ṣiṣe fun iyara.
Real-World imuse
LLAMA, Mistral, ati Qwen gbogbo ropo LayerNorm pẹlu RMSNorm lati fá airotẹlẹ inferan lori gbogbo ami
Pre-LN jẹ ki awọn awoṣe ara GPT ṣe ikẹkọ laisi igbona-oṣuwọn ẹkọ ti oluyipada Post-LN 2017 nilo
QK-normalization nlo RMSNorm lori awọn ibeere akiyesi ati awọn bọtini lati da awọn logits duro lati gbamu ni awọn awoṣe nla.
Alagbeka ati awọn oluyipada eti gba RMSNorm nitori jisilẹ tumọ ati ojuṣaaju dinku ijabọ iranti
Awọn ewu & Awọn ọna iṣọ
Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Ilana Ilana imuse
Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the RMSNorm and Pre-Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
Layer Deede
Awọn ibeere ti a beere nigbagbogbo
What is RMSNorm and Pre-Layer Normalization?
RMSNorm jẹ fẹlẹfẹlẹ deede iwuwo fẹẹrẹ ti o ṣe atunṣe awọn iṣẹ ṣiṣe nipasẹ onigun mẹrin tumọ root wọn, ati awọn aaye isọdi-tẹlẹ-Layer ti o ṣe igbesẹ ṣaaju sublayer kọọkan ju lẹhin. Papọ wọn jẹ ki awọn oluyipada jinlẹ ṣe ikẹkọ ni iduroṣinṣin laisi awọn ẹtan igbona.
Kini RMSNorm yọkuro ni akawe si LayerNorm boṣewa?
RMSNorm fo iširo ati iyokuro awọn iwọn, deede nikan nipasẹ awọn root tumosi square ti awọn iṣẹ-ṣiṣe.
Nipa iye wo ni RMSNorm ṣe pin awọn fekito imuṣiṣẹ kọọkan?
RMSNorm pin nipasẹ sqrt ti iwọn awọn eroja onigun mẹrin, ie root tumọ onigun mẹrin, lẹhinna lo ere ti o kọ ẹkọ.
Kini anfani akọkọ ti isọdi-ila-ila-tẹlẹ lori isọdọtun-Layer ti deede?
Gbigbe iwuwasi ṣaaju sublayer kọọkan pẹlu ọna aloku ti o mọ di iwọn iwọn mimu, yiyọ iwulo fun igbona-oṣuwọn ẹkọ.
Ni a Pre-LN Àkọsílẹ, ọna wo ni awọn normalization Layer koto fi untouched?
Pre-LN ṣe deede iṣagbewọle si sublayer ṣugbọn ọna abuja ti o ṣẹku kọja rẹ, titọju ọna opopona ti o mọ.
Awọn idile awoṣe-ìmọ iwuwo ode oni wo lo RMSNorm nipasẹ aiyipada?
RMSNorm di deede deede ni LLAMA, Mistral, Qwen, Gemma ati awọn LLM aipẹ julọ.