RMSNorm na Pre-Layer Normalization
RMSNorm bụ oyi akwa normalization dị fechaa nke na-eweghachi ọrụ site na mgbọrọgwụ ha pụtara square, yana ebe a na-emezi ihe tupu oyi akwa na-aga n'ihu sublayer ọ bụla karịa ka emechaa.
Nchịkọta
Together they make deep transformers train stably without warmup tricks.
Ime miri emi
Standard LayerNorm na-ewepụ ihe ọ pụtara wee kewaa site na ọkọlọtọ ọkọlọtọ gafee vector atụmatụ, wee tinye usoro mmụta na mgbanwe. RMSNorm, nke Zhang na Sennrich webatara na 2019, tụfuru ihe na-akpachaghị anya na nhụsianya kpamkpam: ọ na-ekewa vector ọ bụla site na mgbọrọgwụ n'akụkụ akụkụ nke ihe ya wee na-amụba site na uru njiri mara nke ọ bụla. Nke a na-ewepụ otu ọnụ ọgụgụ na ọtụtụ arụmọrụ, na-egbutu ịgbakọ site na 10-50% na oyi akwa nkịtị ka ọ dabara nke ọma. Iche iche, ntinye 'Pre-LN' (usoro tupu nlebara anya / MLP, nke nwere ụzọ dị ọcha gbara ya gburugburu) na-edobe ogo gradient na mbido, yabụ ụdị dị ka GPT-3, LLAMA, na PaLM ụgbọ oloko na-enweghị mmụta-ọnụego ọkụ hacks nke mbụ Post-LN transformer chọrọ.
Nghọta nka nka
Maka vector x nke akụkụ d, RMSNorm na-agbakọ x_i * g_i / sqrt ((1/d) * nchikota (x_j^2) + epsilon), ebe g bụ vector uru mmụta. Enweghị mwepu pụtara na enweghị nhụsianya. N'ihi na iyi nke fọdụrụ na ngọngọ Pre-LN na-agafe nhazi ahụ, ụzọ njirimara adịghị emetụ ya aka na gradients na-asọpụta ozugbo site na mmepụta gaa na ntinye, nke mere na nchịkọta dị omimi na-agbakọta.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke RMSNorm na nhazi nke pre-Layer
RMSNorm bụ ihe ndabara na ọtụtụ LLM dị arọ mepere emepe (LLaMA, Mistral, Qwen, Gemma), yabụ na-atụ anya na ọ ga-abụ ọkọlọtọ. Nchọpụta na-emezi usoro nhazi ahụ: QK-norm na-emetụta RMSNorm na ajụjụ nlebara anya na igodo iji mebie uto logit, ụfọdụ ụlọ nyocha na-ejikọta pre- na post-norm ('sandwich' ma ọ bụ 'peri-LN') maka nkwụsi ike na nha trillion-parameter. Mkpụrụ akụrụngwa na-aga n'ihu na-ejikọta ọrụ ahụ maka ọsọ.
Mmejuputa n'ezie n'ụwa
LLAMA, Mistral, na Qwen niile dochie LayerNorm na RMSNorm iji kpụọ nkụchi obi na akara ọ bụla.
Pre-LN na-ahapụ ụdị ụdị GPT ka ọ zụọ ya na-enweghị oke mmụta mmụta nke 2017 Post-LN transformer chọrọ.
QK-normalization na-eji RMSNorm na ajụjụ nlebara anya na igodo iji kwụsị ịgbawa n'ụdị buru ibu.
Ndị na-agbanwe ekwentị mkpanaaka na ihu na-anakwere RMSNorm n'ihi na idobe isi na enweghị isi na-ebelata okporo ụzọ ebe nchekwa
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the RMSNorm and Pre-Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nkịtị oyi akwa
Ajụjụ a na-ajụkarị
What is RMSNorm and Pre-Layer Normalization?
RMSNorm bụ oyi akwa normalization dị fechaa nke na-eweghachi ọrụ site na mgbọrọgwụ ha pụtara square, yana ebe a na-emezi ihe tupu oyi akwa na-aga n'ihu sublayer ọ bụla karịa ka emechaa. Ha na-ejikọ ọnụ na-eme ka igwe ngbanwe dị omimi na-azụ nke ọma na-enweghị aghụghọ aghụghọ.
Kedu ihe RMSNorm na-ahapụ ma e jiri ya tụnyere LayerNorm ọkọlọtọ?
RMSNorm na-amali ịgbakọ ma na-ewepụ ihe ọ pụtara, na-emezigharị naanị site na mpụta obosara nke mgbanaka.
Kedu ọnụọgụ RMSNorm na-ekekọrịta vector ọ bụla?
RMSNorm na-ekewa site na sqrt nke ihe ndị nwere squared, ya bụ mgbọrọgwụ pụtara square, wee tinye uru mmụta.
Gịnị bụ isi uru nke pre-layer normalization n'elu post-layer normalization?
Ịtụkwasị ụkpụrụ n'ihu sublayer nke ọ bụla na ụzọ dị mma fọdụrụnụ na-ejikọta oke gradient, wepụ mkpa maka ikpo ọkụ-ọnụego mmụta.
N'ime mgbochi Pre-LN, kedu ụzọ oyi akwa normalization na-ama ụma hapụ imetụ ya aka?
Pre-LN na-ahazi ntinye ntinye na sublayer mana ụzọ mkpirisi fọdụrụ na-agafe ya, na-echekwa okporo ụzọ gradient dị ọcha.
Kedu ezinaụlọ ụdịdị arọ mepere emepe na-eji RMSNorm na ndabara?
RMSNorm ghọrọ ọkọlọtọ ọkọlọtọ na LLAMA, Mistral, Qwen, Gemma na ndị LLM kacha ọhụrụ.