RMSNorm da Pre-Layer Normalization
RMSNorm wani yanki ne na daidaita nauyi mai nauyi wanda ke sake daidaita kunnawa ta tushen ma'anar murabba'in su, da wuraren daidaitawa na pre-layer waɗanda ke mataki a gaban kowane mai yin ƙasa maimakon bayan.
Dubawa
Together they make deep transformers train stably without warmup tricks.
Zurfafa nutsewa
Standard LayerNorm yana cire ma'ana kuma yana rarraba ta daidaitaccen karkatacciyar hanya a cikin sifa, sannan yana amfani da sikelin koyo da motsi. RMSNorm, wanda Zhang da Sennrich suka gabatar a shekarar 2019, ya sauke ma'anar tsaka-tsaki da son zuciya gabaɗaya: kawai yana raba kowane vector ta tushen ma'anar ma'anar abubuwansa kuma yana ninka ta hanyar samun koyan kowane nau'i. Wannan yana cire ƙididdiga ɗaya da ayyuka da yawa, yankan ƙididdigewa da kusan 10-50% a cikin tsarin al'ada yayin daidaita daidaito. Na dabam, jeri na 'Pre-LN' (ka'ida kafin hankali/MLP, tare da tsaftatacciyar hanya a kusa da shi) yana kiyaye girman gradient a lokacin farawa, don haka samfura kamar GPT-3, LLAMA, da PaLM jirgin ƙasa ba tare da koyo-kudin dumama hacks cewa asali Post-LN transformer bukata.
Fahimtar Fasaha
Don vector x na girma d, RMSNorm yana lissafta x_i * g_i / sqrt((1/d) * jimlar(x_j^2) + epsilon), inda g shine ingantaccen vector riba. Babu ragi mai ma'ana kuma babu son zuciya. Saboda ragowar rafi a cikin toshewar Pre-LN yana ƙetare daidaitawa, hanyar ainihi ta tsaya ba a taɓa ba kuma gradients suna gudana kai tsaye daga fitarwa zuwa shigarwa, wanda shine dalilin da yasa tarin zurfafa ke haɗuwa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar RMSNorm da Pre-Layer Normalization
RMSNorm yanzu shine tsoho a yawancin LLMs masu buɗaɗɗen nauyi (LLaMA, Mistral, Qwen, Gemma), don haka tsammanin zai kasance daidai. Bincike yana tace girke-girke: QK-al'ada ya shafi RMSNorm ga tambayoyin kulawa da maɓallan don inganta haɓakar logit, kuma wasu labs suna haɗa pre- da bayan-na'a ('sandwich' ko 'peri-LN') don ƙarin kwanciyar hankali a sikelin-trillion-parameter. Kwayoyin kayan aikin suna ci gaba da haɗa aikin don saurin aiki.
Aiwatar da Gaskiyar Duniya
LLAMA, Mistral, da Qwen duk sun maye gurbin LayerNorm tare da RMSNorm don aske larwar ƙima akan kowace alama.
Pre-LN yana ƙyale samfuran salon GPT su horar ba tare da dumama ƙimar koyo wanda 2017 Post-LN transfomer ke buƙata
QK-normalization yana amfani da RMSNorm akan tambayoyin hankali da maɓalli don dakatar da fashe fashe a cikin manyan samfura.
Wayoyin hannu da na'urorin wuta na gefe suna ɗaukar RMSNorm saboda faduwa ma'ana da son zuciya yana rage zirga-zirgar ƙwaƙwalwar ajiya
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the RMSNorm and Pre-Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Daidaita Layer
Tambayoyin da ake yawan yi
What is RMSNorm and Pre-Layer Normalization?
RMSNorm wani yanki ne na daidaita nauyi mai nauyi wanda ke sake daidaita kunnawa ta tushen ma'anar murabba'in su, da wuraren daidaitawa na pre-layer waɗanda ke mataki a gaban kowane mai yin ƙasa maimakon bayan. Tare suna sa masu canji masu zurfi suna yin horo a tsaye ba tare da dabarun dumama ba.
Menene RMSNorm ke tsallakewa idan aka kwatanta da daidaitaccen LayerNorm?
RMSNorm ya tsallake lissafi da rage ma'ana, daidaitawa kawai ta tushen ma'anar murabba'in kunnawa.
Ta wane adadi RMSNorm ke raba kowane vector kunnawa?
RMSNorm yana raba ta hanyar sqrt na ma'anar abubuwa masu murabba'i, watau tushen ma'anar murabba'i, sannan yana amfani da riba da aka koya.
Menene babban fa'idar daidaitawar pre-Layer akan daidaitawar bayan-layi?
Sanya al'ada a gaban kowane sublayer tare da madaidaiciyar hanya mai tsafta yana iyaka girman girma, cire buƙatar dumama ƙimar koyo.
A cikin toshewar Pre-LN, wace hanya ce layin daidaitawa ya bar da gangan ba a taɓa shi ba?
Pre-LN yana daidaita shigarwar zuwa sublayer amma saura gajeriyar hanyar wucewa ta, tana kiyaye tsaftataccen babban titin gradient.
Wadanne iyalai na zamani masu budadden nauyi ne ke amfani da RMSNorm ta tsohuwa?
RMSNorm ya zama daidaitaccen daidaitawa a cikin LLAMA, Mistral, Qwen, Gemma da LLM na baya-bayan nan.