Nkịtị oyi akwa
Nhazi nke oyi akwa na-eme ka ọzụzụ kwụsie ike site n'ịkwaliteghachi mmemme n'ime ihe atụ nke ọ bụla ka ha wee nwee ọdịiche efu na otu.
Nchịkọta
It is a quiet but essential ingredient that makes deep transformers trainable.
Ime miri emi
Ba, Kiros, na Hinton webatara ya na 2016, oyi akwa normalization (LayerNorm) na-ekwu maka nsogbu nke ịgbalite n'ime netwọk miri emi nwere ike ịfefe na akpịrịkpa dị iche iche ka akara na-agafe n'ọtụtụ ọkwa, na-ebelata ma ọ bụ na-akụda mmụta. N'adịghị ka nhazi nhazi nke ogbe, nke na-emezi njirimara ọ bụla n'ofe ihe atụ dị na obere obere, LayerNorm na-emezigharị n'ofe atụmatụ nke otu ihe atụ. Nke a na-eme ka ọ nweere onwe ya na nha batch ma bụrụkwa nke a na-eji na ọzụzụ na ntinye aka, ọ na-arụkwa ọrụ nke ọma na usoro ogologo ogologo, ya mere ọ ghọrọ ọkọlọtọ maka ndị na-agbanwe agbanwe na-akwado ụdị asụsụ ọgbara ọhụrụ. Mgbe emezichara nke ọma, ọ na-etinye usoro mmụta mmụta (gamma) na mgbanwe (beta) ka netwọk wee nwetaghachi ihe nnọchite anya ọ bụla ọ chọrọ.
Nghọta nka nka
Maka njirimara vector x, LayerNorm na-agbakọ pụtara na ọdịiche dị n'etiti ihe ndị ahụ, wee wepụta gamma * (x - mean) / sqrt(variance + epsilon) + beta. N'ihi na ọnụ ọgụgụ na-abịa site na otu sample, omume bụ otu ma ogbe nwere 1 ma ọ bụ 1000 atụ. Ọdịiche dị mfe, RMSNorm, skips pụtara mwepu na kewaa naanị site na mgbọrọgwụ- mean-square, na-echekwa mkpokọta; a na-eji ya na ụdị dị ka Llama. Ndobe dịkwa mkpa: 'pre-norm' (normalizing tupu onye ọ bụla sublayer) na-eme ka ntụgharị miri emi dị mfe ịzụ karịa 'post-norm'.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke nhazi nke oyi akwa
A na-ahazi normalization maka arụmọrụ n'ọtụtụ. RMSNorm edochila LayerNorm n'ụdị asụsụ ọhụrụ ọhụrụ n'ihi na ọ dị ọnụ ala ma na-arụkwa ọrụ nke ọma, na ntinye akwụkwọ mbụ bụzi ndabara maka nchịkọta miri emi. Ndị na-eme nchọpụta na-aga n'ihu na-enyocha ụlọ ọrụ na-enweghị nhazi nke na-eji nlezianya mmalite ma ọ bụ aghụghọ eme ihe kama, na-achọ igbutu elu ma na-edobe nkwụsi ike ọzụzụ nke nhazi ahụ na-enye.
Mmejuputa n'ezie n'ụwa
Na-eme ka ngọngọ mgbanwe ọ bụla guzosie ike n'ụdị asụsụ dị ka GPT na BERT.
Na-eme ka RMSNorm dị ka nhọrọ ngbanwe dị mfe n'ime ụdị ezinụlọ Llama.
Na-ahazi data usoro ogologo agbanwe agbanwe na ụdị okwu na ntụgharị asụsụ ebe nha batch dị iche.
Ikwe ka ọzụzụ a pụrụ ịdabere na ya na nha nke otu, dị ka n'ụfọdụ ntọala mmụta nkwado.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
RMSNorm na Pre-Layer Normalization
Ajụjụ a na-ajụkarị
What is Layer Normalization?
Nhazi nke oyi akwa na-eme ka ọzụzụ kwụsie ike site n'ịkwaliteghachi mmemme n'ime ihe atụ nke ọ bụla ka ha wee nwee ọdịiche efu na otu. Ọ bụ ihe dị jụụ ma dị mkpa na-eme ka ntụgharị dị omimi na-azụ.
N'ofe gịnị ka nhazi oyibo na-agbakọ pụtara na ọdịiche ya?
LayerNorm na-emezigharị akụkụ njiri mara nke otu nlele, na-eme ka ọ kwụpụrụ onwe ya na ihe atụ ndị ọzọ na ogbe.
Gịnị kpatara na oyi akwa normalization họọrọ karịa ogbe normalization na transformers?
N'ihi na ọnụ ọgụgụ ya sitere na otu ọmụmaatụ, LayerNorm na-akpa àgwà mgbe niile n'agbanyeghị nha batch ma dabara n'usoro ederede agbanwe agbanwe.
Kedu ihe parampat gamma na beta na-ekwe ka LayerNorm mee?
Ka emechara ka ọ bụrụ ihe efu na ọdịiche otu nkeji, akpịrịkpa gamma na beta na-agbanwe nsonaazụ ya ka a ghara ịmanye ihe nlereanya ahụ na nkesa edoziri.
Kedu ka RMSNorm si dị iche na LayerNorm ọkọlọtọ?
RMSNorm na-ahapụ nzọụkwụ na-adị n'etiti na akpịrịkpa site na mgbọrọgwụ- mean-square nke mmemme, nke dị ọnụ ala ma jiri ya mee ihe na ụdị dịka Llama.
Kedu nsogbu oyi akwa normalization bụ isi nyere aka dozie na netwọk miri emi?
Ka akara na-agafe n'ọtụtụ ọkwa, ọnụ ọgụgụ ha nwere ike ịfụ ma ọ bụ daa; normalization na-edobe mmegharị n'ebe kwụsiri ike ka gradients na-akpakwa àgwà nke ọma.