Normalisasioŋu couche
Normalisasioŋ couche dafay dakkal tàggat yaram ci rescaling activations yi ci biir misaal bu nekk suko defee ñu am zero moyenne ak variance unitaire.
Résumé
It is a quiet but essential ingredient that makes deep transformers trainable.
Plongeur bu xóot
Ba, Kiros, ak Hinton ñoo ko dugal ci 2016, normalisation layer (LayerNorm) dafay saafara jafe-jafe bi ci biir reso bu xóot bi mën na dem ba ci eskaal yu wuute lool ndax siñaal yi dañuy jaar ci layer yu bari, di yeexal wala di destabilise jàng. LayerNorm wuute na ak normalisasioŋ batch, mooy normalise bépp màndarga ci misaal yi ci benn mini-batch, LayerNorm dafay normalise màndarga yi ci benn misaal. Loolu moo tax mu moom boppam ci dayo batch bi, te mën nañu ko jëfandikoo ci tàggat ak ci inference, te dafay dox ci anam wu natureel ak sequence yu am guddaay bu mën soppiku, moo tax mu nekk standard ci transformateur yiy dooleel modeli làkk yu bees yi. Ginaaw buñu ko normalisee, dafay jëfandikoo echel buñ mëna jàng (gamma) ak shift (beta) suko defee reso bi mëna am bépp représentation bumu soxla.
Gis-gis xarala
Ngir vecteur x bu am màndarga, LayerNorm dafay xayma moyenn bi ak variance bi ci kaw élément vecteur bi, ba noppi génne gamma * (x - moyenne) / sqrt (variance + epsilon) + beta. Ndax lim yi dañu bawoo ci benn misaal, doxalin bi dafay nuru doonte lote bi amna 1 wala 1000 misaal. Benn anam bu gëna yomba, RMSNorm, dafay sànni dindi moyenne bi ba noppi xaaj ko ci root-moyenne-carré bi kese, baña yàq xayma bi; ñu koy jëfandikoo ci model yu melni Llama. Tegtal bi itam amna solo: 'pre-norm' (normalise balaa sublayer bu nekk) dafay tax transformateur yu xóot yi gëna yomba tàggat 'post-norm'.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu normalisasioŋ bu couche
Normalisation ñu ngi koy rataxal ngir gëna am njariñ ci eskaal bi. RMSNorm dafa wecci LayerNorm ci xeetu làkk yu bees ndax moo gëna xéewale te dafay dox bu baax, te plasement pre-norm mooy leegi default ci stack yu xóot lool. Gëstukat yi wéy di jàngat architecture yu amul normalisation yu jëfandikoo initialisation bu baax wala pexe scaling, ngir dagg overhead bi boole ci tëye stabilite tàggat yaram bi normalisation di joxe.
Doxal ci àdduna dëgg
Dakkal bépp bloku transformatër ci xeetu làkk yu melni GPT ak BERT.
Fexe ba RMSNorm nekk tànneef bu gëna woyof ci biir xeetu famiy Llama.
Normalise done yu toppalante yu am guddaay bu mën soppiku ci misaali wax ak tekki làkk, fu tolluwaayu lote yi wuute.
May tàggat bu wóor ak dayo lots bu benn, lu melni ci yenn tabb njàngum dooleel.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
RMSNorm ak yamale bu njëkk
Laaj yi ñuy faral di laaj
What is Layer Normalization?
Normalisasioŋ couche dafay dakkal tàggat yaram ci rescaling activations yi ci biir misaal bu nekk suko defee ñu am zero moyenne ak variance unitaire. Ingredient bu tekkaaral la waaye mën nañu ko tàggat ci transformateur yu xóot yi.
Ci lan la normalisasioŋ couche di xayma moyenne bi ak variance bi?
LayerNorm dafay normalise dimension yi ci benn misaal, moo tax mu moom boppam ci yeneen misaal yi ci dosiye bi.
Lan moo tax ñu taamu normalisasioŋ couche moo gën normalisasioŋ lote ci transformatër yi?
Ndax lim yi dañu bawoo ci benn misaal, LayerNorm dafay doxalee ci anam wu méngoo ak dayo batch bi, te méngoo ak mbind yu am guddaay bu mën soppiku.
Lan la parametre yiñ mëna jàng gamma ak beta may LayerNorm mu def?
Ginaaw ñu normalisee ko ci nul moyenne ak variance unitaire, balance gamma yi ak beta yi dañuy soppi resultaa bi suko defee ñu baña forse model bi mu dugg ci distribution fixe.
Lan moo wuutale RMSNorm ak LayerNorm biñ miin?
RMSNorm bàyyi na jéego bu digg bi ak balans yi ci root-moyenne-carré bi ci aktivasioŋ yi, te loolu moo gëna xéewale te ñu koy jëfandikoo ci model yu melni Llama.
Ban jafe-jafe la normalisation couche di njëkka jàppale ci saafara ci reso yu xóot yi?
Bu siñaal yi di jaar ci diisaay yu bari, seen balans mën na yéeg wala wàññeeku; normalisasioŋ dafay tëye aktivaasioŋ yi ci diggante bu dëgër suko defee gradient yi mëna doxal bu baax.