Ikibazo gihuza ibibazo · Biragoye Urwego
RMSNorm na Pre-Layeri Ubusanzwe Ikibazo
Kugenzura imibare idasanzwe ya RMS, Gushyira mbere ya LN, inzira zisigaye hamwe nibibazo-urufunguzo rusanzwe utitiranya amahitamo yigenga.
Ikibazo 1 Bya 8
Niki RMSNorm ikuraho ugereranije na Layeri isanzwe?
Reba buri kibazo8 ibibazo
Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.
- Niki RMSNorm ikuraho ugereranije na Layeri isanzwe?
- Gukuramo gukuramo (re-centering)
- Igipimo cyize
- Ihuza risigaye
- Igikorwa cya softmax
- Nuwuhe mubare RMSNorm igabanya buri gikorwa cyo gukora?
- Itandukaniro ryayo
- Imizi yacyo isobanura kare
- Agaciro ntarengwa ntarengwa
- L1 isanzwe
- What gradient behavior does Pre-LN support at initialization in deep Transformers?
- It guarantees stable training for every model without tuning
- It removes the residual stream from the block
- It can make gradients better behaved at initialization and reduce some stability problems
- It makes all gradients exactly equal across layers
- Mubice byabanjirije LN, ni ubuhe buryo busanzwe urwego rusanzwe rusiga nkana?
- Ibiribwa-byimbere
- Ikimenyetso
- Ibisohoka softmax
- Ihuza risigaye (indangamuntu)
- Niyihe miryango igezweho yuburemere yuburemere ikoresha RMSNorm kubisanzwe?
- LLaMA, Mistral, na Qwen
- AlexNet na VGG
- ResNet na Intangiriro
- BERT-shingiro na Transformer yumwimerere
- Niki 'QK-norm,' ni ubuhe buryo bwa RMSNorm?
- Kumenyekanisha ibisohoka byanyuma
- Koresha RMSNorm kubibazo byibanze nurufunguzo rwo kugenzura imikurire ya logit
- Kugabanya urufunguzo-agaciro cache
- Inzira yo gusimbuka bisanzwe
- A feature vector has a nonzero mean before RMSNorm. What does the normalization do to that mean?
- It always sets the mean exactly to zero
- It replaces each feature with its rank
- It does not explicitly subtract the mean, though rescaling can change its value
- It subtracts the maximum activation from every feature
- How are RMSNorm and Pre-LN related as design choices?
- They are two names for subtracting the mean
- RMSNorm always means normalization after the residual
- Pre-LN specifies the optimizer rather than the block
- RMSNorm specifies the normalization calculation; Pre-LN specifies where normalization sits in the residual block