Ikibazo gihuza ibibazo · Biragoye Urwego

RMSNorm na Pre-Layeri Ubusanzwe Ikibazo

Kugenzura imibare idasanzwe ya RMS, Gushyira mbere ya LN, inzira zisigaye hamwe nibibazo-urufunguzo rusanzwe utitiranya amahitamo yigenga.

Inzira zijyanye no kuyoboraRmsnorm Na Pre Layeri Ubusanzwe
Ikibazo 1 Bya 8

Niki RMSNorm ikuraho ugereranije na Layeri isanzwe?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Niki RMSNorm ikuraho ugereranije na Layeri isanzwe?
    • Gukuramo gukuramo (re-centering)
    • Igipimo cyize
    • Ihuza risigaye
    • Igikorwa cya softmax
  2. Nuwuhe mubare RMSNorm igabanya buri gikorwa cyo gukora?
    • Itandukaniro ryayo
    • Imizi yacyo isobanura kare
    • Agaciro ntarengwa ntarengwa
    • L1 isanzwe
  3. What gradient behavior does Pre-LN support at initialization in deep Transformers?
    • It guarantees stable training for every model without tuning
    • It removes the residual stream from the block
    • It can make gradients better behaved at initialization and reduce some stability problems
    • It makes all gradients exactly equal across layers
  4. Mubice byabanjirije LN, ni ubuhe buryo busanzwe urwego rusanzwe rusiga nkana?
    • Ibiribwa-byimbere
    • Ikimenyetso
    • Ibisohoka softmax
    • Ihuza risigaye (indangamuntu)
  5. Niyihe miryango igezweho yuburemere yuburemere ikoresha RMSNorm kubisanzwe?
    • LLaMA, Mistral, na Qwen
    • AlexNet na VGG
    • ResNet na Intangiriro
    • BERT-shingiro na Transformer yumwimerere
  6. Niki 'QK-norm,' ni ubuhe buryo bwa RMSNorm?
    • Kumenyekanisha ibisohoka byanyuma
    • Koresha RMSNorm kubibazo byibanze nurufunguzo rwo kugenzura imikurire ya logit
    • Kugabanya urufunguzo-agaciro cache
    • Inzira yo gusimbuka bisanzwe
  7. A feature vector has a nonzero mean before RMSNorm. What does the normalization do to that mean?
    • It always sets the mean exactly to zero
    • It replaces each feature with its rank
    • It does not explicitly subtract the mean, though rescaling can change its value
    • It subtracts the maximum activation from every feature
  8. How are RMSNorm and Pre-LN related as design choices?
    • They are two names for subtracting the mean
    • RMSNorm always means normalization after the residual
    • Pre-LN specifies the optimizer rather than the block
    • RMSNorm specifies the normalization calculation; Pre-LN specifies where normalization sits in the residual block