Kedis ku xidhan hagaha · Adag Heerka

RMSNorm iyo Kediska Caadiga Hore ee Lakabka

Wuxuu hubiyaa tirakoobka RMSNorm, meelaynta Pre-LN, dariiqyada hadhaaga ah iyo su'aalaha-furaha caadiga ah iyada oo aan la jahawareerin doorashooyinkan naqshadaynta madaxa banaan.

Wadooyinka haga ee la xidhiidhaRmsnorm iyo Kahor Caadiga Lakabka
Su'aal 1 ee 8

Muxuu RMSNorm ka tagayaa marka la barbar dhigo heerka LayerNorm?

Horudhac su'aal kasta8 su'aalo

Jawaabaha iyo sharraxaadaha waxay soo baxayaan markaad qaadanayso kediska.

  1. Muxuu RMSNorm ka tagayaa marka la barbar dhigo heerka LayerNorm?
    • Kala-goynta dhexdhexaadka ah (dib-u-dhexaynta)
    • Halbeegga cabbirka la bartay
    • Isku xirka haraaga
    • Hawlgalka softmax
  2. Immisa ayay RMSNorm u qaybinaysaa vector kasta oo firfircooni?
    • Kala duwanaanshaheeda
    • Xididkiisu waa afargeesle
    • Qiimaha ugu sarreeya ee ugu sarreeya
    • Waa L1 caadi ah
  3. What gradient behavior does Pre-LN support at initialization in deep Transformers?
    • It guarantees stable training for every model without tuning
    • It removes the residual stream from the block
    • It can make gradients better behaved at initialization and reduce some stability problems
    • It makes all gradients exactly equal across layers
  4. Gudaha ka hor-LN-ka, jidkee buu lakabka caadiga ka dhigayaa si badheedh ah oo aan la taaban?
    • Miisaanka quudinta-horumarineed
    • Ku-xidhnaanta calaamadda
    • Soo saarida softmax
    • Isku xirka haraaga (aqoonsiga).
  5. Qoysaskee moodeelka furfuran ee casriga ah ayaa si caadi ah u isticmaala RMSNorm?
    • LLAMA, Mistral, iyo Qwen
    • AlexNet iyo VGG
    • ResNet iyo Inception
    • BERT-saldhig iyo Transformer-kii asalka ahaa
  6. Waa maxay 'QK-norm,' codsigii u dambeeyay ee RMSNorm?
    • Caadiyan saadka wax soo saarka kama dambaysta ah
    • Dalbashada RMSNorm su'aalaha dareenka iyo furayaasha si loo xakameeyo kobaca saadka
    • Tirinta kaydka-qiimaha furaha
    • Habka looga boodo caadinimada gabi ahaanba
  7. A feature vector has a nonzero mean before RMSNorm. What does the normalization do to that mean?
    • It always sets the mean exactly to zero
    • It replaces each feature with its rank
    • It does not explicitly subtract the mean, though rescaling can change its value
    • It subtracts the maximum activation from every feature
  8. How are RMSNorm and Pre-LN related as design choices?
    • They are two names for subtracting the mean
    • RMSNorm always means normalization after the residual
    • Pre-LN specifies the optimizer rather than the block
    • RMSNorm specifies the normalization calculation; Pre-LN specifies where normalization sits in the residual block