Maswali yanayounganishwa na mwongozo · Ngumu Kiwango

Maswali ya Kusawazisha ya RMSNorm na Kabla ya Tabaka

Hukagua takwimu za RMSNorm, uwekaji wa Pre-LN, njia za mabaki na urekebishaji wa ufunguo wa hoja bila kuchanganya chaguo hizi huru za muundo.

Njia za mwongozo zinazohusianaRmsnorm na Urekebishaji wa Kabla ya Tabaka
Swali 1 ya 8

Je, RMSNorm inaacha nini ikilinganishwa na LayerNorm ya kawaida?

Hakiki kila swali8 maswali

Majibu na maelezo huonekana unapojibu maswali.

  1. Je, RMSNorm inaacha nini ikilinganishwa na LayerNorm ya kawaida?
    • Maana ya kutoa (kuweka katikati tena)
    • Kigezo cha mizani iliyojifunza
    • Uunganisho wa mabaki
    • Uendeshaji wa softmax
  2. Je, RMSNorm inagawanya kila vekta ya kuwezesha kwa kiasi gani?
    • Tofauti yake
    • Mzizi wake unamaanisha mraba
    • Thamani yake ya juu kabisa
    • Kawaida yake ya L1
  3. What gradient behavior does Pre-LN support at initialization in deep Transformers?
    • It guarantees stable training for every model without tuning
    • It removes the residual stream from the block
    • It can make gradients better behaved at initialization and reduce some stability problems
    • It makes all gradients exactly equal across layers
  4. Katika kizuizi cha Pre-LN, ni njia gani ambayo safu ya urekebishaji huacha kwa makusudi bila kuguswa?
    • Vipimo vya kulisha-mbele
    • Upachikaji wa ishara
    • Pato la softmax
    • Muunganisho wa mabaki (kitambulisho).
  5. Ni familia zipi za kisasa za uzani wazi hutumia RMSNorm kwa chaguo-msingi?
    • LLaMA, Mistral, na Qwen
    • AlexNet na VGG
    • ResNet na Kuanzishwa
    • BERT-base na Transformer asili
  6. 'QK-kawaida' ni nini, matumizi ya hivi majuzi ya RMSNorm?
    • Kurekebisha kumbukumbu za matokeo ya mwisho
    • Kutumia RMSNorm kwa hoja za umakini na funguo ili kudhibiti ukuaji wa kumbukumbu
    • Kuhesabu akiba ya thamani-msingi
    • Njia ya kuruka hali ya kawaida kabisa
  7. A feature vector has a nonzero mean before RMSNorm. What does the normalization do to that mean?
    • It always sets the mean exactly to zero
    • It replaces each feature with its rank
    • It does not explicitly subtract the mean, though rescaling can change its value
    • It subtracts the maximum activation from every feature
  8. How are RMSNorm and Pre-LN related as design choices?
    • They are two names for subtracting the mean
    • RMSNorm always means normalization after the residual
    • Pre-LN specifies the optimizer rather than the block
    • RMSNorm specifies the normalization calculation; Pre-LN specifies where normalization sits in the residual block