RMSNorm and Pre-Layer Normalization Quiz

Answers and explanations appear as you take the quiz.

1. What does RMSNorm omit compared to standard LayerNorm?
   • Mean subtraction (re-centering)
   • The learned scale parameter
   • The residual connection
   • The softmax operation
2. By what quantity does RMSNorm divide each activation vector?
   • Its variance
   • Its root mean square
   • Its maximum absolute value
   • Its L1 norm
3. What gradient behavior does Pre-LN support at initialization in deep Transformers?
   • It guarantees stable training for every model without tuning
   • It removes the residual stream from the block
   • It can make gradients better behaved at initialization and reduce some stability problems
   • It makes all gradients exactly equal across layers
4. In a Pre-LN block, what path does the normalization layer deliberately leave untouched?
   • The feed-forward weights
   • The token embeddings
   • The output softmax
   • The residual (identity) connection
5. Which modern open-weight model families use RMSNorm by default?
   • LLaMA, Mistral, and Qwen
   • AlexNet and VGG
   • ResNet and Inception
   • BERT-base and the original Transformer
6. What is 'QK-norm,' a recent application of RMSNorm?
   • Normalizing the final output logits
   • Applying RMSNorm to attention queries and keys to control logit growth
   • Quantizing the key-value cache
   • A way to skip normalization entirely
7. A feature vector has a nonzero mean before RMSNorm. What does the normalization do to that mean?
   • It always sets the mean exactly to zero
   • It replaces each feature with its rank
   • It does not explicitly subtract the mean, though rescaling can change its value
   • It subtracts the maximum activation from every feature
8. How are RMSNorm and Pre-LN related as design choices?
   • They are two names for subtracting the mean
   • RMSNorm always means normalization after the residual
   • Pre-LN specifies the optimizer rather than the block
   • RMSNorm specifies the normalization calculation; Pre-LN specifies where normalization sits in the residual block