Layer Normalization Quiz

Answers and explanations appear as you take the quiz.

1. Across what does layer normalization compute its mean and variance?
   • Across the features of a single example
   • Across all examples in the mini-batch
   • Across all weights in the network
   • Across multiple training epochs
2. Why is layer normalization preferred over batch normalization in transformers?
   • It requires a very large batch size
   • It is independent of batch size and handles variable-length sequences
   • It eliminates the need for any learnable parameters
   • It only works during inference
3. What do the learnable parameters gamma and beta let LayerNorm do?
   • Choose the learning rate
   • Set the number of layers
   • Rescale and shift the normalized output so the network can recover useful representations
   • Remove the need for an activation function
4. How does RMSNorm differ from standard LayerNorm?
   • It normalizes across the batch instead
   • It uses no parameters and no division
   • It only works on images
   • It skips mean subtraction and divides only by the root-mean-square
5. What problem does layer normalization primarily help solve in deep networks?
   • Activations drifting to very different scales, which destabilizes training
   • Running out of disk space
   • Choosing the right dataset
   • Encrypting the model weights
6. Why is 'pre-norm' placement often used in very deep transformers?
   • It makes the model smaller
   • Normalizing before each sublayer makes deep stacks easier and more stable to train
   • It removes the attention mechanism
   • It doubles the learning rate automatically
7. Why is a small epsilon added to LayerNorm’s variance before taking its square root?
   • To make the batch size larger
   • To replace the learnable scale
   • To avoid division by zero and improve numerical stability
   • To compute the sequence length
8. If gamma is all ones and beta is all zeros, what does LayerNorm’s affine stage do?
   • It restores the original unnormalized inputs exactly
   • It applies a softmax over features
   • It changes the statistics to be batch-dependent
   • It leaves the normalized values unchanged after the mean-and-variance scaling