SwiGLU and Gated Activations Quiz

Answers and explanations appear as you take the quiz.

1. What core operation defines a Gated Linear Unit?
   • Element-wise multiplication of one projection by an activated second projection
   • Adding two linear projections together
   • Concatenating attention heads
   • Dividing activations by their norm
2. Which activation function does SwiGLU use for its gate branch?
   • ReLU
   • Swish (SiLU)
   • Tanh
   • Softmax
3. How many weight matrices does a SwiGLU feed-forward block use?
   • One
   • Two
   • Three
   • Four
4. Why is the hidden dimension typically scaled to about two-thirds in SwiGLU layers?
   • To improve numerical precision
   • To make the model deeper
   • To remove the need for normalization
   • To keep total parameters and FLOPs comparable to a standard MLP despite the extra matrix
5. A distinctive property of Swish compared to ReLU is that it is:
   • Smooth and non-monotonic, allowing small negatives
   • Strictly increasing and capped at 1
   • Identical to the sigmoid function
   • Only defined for positive inputs
6. Who popularized SwiGLU for transformer feed-forward layers in 2020?
   • Geoffrey Hinton
   • Noam Shazeer
   • Yann LeCun
   • Andrej Karpathy
7. In the SwiGLU product Swish(xW) multiplied by xV, what happens when one gate value is near zero?
   • The entire Transformer layer is skipped
   • The output projection is replaced by a sigmoid
   • The matching value-branch dimension contributes little to the output
   • The value branch is permanently frozen
8. Why can keeping a standard MLP’s hidden width unchanged increase SwiGLU cost?
   • The SiLU gate removes the output projection
   • SwiGLU computes a full Hessian
   • The value branch has no learned weights
   • SwiGLU uses an extra projection matrix, so the comparable-width design would have more parameters and FLOPs