Mixture of Depths Quiz

Answers and explanations appear as you take the quiz.

1. What does Mixture of Depths vary across tokens?
   • How much compute (which layers) each token receives
   • The vocabulary used
   • The training dataset
   • The output language
2. How does MoD keep its compute budget predictable?
   • By randomly dropping half the model weights
   • By processing a fixed top-k fraction of tokens at each block
   • By using a different model for each sentence
   • By disabling attention entirely
3. What happens to tokens that the router does NOT select at a given block?
   • They are deleted from the sequence
   • They are sent to a separate model
   • They skip the block via the residual connection, passing through unchanged
   • They are converted to images
4. Which lab published the Mixture of Depths technique?
   • OpenAI in 2019
   • Meta in 2015
   • Microsoft in 2012
   • Google DeepMind in 2024
5. What does combining MoD with Mixture-of-Experts produce?
   • MoDE, which routes on both depth and expert choice
   • A model with no parameters
   • A pure convolutional network
   • A tokenizer-only system
6. Why is a fixed top-k preferred over a per-token threshold in MoD?
   • It makes the model output random text
   • It keeps tensor shapes static, which is efficient on hardware
   • It removes the need for training
   • It increases the vocabulary size
7. What does the learned router do at a Mixture-of-Depths block?
   • Selects a new model family for every prompt
   • Predicts labels for the next training batch
   • Scores tokens so the block can select which ones receive full attention and MLP computation
   • Changes the vocabulary size at inference
8. Why does causal generation in MoD use auxiliary predictors for routing?
   • To let a model read the entire answer before generating
   • To remove the fixed top-k compute budget
   • To convert the sequence into an image
   • To make routing decisions without looking at future tokens