Mixture of Experts Quiz

Answers and explanations appear as you take the quiz.

1. In a Mixture of Experts layer, what decides which experts process a given token?
   • A small learned router (gating network) that scores experts
   • A fixed lookup table set before training
   • The user's prompt length
   • A random draw at every step
2. Why can an MoE model have far more total parameters than a dense model without a matching increase in cost per token?
   • It compresses parameters into a single shared weight
   • It skips the attention layers entirely
   • Only a few experts (top-k) are activated for each token
   • It runs every expert but averages the results
3. What problem does a load-balancing (auxiliary) loss address during MoE training?
   • Tokens being too long for the context window
   • The router overusing a few experts and starving the rest
   • Gradients vanishing in attention heads
   • Overfitting to the validation set
4. Mixtral 8x7B activates 2 of its 8 experts per token. What does this imply about its compute versus its size?
   • It uses all 8 experts' compute on every token
   • It has fewer total parameters than a 7B dense model
   • It cannot run on a GPU at all
   • Its active compute per token is far smaller than its total parameter count suggests
5. Why can a sparse MoE still have substantial memory or storage requirements?
   • Every token necessarily activates all experts
   • Inactive experts cease to have parameters
   • Inactive expert weights still need accessible storage even when not selected for a token
   • The router removes all attention layers
6. Two selected MoE expert outputs are 2 and 6, with combination weights 0.25 and 0.75. What is their weighted output?
   • 5
   • 8
   • 2
   • 0.5
7. Increasing an MoE’s expert count keeps the same top-k and expert size. What can still increase serving cost?
   • The selected expert count must automatically become the total count
   • The model must stop using attention
   • Every inactive weight is automatically deleted
   • Router or communication overhead and the storage required for more experts
8. A sparse MoE calls a learned feed-forward subnetwork an expert. What does that name establish?
   • It is certified to answer only one academic discipline
   • It is one of the networks selected by routing, without proving a named subject specialty
   • It must be a human-written collection of rules
   • It contains no learned parameters