Expert Parallelism for MoE Serving Quiz

Answers and explanations appear as you take the quiz.

1. What does expert parallelism distribute across GPUs?
   • Copies of the full model for redundancy
   • Individual attention heads only
   • Different feed-forward experts of an MoE layer
   • Layers of the embedding table
2. Which communication pattern is central to MoE expert parallelism?
   • Two all-to-all exchanges (dispatch and combine)
   • A single broadcast
   • Point-to-point gradient sync
   • No communication is needed
3. Why does MoE keep per-token compute low despite huge total parameters?
   • It compresses weights to 1 bit
   • It skips the attention layers
   • Experts share all their weights
   • Only the top-k selected experts run per token
4. What problem arises because routing is data-dependent?
   • Gradient explosion
   • Load imbalance, where some experts get far more tokens
   • The router cannot be trained
   • Tokens lose their position embeddings
5. Why are capacity limits and padding used in some MoE implementations?
   • To control expert buffer sizes and make computation shapes more predictable
   • To require every expert to process every token
   • To eliminate all token routing
   • To guarantee that every MoE system drops tokens
6. How do systems hide the latency of the all-to-all steps?
   • By running on CPU instead
   • By removing the combine step
   • By increasing the learning rate
   • By overlapping communication with expert computation
7. A routed token chooses two MoE experts on different devices. What must expert-parallel execution do?
   • Run every expert in the model regardless of routing
   • Dispatch its representation to the selected experts and combine their outputs
   • Replace both expert networks with an attention head
   • Drop the token solely because it selected two devices
8. One MoE expert receives far more tokens than others. Why can the layer’s latency increase?
   • The total expert parameter count automatically doubles
   • The router’s outputs become attention positions
   • The overloaded expert becomes a straggler while other devices wait
   • All other experts are removed from memory