AI Glossary Term

What does “Mixture of Experts (MoE)” mean?

Definition

An architecture with specialized subnetworks where only selected experts run per input.

Browse the full AI Glossary Explore all free AI guides

How Mixture of Experts (MoE) works

Example

A model may hold many expert subnetworks but send each token to only two of them. This way, it has a very large total number of parameters while using only a fraction of them for each token.

Why it matters

MoE lets models grow in capacity without a proportional rise in compute per token. It makes large models cheaper to run, though they still need lots of memory.

Common questions

Do experts specialize in topics?

Not necessarily in human-readable ways. They tend to specialize in patterns that help the model, which may not match neat subjects.

What is the router?

A small network that scores the experts and decides which ones process each token.

What is the trade-off?

All experts must be stored in memory, and balancing the load across them adds training complexity.

ClassificationA task where a model assigns an input to one or more predefined categories.Sliding Window AttentionAn attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.Attention MechanismA model component that dynamically focuses on relevant parts of an input when producing an output.Chain-of-ThoughtA reasoning style where an AI model decomposes a problem into intermediate steps.CLIPA multimodal model architecture that learns shared representations between text and images.ComputeThe processing resources required to train and run models, often measured in FLOPS or GPU hours.

Learn more in our free guides

Expert Parallelism for MoE Serving Mixture-of-Agents Aggregation Mixture of Depths Mixture of LoRA Experts

Test yourself

Expert Parallelism for MoE Serving Quiz Mixture-of-Agents Aggregation Quiz Mixture of Depths Quiz