Language AI GUIDE
Mixture of Depths
Mixture of Depths (MoD) lets a transformer spend different amounts of compute on different tokens, routing only the 'important' tokens through each layer's heavy computation.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
It cuts the cost of processing easy tokens while keeping a fixed, predictable compute budget.
Deep Dive
Standard transformers apply every layer to every token, even trivial ones like punctuation. Mixture of Depths, introduced by Google DeepMind in 2024, adds a small router at each block that selects a fixed top-k fraction of tokens to undergo the full self-attention and MLP computation; the rest skip the block via a residual connection. Because only k tokens are processed per layer, the total compute (FLOPs) is capped and known in advance, unlike earlier dynamic-depth methods that varied unpredictably. This makes batching and hardware utilization efficient. MoD-trained models can match a baseline transformer's quality using fewer FLOPs per forward pass, or reach higher quality at the same compute, and the idea composes naturally with Mixture-of-Experts to give 'MoDE' models that route on both depth and width.
Real-World Implementation
Reducing the FLOPs needed to process long documents by skipping deep computation on filler tokens
Training a model that matches baseline quality at lower compute, lowering serving cost
Combining with Mixture-of-Experts (MoDE) to route on both layer depth and expert choice
Keeping predictable, fixed latency per token because the per-layer compute budget is fixed in advance
The Future of Mixture of Depths
Conditional computation is a major lever for efficiency as models scale, and MoD is an early, clean example. Expect deeper integration with Mixture-of-Experts (routing on both depth and experts), adaptive budgets that shrink for easy inputs, and learned routers that better identify which tokens truly need deep processing. As inference costs dominate deployment economics, techniques that let models 'think harder' only where needed, while keeping predictable latency, are likely to become standard in large-scale architectures.
Putting it into practice
Why it matters
- Speed and scale. Language workflows can move faster without sacrificing consistency.
- Access and reach. It expands access across languages and communication styles.
- Clearer decisions. Teams can spend more time on judgment while automation handles repetition.
Watch out for
- Hallucinated facts can quietly enter reports, support flows, or research outputs.
- Prompt sensitivity can create inconsistent results across similar requests.
- Sensitive text data may be exposed if access controls are weak.
Next steps
- Define output format, tone, and quality standards before rollout.
- Ground responses with trusted sources whenever accuracy matters.
- Keep a human review checkpoint for high-stakes outputs.
- Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixture of Depths quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Mixture of Depths?
Mixture of Depths (MoD) lets a transformer spend different amounts of compute on different tokens, routing only the 'important' tokens through each layer's heavy computation. It cuts the cost of processing easy tokens while keeping a fixed, predictable compute budget.
What does Mixture of Depths vary across tokens?
MoD routes only some tokens through each layer's heavy computation, so different tokens effectively get different depth.
How does MoD keep its compute budget predictable?
Choosing a fixed top-k of tokens per block caps FLOPs and keeps tensor shapes constant, which is hardware-friendly.
What happens to tokens that the router does NOT select at a given block?
Unselected tokens bypass the block's computation and are carried forward unchanged by the residual path.
Which lab published the Mixture of Depths technique?
Mixture of Depths was introduced by Google DeepMind in a 2024 paper on dynamic transformer compute.
What does combining MoD with Mixture-of-Experts produce?
Combining depth routing with expert routing yields 'MoDE' models that condition compute on both layers and experts.
Keep learning
Related guides
More guides picked for this topic