Mixture of Depths
Mixture of Depths (MoD) lets a transformer spend different amounts of compute on different tokens, routing only the 'important' tokens through each layer's heavy computation.
Overview
It cuts the cost of processing easy tokens while keeping a fixed, predictable compute budget.
Deep Dive
Standard transformers apply every layer to every token, even trivial ones like punctuation. Mixture of Depths, introduced by Google DeepMind in 2024, adds a small router at each block that selects a fixed top-k fraction of tokens to undergo the full self-attention and MLP computation; the rest skip the block via a residual connection. Because only k tokens are processed per layer, the total compute (FLOPs) is capped and known in advance, unlike earlier dynamic-depth methods that varied unpredictably. This makes batching and hardware utilization efficient. MoD-trained models can match a baseline transformer's quality using fewer FLOPs per forward pass, or reach higher quality at the same compute, and the idea composes naturally with Mixture-of-Experts to give 'MoDE' models that route on both depth and width.
Technical Insight
At each MoD block, a learned linear router scores every token and keeps the top-k by score; selected tokens pass through attention and the MLP, while unselected tokens are carried forward unchanged by the residual path. Using a fixed top-k (rather than a per-token threshold) makes the compute graph static and tensor shapes constant, which is hardware-friendly. The router is trained with the rest of the network, and causal generation uses auxiliary predictors so routing decisions don't peek at future tokens.
Strategic Impact
Speed and scale
Language workflows can move faster without sacrificing consistency.
Access and reach
It expands access across languages and communication styles.
Clearer decisions
Teams can spend more time on judgment while automation handles repetition.
The Future of Mixture of Depths
Conditional computation is a major lever for efficiency as models scale, and MoD is an early, clean example. Expect deeper integration with Mixture-of-Experts (routing on both depth and experts), adaptive budgets that shrink for easy inputs, and learned routers that better identify which tokens truly need deep processing. As inference costs dominate deployment economics, techniques that let models 'think harder' only where needed, while keeping predictable latency, are likely to become standard in large-scale architectures.
Real-World Implementation
Reducing the FLOPs needed to process long documents by skipping deep computation on filler tokens
Training a model that matches baseline quality at lower compute, lowering serving cost
Combining with Mixture-of-Experts (MoDE) to route on both layer depth and expert choice
Keeping predictable, fixed latency per token because the per-layer compute budget is fixed in advance
Risks & Guardrails
Hallucinated facts can quietly enter reports, support flows, or research outputs.
Prompt sensitivity can create inconsistent results across similar requests.
Sensitive text data may be exposed if access controls are weak.
Implementation Roadmap
Define output format, tone, and quality standards before rollout.
Ground responses with trusted sources whenever accuracy matters.
Keep a human review checkpoint for high-stakes outputs.
Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Mixture of Depths quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
Mixture-of-Agents Aggregation
Frequently asked questions
What is Mixture of Depths?
Mixture of Depths (MoD) lets a transformer spend different amounts of compute on different tokens, routing only the 'important' tokens through each layer's heavy computation. It cuts the cost of processing easy tokens while keeping a fixed, predictable compute budget.
What does Mixture of Depths vary across tokens?
MoD routes only some tokens through each layer's heavy computation, so different tokens effectively get different depth.
How does MoD keep its compute budget predictable?
Choosing a fixed top-k of tokens per block caps FLOPs and keeps tensor shapes constant, which is hardware-friendly.
What happens to tokens that the router does NOT select at a given block?
Unselected tokens bypass the block's computation and are carried forward unchanged by the residual path.
Who introduced Mixture of Depths?
Mixture of Depths was introduced by Google DeepMind in a 2024 paper on dynamic transformer compute.
What does combining MoD with Mixture-of-Experts produce?
Combining depth routing with expert routing yields 'MoDE' models that condition compute on both layers and experts.