Technical GUIDE
Gating and Routing in Conditional Computation
Gating and routing let a neural network activate only the parts it needs for each input instead of running the whole model every time.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
This decouples model size from compute cost, enabling enormous models that stay fast and cheap to run.
Deep Dive
Conditional computation means the network makes data-dependent decisions about which sub-modules to use. A small learned 'gating' or 'router' network looks at each input (often each token) and produces scores selecting which 'experts' to send it to. In a Mixture-of-Experts (MoE) layer, dozens or hundreds of expert sub-networks exist, but the router picks only the top one or two per token, so most experts stay idle for any given input. The result is a model with a huge total parameter count but a small active count, giving the representational power of a giant model at the runtime cost of a much smaller one. This is how models like the Switch Transformer, GLaM, and many frontier large language models scale to trillions of parameters affordably.
Real-World Implementation
The Switch Transformer routing each token to a single expert, scaling to over a trillion parameters while keeping per-token compute low.
Frontier large language models using Mixture-of-Experts layers so only a fraction of weights activate per token.
Early-exit image classifiers that stop at a shallow layer for easy images and run deeper only for hard ones.
Multilingual models whose routers learn to send tokens from different languages to different specialized experts.
The Future of Gating and Routing in Conditional Computation
Sparse gating is now central to scaling frontier models, and the trend is toward finer-grained experts, smarter routers, and routing at multiple layers. Expect better techniques for stable training, reduced communication overhead when experts are spread across many accelerators, and 'expert specialization' analysis to understand what each expert learns. Conditional computation is also spreading beyond MoE into early-exit networks and dynamic-depth models that spend more compute only on harder inputs.
Putting it into practice
Why it matters
- Cost and budget. Architecture decisions drive performance and operating cost for years.
- Clearer decisions. Technical education helps teams choose the right stack, not just the newest one.
- Quality control. Better engineering choices reduce reliability incidents in production.
Watch out for
- Optimizing one benchmark can hide broader system weaknesses.
- Infrastructure and maintenance costs are often underestimated.
- Security and observability gaps can grow as systems become more complex.
Next steps
- Define latency, quality, and cost targets before implementation.
- Benchmark under realistic load and data conditions.
- Instrument monitoring for errors, drift, and user impact.
- Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Gating and Routing in Conditional Computation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Gating and Routing in Conditional Computation?
Gating and routing let a neural network activate only the parts it needs for each input instead of running the whole model every time. This decouples model size from compute cost, enabling enormous models that stay fast and cheap to run.
What is conditional computation?
The guide defines data-dependent selection of submodules as conditional computation.
In a mixture-of-experts layer, what does the router do?
The guide says a learned router selects experts based on its input.
What is active parameter count?
Sparse routing can activate only part of the total parameter set per input.
Why can routing be computationally efficient?
The guide explains that most experts can remain idle for a particular token.
What is expert load imbalance?
The guide notes routers can favor a few experts, leaving others underused.
Keep learning
Related guides
More guides picked for this topic