Speculative Decoding
Speculative decoding makes large language models generate text faster by using a small, fast 'draft' model to guess several tokens ahead, then having the big model verify them all at once.
Overview
It speeds up inference 2-3x with identical output quality.
Deep Dive
Normally an LLM generates text one token at a time: each token requires a full forward pass through the giant model, and you can't start the next until the current one finishes. This is slow because it's memory-bound, not compute-bound — the GPU spends most of its time loading weights, not doing math. Speculative decoding breaks the bottleneck. A small, cheap draft model proposes a chunk of, say, five candidate tokens. The large 'target' model then processes all five in a single parallel forward pass and checks them. Tokens that match what it would have produced are accepted; at the first disagreement it corrects and discards the rest. Because verifying many tokens costs about the same as generating one, accepted guesses are nearly free.
Technical Insight
The clever part is a rejection-sampling rule that guarantees the output distribution is mathematically identical to running the target model alone — so quality is not approximated, it's exact. Acceptance rate drives the speedup: the better the small model predicts the big one, the more tokens stick per verification step. Variants like Medusa add extra prediction heads to the target model itself, and EAGLE drafts in feature space, removing the need for a separate draft model.
Strategic Impact
Cost and budget
Architecture decisions drive performance and operating cost for years.
Clearer decisions
Technical education helps teams choose the right stack, not just the newest one.
Quality control
Better engineering choices reduce reliability incidents in production.
The Future of Speculative Decoding
Speculative decoding is becoming default in serving stacks like vLLM and TensorRT-LLM. Expect self-drafting methods (Medusa, EAGLE, Lookahead) to dominate since they avoid maintaining a second model, plus tree-based speculation that verifies multiple candidate branches per step. As models grow, the memory-bound bottleneck worsens, making speculation even more valuable, and hardware-aware drafters will push real-world speedups higher.
Real-World Implementation
A 7B draft model proposing tokens for a 70B chat model to cut response latency in a production assistant
Medusa heads bolted onto an LLM so it predicts several future tokens at once without a separate draft model
vLLM enabling speculative decoding to raise tokens-per-second throughput on a serving cluster
EAGLE drafting in the model's hidden-feature space to boost the acceptance rate and overall speed
Risks & Guardrails
Optimizing one benchmark can hide broader system weaknesses.
Infrastructure and maintenance costs are often underestimated.
Security and observability gaps can grow as systems become more complex.
Implementation Roadmap
Define latency, quality, and cost targets before implementation.
Benchmark under realistic load and data conditions.
Instrument monitoring for errors, drift, and user impact.
Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Speculative Decoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
Speculative Decoding with EAGLE
Frequently asked questions
What is Speculative Decoding?
Speculative decoding makes large language models generate text faster by using a small, fast 'draft' model to guess several tokens ahead, then having the big model verify them all at once. It speeds up inference 2-3x with identical output quality.
What is the core idea of speculative decoding?
A fast draft model proposes multiple tokens, and the large target model checks them all in a single parallel pass.
Why is normal one-token-at-a-time LLM generation slow?
Each step mainly loads the huge weight matrices from memory rather than doing heavy computation, so the GPU is underused.
What happens at the first token where the draft and target models disagree?
Tokens up to the disagreement are accepted; from the mismatch onward they're rejected and the target model's correct token is kept.
How does speculative decoding affect output quality?
A rejection-sampling rule guarantees the final distribution exactly matches the target model, so quality is unchanged.
What most directly determines the speedup from speculative decoding?
The more drafted tokens the target model accepts per verification step, the greater the speedup.