Speculative Decoding Draft Models
Speculative decoding uses a small, fast 'draft' model to guess several upcoming tokens that a large model then verifies in one pass.
Overview
It speeds up text generation 2-3x with no change to the output.
Deep Dive
Large language models generate text one token at a time, and each step requires a full forward pass through billions of parameters — slow and memory-bound. Speculative decoding attacks this by pairing the big 'target' model with a cheap 'draft' model. The draft model rapidly proposes a chunk of, say, 4-8 candidate tokens. The big model then processes all of them in a single parallel forward pass and checks each one. Tokens that match what the big model would have produced are accepted; the first mismatch is corrected and the rest discarded. Because verifying several tokens at once costs roughly the same as generating one, accepted runs are nearly free. Crucially, a rejection-sampling step guarantees the final distribution is identical to running the big model alone — speed without quality loss.
Technical Insight
The key trick is a modified rejection-sampling test. For each drafted token, the target model's probability is compared to the draft model's. If the target assigns equal or higher probability, the token is accepted; otherwise it is accepted with probability equal to the ratio, and on rejection a corrected token is sampled from an adjusted residual distribution. This math makes the output provably equivalent to sampling directly from the large model.
Strategic Impact
Speed and scale
Language workflows can move faster without sacrificing consistency.
Access and reach
It expands access across languages and communication styles.
Clearer decisions
Teams can spend more time on judgment while automation handles repetition.
The Future of Speculative Decoding Draft Models
Expect draft models to become standard infrastructure in inference servers like vLLM and TensorRT-LLM. Self-speculation variants (Medusa, EAGLE) drop the separate draft model entirely by adding lightweight prediction heads, and tree-based drafting verifies many candidate continuations at once. As context windows grow and serving costs dominate, smarter, model-matched drafters and hardware-aware verification will push acceptance rates and throughput higher.
Real-World Implementation
Anthropic, OpenAI, and Google use speculative decoding to cut latency and serving cost on chat assistants serving millions of users.
vLLM and NVIDIA TensorRT-LLM ship built-in speculative decoding so self-hosters can speed up Llama or Mistral deployments.
Pairing a 7B draft model with a 70B target (e.g., Llama-3 family) to roughly double tokens-per-second on a single GPU.
Code-completion tools use a tiny draft model to propose boilerplate that the larger model verifies, keeping suggestions snappy in the editor.
Risks & Guardrails
Hallucinated facts can quietly enter reports, support flows, or research outputs.
Prompt sensitivity can create inconsistent results across similar requests.
Sensitive text data may be exposed if access controls are weak.
Implementation Roadmap
Define output format, tone, and quality standards before rollout.
Ground responses with trusted sources whenever accuracy matters.
Keep a human review checkpoint for high-stakes outputs.
Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Speculative Decoding Draft Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
Speculative Edits for Code Models
Frequently asked questions
What is Speculative Decoding Draft Models?
Speculative decoding uses a small, fast 'draft' model to guess several upcoming tokens that a large model then verifies in one pass. It speeds up text generation 2-3x with no change to the output.
What is the primary role of the 'draft' model in speculative decoding?
The small draft model cheaply guesses upcoming tokens, which the large target model then checks in a single parallel pass.
Why does verifying multiple drafted tokens at once save time?
Generation is memory-bound; checking several tokens in one batched pass is nearly as cheap as one step, so accepted runs are almost free.
What happens to the drafted tokens after the first token the target model rejects?
Acceptance proceeds until the first disagreement; that token is corrected and all subsequent drafted tokens are thrown away.
How does speculative decoding ensure output quality is not degraded?
A modified rejection-sampling test guarantees the final token distribution matches sampling directly from the target model.
Which of these is a 'self-speculation' approach that avoids a separate draft model?
Medusa and EAGLE add lightweight extra prediction heads to the main model so it can draft its own future tokens.