AI Glossary Term
What does “Speculative Decoding” mean?
An inference acceleration method where a small draft model proposes tokens that a larger model verifies in parallel.
How Speculative Decoding works
Example
A fast small model drafts 'The capital of France is Paris'. The large model verifies the whole phrase in a single pass instead of generating one token at a time, and keeps every token it agrees with.
Why it matters
It speeds up generation without changing the large model's output distribution when done correctly. The gain depends on how often the draft guesses right.
Common questions
Does it change the answer quality?
In its standard form, no: the verification step ensures outputs match what the large model would produce.
Why is it faster?
Checking several tokens at once uses hardware more efficiently than producing them one by one.
What does it need?
A suitable smaller draft model, or a built-in drafting method, and extra memory.