Speculative Decoding and Draft Models

Large language models write text one token at a time, where a token is a word or word fragment. Each new token needs a full forward pass through the model, a computation that reads all of its billions of weights from memory. Speculative decoding is a way to produce several tokens per expensive pass. A small, cheap model, called the draft model or drafter, quickly guesses the next few tokens. The large model, called the target model, then scores all of those guesses at once and keeps the ones it agrees with. When the guesses are good, the large model has effectively written several tokens for the price of one pass.

The method works step by step. First the drafter generates a short run of candidate tokens, say four or five, one after another; this is cheap because the drafter is small. Second, the target model runs one forward pass over the prompt plus all candidates together. Because Transformers process a whole sequence in parallel, this single pass yields the target's own probability for every candidate position. Third, each candidate is accepted or rejected in order using a rule that compares the target's probability with the drafter's. Fourth, at the first rejection the remaining guesses are thrown away and the target supplies a replacement token from a corrected distribution. If every guess is accepted, the target also contributes one bonus token. The loop then repeats from the new position.

The speedup comes from a hardware fact. Google researchers explained in a December 2024 retrospective that inference is usually limited by memory bandwidth, not arithmetic: chips can perform hundreds of operations per byte they read, while Transformer decoding performs only a few. Checking five tokens therefore costs about the same wall-clock time as generating one, because the weights are read from memory once either way. The second fact is that many tokens are easy. After "the United States of", almost any model predicts "America", so a tiny drafter is often right.

The idea was published twice in quick succession. Yaniv Leviathan, Matan Kalman and Yossi Matias of Google posted "Fast Inference from Transformers via Speculative Decoding" in November 2022 (ICML 2023), reporting 2 to 3 times faster decoding of the 11-billion-parameter T5-XXL with identical outputs; their retrospective cites about 3 times on translation using the 60-million-parameter T5-small as drafter. In February 2023 a DeepMind team led by Charlie Chen independently described speculative sampling and sped up the 70-billion-parameter Chinchilla 2 to 2.5 times. Hugging Face shipped the technique as "assisted generation" in Transformers in May 2023. Later work replaced the separate drafter: Medusa (January 2024) adds extra prediction heads to the target model and reports 2.2 to 3.6 times speedups, and EAGLE (January 2024) trains a light drafter on the target's internal features, reporting 2.7 to 3.5 times lower latency on LLaMA2-Chat 70B.

Compared with the alternatives, speculative decoding is unusual because it is lossless. Quantization (storing weights at lower precision), distillation (training a smaller model to imitate a large one) and pruning all make inference cheaper by changing the model, which can change its answers. Speculative decoding leaves the target model untouched, and the acceptance rule guarantees the same output distribution. vLLM's documentation adds a practical caveat: results are identical in theory but can differ slightly in practice because of floating-point precision and batching. Some variants trade this guarantee for speed; Medusa-2 fine-tunes the backbone, and vLLM labels its FLy verification mode as lossy.

The main limitation is that the gain depends entirely on the acceptance rate, the share of drafted tokens the target keeps. A drafter that disagrees often wastes work, so open-ended creative writing at high temperature gains less than code completion or summarization. Hugging Face found speedups of up to 2 times in ordinary GPU setups (3 times with 8-bit weights) and up to 10 times when the model was too large for GPU memory, but warned that the method is not a cure-all and should be benchmarked. Load matters too. At high request rates the GPU is already busy with large batches, spare compute shrinks, and vLLM's guide rates the benefit of draft models in that regime as only medium. Its first Transformers release supported only a batch size of one.

In practice a separate drafter must share the target's tokenizer, meaning the same mapping from text to token IDs, so that both models are guessing the same units; Hugging Face recommends a drafter at least ten times smaller. It also costs GPU memory to hold both models, plus engineering time to tune the number of drafted tokens. Model families that ship in several sizes, such as a 1-billion and a 70-billion version trained on the same data, make natural pairs. Feature-based drafters such as EAGLE need a short training run on the target model but no tokenizer search, and n-gram methods, which copy likely continuations from the prompt itself, need no extra model at all.