概述
使用標準接受規則,輸出的機率分佈與單獨的大型模型完全相同,因此產生的文字速度更快且品質沒有改變,在已發布的測試中通常快兩到三倍。
重點摘要
- 推測解碼讓小型草稿模型猜測幾個 token,並由大型目標模型在一次前向傳遞中進行驗證。
- 使用標準接受規則,輸出分佈在數學上與僅從大型模型取樣的結果完全相同。
- Google 的 2022 年論文報告稱 T5-XXL 解碼速度提升了 2 到 3 倍,而 DeepMind 則將 700 億參數的 Chinchilla 模型速度提升了 2 到 2.5 倍。
- 這之所以可行,是因為語言模型的推論受限於記憶體頻寬,因此有剩餘的運算能力可同時檢查多個 Token。
- 加速效果取決於接受率,因此摘要和程式碼編寫等可預測性高的任務,其加速效果會比高溫度的創意寫作更顯著。
深入探討
Large language models write text one token at a time, where a token is a word or word fragment. Each new token needs a full forward pass through the model, a computation that reads all of its billions of weights from memory. Speculative decoding is a way to produce several tokens per expensive pass. A small, cheap model, called the draft model or drafter, quickly guesses the next few tokens. The large model, called the target model, then scores all of those guesses at once and keeps the ones it agrees with. When the guesses are good, the large model has effectively written several tokens for the price of one pass.
The method works step by step. First the drafter generates a short run of candidate tokens, say four or five, one after another; this is cheap because the drafter is small. Second, the target model runs one forward pass over the prompt plus all candidates together. Because Transformers process a whole sequence in parallel, this single pass yields the target's own probability for every candidate position. Third, each candidate is accepted or rejected in order using a rule that compares the target's probability with the drafter's. Fourth, at the first rejection the remaining guesses are thrown away and the target supplies a replacement token from a corrected distribution. If every guess is accepted, the target also contributes one bonus token. The loop then repeats from the new position.
The speedup comes from a hardware fact. Google researchers explained in a December 2024 retrospective that inference is usually limited by memory bandwidth, not arithmetic: chips can perform hundreds of operations per byte they read, while Transformer decoding performs only a few. Checking five tokens therefore costs about the same wall-clock time as generating one, because the weights are read from memory once either way. The second fact is that many tokens are easy. After "the United States of", almost any model predicts "America", so a tiny drafter is often right.
The idea was published twice in quick succession. Yaniv Leviathan, Matan Kalman and Yossi Matias of Google posted "Fast Inference from Transformers via Speculative Decoding" in November 2022 (ICML 2023), reporting 2 to 3 times faster decoding of the 11-billion-parameter T5-XXL with identical outputs; their retrospective cites about 3 times on translation using the 60-million-parameter T5-small as drafter. In February 2023 a DeepMind team led by Charlie Chen independently described speculative sampling and sped up the 70-billion-parameter Chinchilla 2 to 2.5 times. Hugging Face shipped the technique as "assisted generation" in Transformers in May 2023. Later work replaced the separate drafter: Medusa (January 2024) adds extra prediction heads to the target model and reports 2.2 to 3.6 times speedups, and EAGLE (January 2024) trains a light drafter on the target's internal features, reporting 2.7 to 3.5 times lower latency on LLaMA2-Chat 70B.
Compared with the alternatives, speculative decoding is unusual because it is lossless. Quantization (storing weights at lower precision), distillation (training a smaller model to imitate a large one) and pruning all make inference cheaper by changing the model, which can change its answers. Speculative decoding leaves the target model untouched, and the acceptance rule guarantees the same output distribution. vLLM's documentation adds a practical caveat: results are identical in theory but can differ slightly in practice because of floating-point precision and batching. Some variants trade this guarantee for speed; Medusa-2 fine-tunes the backbone, and vLLM labels its FLy verification mode as lossy.
The main limitation is that the gain depends entirely on the acceptance rate, the share of drafted tokens the target keeps. A drafter that disagrees often wastes work, so open-ended creative writing at high temperature gains less than code completion or summarization. Hugging Face found speedups of up to 2 times in ordinary GPU setups (3 times with 8-bit weights) and up to 10 times when the model was too large for GPU memory, but warned that the method is not a cure-all and should be benchmarked. Load matters too. At high request rates the GPU is already busy with large batches, spare compute shrinks, and vLLM's guide rates the benefit of draft models in that regime as only medium. Its first Transformers release supported only a batch size of one.
In practice a separate drafter must share the target's tokenizer, meaning the same mapping from text to token IDs, so that both models are guessing the same units; Hugging Face recommends a drafter at least ten times smaller. It also costs GPU memory to hold both models, plus engineering time to tune the number of drafted tokens. Model families that ship in several sizes, such as a 1-billion and a 70-billion version trained on the same data, make natural pairs. Feature-based drafters such as EAGLE need a short training run on the target model but no tokenizer search, and n-gram methods, which copy likely continuations from the prompt itself, need no extra model at all.
04工作範例
追蹤一輪推測解碼
從提示詞「The capital of France is」開始,讓草稿模型提出四個 Token:「Paris」、「,」、「a」、「city」。
將提示詞加上所有四個候選 Token 執行目標模型一次,以取得其在四個位置中每一個位置的機率。
檢查「Paris」:草稿模型給出的機率是 0.90,目標模型是 0.95,因此 p/q 大於 1 並被接受。
檢查「,」:草稿模型給出 0.50,目標模型 0.40,因此以 0.8 的機率接受;假設隨機抽樣接受了它。
檢查「a」:草稿模型給出 0.30,目標模型 0.06,因此以 0.2 的機率接受;假設它被拒絕。
捨棄「city」,從剩餘值 max(0, p - q) 中抽樣一個替代項目,例如「which」,然後從「Paris, which」開始下一輪。
它顯示了什麼
一次目標模型傳遞產生了三個 Token 而非一個。為了驗證邏輯,使用隨機亂數種子重複多次此回合,並確認每個最終 Token 的頻率與目標模型的機率相符,這就是無損保證的實際運作。
現實世界的實施
Google 在多個產品中應用了推測解碼,包括 Google 搜尋中的 AI 總覽(AI Overviews),其研究人員在 2024 年 12 月表示,現在傳回結果的速度更快,且回應品質相同。
在 Hugging Face Transformers 中,將諸如 pythia-160m 的小型模型作為 assistant_model 傳遞給 generate(),即可為 pythia-1.4b 草擬 Token,前提是兩者共用完全相同的 Tokenizer。
vLLM 推論伺服器支援草稿模型、EAGLE、多 Token 預測頭、n-gram 與字尾解碼,這些功能是透過具有 num_speculative_tokens 的 speculative_config 設定進行配置。
DeepMind 於 2023 年發表的推測取樣論文在分散式設定中,在不改變其輸出的情況下,將 70 億參數的 Chinchilla 模型速度提升了 2 到 2.5 倍。
根據 Hugging Face 的輔助生成說明文件,語音辨識與摘要能獲得顯著的效益,因為其輸出緊密跟隨輸入,這使得小型草稿模型很容易猜出下一個 Token。
推測解碼與草稿模型的未來
Research since 2024 has moved from separate draft models toward drafters built into or trained on the target model. EAGLE-3, posted in March 2025, reports up to 6.5 times speedup and a 1.38 times throughput gain at batch size 64 in the SGLang serving framework, addressing the older weakness that speculation helped little under heavy load. Serving engines are adding dynamic speculation that changes the drafted length per request based on measured acceptance; vLLM's documentation in 2026 lists dynamic speculative decoding, adaptive verification and per-request acceptance metrics. Open problems remain. Gains shrink for high-temperature sampling and for very large batches, keeping two models in memory competes with space for longer contexts, and lossy variants that relax verification for extra speed blur the original guarantee that outputs are unchanged. Google's 2024 retrospective notes that the same draft-and-verify idea has spread to image and speech generation. Whether future models will be trained from the start with built-in drafting heads, removing the need for a separate drafter, is an active direction rather than a settled outcome.
付諸實踐
為什麼這很重要
- 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
- 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
- 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。
留意
- 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
- 及時的敏感性可能會在類似的請求中產生不一致的結果。
- 如果存取控制薄弱,敏感文字資料可能會暴露。
後續步驟
- 在推出之前定義輸出格式、語氣和品質標準。
- 當準確性很重要時,請使用可信任來源進行地面回應。
- 為高風險輸出保留人工審查檢查點。
- 追蹤故障模式並定期重新訓練提示或工作流程。
資料來源與延伸閱讀
- arXiv (ICML 2023)透過投機解碼實現 Transformer 的快速推論
- arXiv (DeepMind)加速大型語言模型解碼的投機抽樣方法
- Google Research回顧推測解碼
- Hugging Face協助生成:低延遲文本生成的新方向
- vLLM Documentation推測性解碼
- arXivMedusa:帶多個解碼頭的簡易LLM推理加速框架
- arXivEAGLE:投機抽樣需要重新思考特徵不確定性
- arXivEAGLE-3:通過訓練時測試擴大大型語言模型的推理加速
不斷探索
常見問題
什麼是推測解碼(Speculative Decoding)與草稿模型(Draft Models)?
推測式解碼(Speculative decoding)透過讓小型、快速的草稿模型猜測多個即將出現的 Token,再由大型目標模型在單次傳遞中進行檢查,來加速大型語言模型。在標準接受規則下,輸出結果遵循與單獨使用大型模型完全相同的機率分佈,因此能在不改變品質的情況下加快文字生成速度,在公開測試中通常能快兩到三倍。
「草案」模型在推測解碼中的主要作用是什麼?
小草稿模型可以廉價地猜測即將到來的令牌,然後大型目標模型在單一平行傳遞中檢查這些令牌。
為什麼在合適的工作負載下,驗證一組草稿 Token 區塊可以加速目標模型的生成?
平行驗證可以降低每個被接受 Token 的成本,但其成本與效益取決於模型、硬體、區塊長度和接受率。
在目標模型拒絕第一個令牌之後,起草的令牌會發生什麼?
在標準程序中,第一次拒絕即終止對草稿前綴的接受。系統會進行修正採樣並捨棄後續的草稿後綴;拒絕是由接受測試所主導,而不僅僅是 argmax 的分歧。
在標準的精確推測式採樣拒絕程序及其假設下,它保留了什麼正確性屬性?
精確性關注的是採樣規律,而非事實正確性或保證逐位元(bit-for-bit)的序列一致性。
哪種機制區分了 EAGLE 的草稿方法與 Medusa 的額外解碼頭?
EAGLE 使用特徵層級的自迴歸,而 Medusa 則引入了多個額外的 Token 預測頭。