AI Glossary Term

What is Speculative Decoding?

Definition

An inference acceleration method where a small draft model proposes tokens that a larger model verifies in parallel.

Related terms

Usage-Based Billing
Pricing where costs scale with API calls, tokens, inference time, or consumed compute.
Sliding Window Attention
An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.
Inference
The runtime phase where a trained model generates predictions or outputs.
RAG (Retrieval-Augmented Generation)
A method that retrieves external knowledge and feeds it into generation at inference time.
Top-k Sampling
A decoding strategy that samples only from the k most likely next tokens.
Distilled Model
A smaller model trained to imitate a larger model's behavior while using less compute at inference.

Learn more in our free guides

Contrastive DecodingMinimum Bayes Risk DecodingSpeculative Sampling VerificationSelf-Consistency Decoding
Browse the full AI GlossaryExplore all free AI guides