What does Speculative Decoding mean? Definition | AI Understanding

Definition

An inference acceleration method where a small draft model proposes tokens that a larger model verifies in parallel.

Related terms

Usage-Based Billing

Pricing where costs scale with API calls, tokens, inference time, or consumed compute.

Sliding Window Attention

An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.

The runtime phase where a trained model generates predictions or outputs.

RAG (Retrieval-Augmented Generation)

A method that retrieves external knowledge and feeds it into generation at inference time.

A decoding strategy that samples only from the k most likely next tokens.

Distilled Model

A smaller model trained to imitate a larger model's behavior while using less compute at inference.

Learn more in our free guides

Contrastive Decoding Minimum Bayes Risk Decoding Speculative Sampling Verification Self-Consistency Decoding

See also

Zero Data Retention KV Cache MCP (Model Context Protocol)Trust Calibration Agentic Loop Test-Time Compute ReAct Structured Output

Browse the full AI Glossary Explore all free AI guides