What does Inference-Time Compute mean? Definition | AI Understanding

Definition

The amount of processing power consumed while producing each response.

Related terms

Test-Time Compute

Additional inference computation used during response generation to improve quality or reasoning.

Usage-Based Billing

Pricing where costs scale with API calls, tokens, inference time, or consumed compute.

Distilled Model

A smaller model trained to imitate a larger model's behavior while using less compute at inference.

The processing resources required to train and run models, often measured in FLOPS or GPU hours.

RAG (Retrieval-Augmented Generation)

A method that retrieves external knowledge and feeds it into generation at inference time.

The extra cost in time, compute, or product velocity required to make systems safer and more controllable.

Learn more in our free guides

AI Inference Computer Vision Sentiment Analysis Real-Time Voice Agents

See also

Inference Instruction Tuning In-Context Learning Intent Classification Hyperparameter Jailbreak Human-in-the-Loop Knowledge Cutoff

Browse the full AI Glossary Explore all free AI guides