AI Glossary Term

What does “Inference” mean?

Definition

The runtime phase where a trained model generates predictions or outputs.

Browse the full AI Glossary Explore all free AI guides

How Inference works

Example

When you send a message to a chatbot, the provider runs inference: the already-trained model reads your message and generates a reply. No learning happens during that step.

Why it matters

Inference is the part users pay for and wait on. Its speed and cost affect product pricing, latency, and energy use.

Common questions

How is inference different from training?

Training adjusts the weights using data. Inference uses fixed weights to produce outputs.

Why can inference be expensive?

Large models need substantial memory and compute for each request, and popular services handle huge numbers of requests.

Can inference run on a phone?

Smaller or compressed models can, which is the aim of on-device AI.

Inference EndpointA deployed API interface that receives model requests and returns predictions in production.Human-in-the-LoopA workflow where humans review, guide, or override AI outputs.Constitutional AIA training and behavior-shaping approach where model outputs are guided by a fixed set of written principles.Distilled ModelA smaller model trained to imitate a larger model's behavior while using less compute at inference.Usage-Based BillingPricing where costs scale with API calls, tokens, inference time, or consumed compute.Speculative DecodingAn inference acceleration method where a small draft model proposes tokens that a larger model verifies in parallel.

Learn more in our free guides

AI Inference AI Inference Optimization LLM Inference Routing and Load Balancing Seldon Core and Inference Graphs

Test yourself

AI Inference Quiz AI Inference Optimization Quiz LLM Inference Routing and Load Balancing Quiz