What is Inference-Time Compute?

Additional inference computation used during response generation to improve quality or reasoning.

Pricing where costs scale with API calls, tokens, inference time, or consumed compute.

A smaller model trained to imitate a larger model's behavior while using less compute at inference.

The processing resources required to train and run models, often measured in FLOPS or GPU hours.

A method that retrieves external knowledge and feeds it into generation at inference time.

The extra cost in time, compute, or product velocity required to make systems safer and more controllable.