AI Glossary Term
What does “Inference-Time Compute” mean?
The amount of processing power consumed while producing each response.
How Inference-Time Compute works
Example
A reasoning-focused model given a tricky math problem may generate a long internal chain of steps before answering. That extra processing costs more time and money than a one-line reply, but can improve accuracy on difficult questions.
Why it matters
Spending more compute at answer time is another way to improve results, besides training bigger models. It also changes pricing and response times.
Common questions
How is it different from training compute?
Training compute is spent once to build the model. Inference-time compute is spent on every request.
Does more thinking always help?
Not always. It helps most on multi-step problems, while simple questions gain little.
How is it billed?
Often by the tokens generated, including reasoning tokens, so longer thinking can raise the bill.