AI Glossary Term

What does “Latency” mean?

Definition

The time between sending a request and receiving the model's output.

Browse the full AI GlossaryExplore all free AI guides

How Latency works

Diagram for Latency, steps in order: You send a request; then Network travel; then Queue and model processing; then Reply returns; then Total wait = latency.You send a requestNetwork travelQueue and model processingReply returnsTotal wait = latency

Example

A voice assistant needs to start speaking within about a second to feel natural. If the model takes five seconds to respond, the pause feels broken even if the answer is excellent.

Why it matters

Responsiveness affects whether people keep using an AI product. Larger models and longer outputs generally increase latency.

Common questions

What is time to first token?

The delay before the first piece of the answer appears, which matters for streaming chat interfaces.

How can latency be reduced?

Smaller or compressed models, caching, faster hardware, shorter outputs, and techniques like speculative decoding.

Is latency the same as throughput?

No. Latency is time per request; throughput is how many requests are handled per unit time.

Related terms

Attention MechanismA model component that dynamically focuses on relevant parts of an input when producing an output.Autonomous SystemA system that can make decisions and act with limited or no direct human control in real time.CLIPA multimodal model architecture that learns shared representations between text and images.Data DriftA shift in real-world input data over time that can degrade model performance.Inference-Time ComputeThe amount of processing power consumed while producing each response.Knowledge CutoffThe latest point in time reflected in a model's training data.

Learn more in our free guides

Tail Latency in Model ServingReducing LLM App LatencyTime to First Token and Latency Metrics

Test yourself

Tail Latency in Model Serving QuizReducing LLM App Latency QuizTime to First Token and Latency Metrics Quiz

See also

Label SmoothingLarge Language Model (LLM)Knowledge GraphLearning RateKnowledge DistillationLoRA (Low-Rank Adaptation)Loss FunctionJailbreak