Time to First Token and Latency Metrics

A streamed language-model response has several timing components. Time to first token (TTFT) measures how long a request waits before the first output token arrives. Inter-token latency (ITL), sometimes reported as time per output token, describes the gaps during generation. End-to-end latency includes the full request through the final output; throughput describes how many requests or tokens a system handles over time.

These metrics help distinguish different problems. Long TTFT can reflect queueing, network delay, long prompts, or prompt processing. High ITL can reflect decode speed, model size, or contention. Aggregate throughput may rise under batching even while an individual request waits longer. A fast average can hide slow tail requests, so teams often inspect p50, p95, and p99 along with error rates and token counts.

Definitions need care. Some tools report time-per-output-token (TPOT), computed from end-to-end latency and TTFT; others report token intervals directly. The vLLM benchmark documentation defines its metrics and advises users to evaluate in serving conditions. Compare measurements only when workload, streaming mode, prompt lengths, output lengths, concurrency, hardware, and network conditions are comparable.

No one metric establishes that a product feels fast. A typing assistant may need low first-token delay; a long research report may tolerate slower initial output if its final answer is good. Set user-centered service objectives, sample real requests, and monitor distributions over time rather than optimizing one average in isolation.