技術指南

第一個令牌的時間和延遲指標

Time to first token (TTFT) measures delay until the first generated token, while inter-token latency (ITL) measures gaps between generated tokens and throughput measures output volume over time.

聆聽此指南AI 語音敘述 · ~3 min

本頁閱讀時間3分鐘
  1. 概述
  2. 深入探討
  3. 現實世界的實施
  4. The Future of Time to First Token and Latency Metrics
  5. 付諸實踐
  6. 不斷探索
  7. 常見問題

概述

Tail percentiles such as p95 and p99 show slow-request behavior, but metric definitions and user experience targets depend on the serving stack and workload.

深入探討

A streamed language-model response has several timing components. Time to first token (TTFT) measures how long a request waits before the first output token arrives. Inter-token latency (ITL), sometimes reported as time per output token, describes the gaps during generation. End-to-end latency includes the full request through the final output; throughput describes how many requests or tokens a system handles over time. These metrics help distinguish different problems. Long TTFT can reflect queueing, network delay, long prompts, or prompt processing. High ITL can reflect decode speed, model size, or contention. Aggregate throughput may rise under batching even while an individual request waits longer. A fast average can hide slow tail requests, so teams often inspect p50, p95, and p99 along with error rates and token counts. Definitions need care. Some tools report time-per-output-token (TPOT), computed from end-to-end latency and TTFT; others report token intervals directly. The vLLM benchmark documentation defines its metrics and advises users to evaluate in serving conditions. Compare measurements only when workload, streaming mode, prompt lengths, output lengths, concurrency, hardware, and network conditions are comparable. No one metric establishes that a product feels fast. A typing assistant may need low first-token delay; a long research report may tolerate slower initial output if its final answer is good. Set user-centered service objectives, sample real requests, and monitor distributions over time rather than optimizing one average in isolation.

現實世界的實施

A team tracks p95 TTFT to see whether a chat interface starts responding promptly under peak load.

An engineer compares ITL across model configurations using identical prompt and output lengths.

A service reports aggregate tokens per second separately from each user’s response time.

A benchmark records p99 latency and request failures instead of reporting only the average.

The Future of Time to First Token and Latency Metrics

Serving stacks may provide richer tracing that separates queue, prefill, decode, and network delays. Better telemetry can help teams meet different interaction targets without overprovisioning every workload. Comparisons will remain meaningful only when workloads and metric definitions are disclosed. Future dashboards should connect tail latency to user tasks and quality outcomes, and show uncertainty and failure rates alongside speed. Shared benchmark formats may improve repeatability across providers, while real-user monitoring will remain essential during traffic peaks and deployments at launch.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the Time to First Token and Latency Metrics quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

What is Time to First Token and Latency Metrics?

Time to first token (TTFT) measures delay until the first generated token, while inter-token latency (ITL) measures gaps between generated tokens and throughput measures output volume over time. Tail percentiles such as p95 and p99 show slow-request behavior, but metric definitions and user experience targets depend on the serving stack and workload.

How does aggregate throughput differ from per-request latency?

A service can raise total tokens per second while individual requests wait longer.

Why inspect p95 or p99 latency in addition to the average?

Tail metrics describe high-latency portions of the request distribution.

Why must benchmark comparisons use similar prompts, output lengths, and load?

Workload characteristics affect measured serving performance.