引導式測驗 · 中等 等級

模型服务测验中的尾部延迟

解釋 p95 和 p99 延遲並評估批次、排隊、逾時和對沖,而不會產生額外的負載或重複的副作用。

問題 1 的 8

p99 延遲對測量請求樣本意味著什麼?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. p99 延遲對測量請求樣本意味著什麼?
    • The single slowest request
    • The average latency of all requests
    • 99 percent of requests completed at or below that value
    • 99 requests per second
  2. 為什麼平均延遲可能掩蓋服務問題?
    • A low mean proves every request met its latency objective
    • A slow tail can affect a smaller share of requests without shifting the average much
    • Averages always equal p99
    • The mean uses the p99 value as its input
  3. 動態批次化可能帶來哪些權衡?
    • Lower throughput with guaranteed lower latency
    • More model accuracy without extra work
    • Reduced memory with no scheduling effects
    • Higher throughput but additional queueing delay
  4. 什麼時候考慮使用對沖推論最安全?
    • For idempotent requests with controlled duplicate work
    • For non-repeatable actions such as charging a customer
    • When the service is already overloaded with no limits
    • For requests that modify shared state
  5. 對沖請求如何使延遲事件變得更糟?
    • They add concurrent attempts and more load when the fleet is already congested
    • They convert the tail distribution into a fixed upper bound
    • They eliminate slowdowns when replicas share a saturated backend
    • They serialize each duplicate so only one request consumes resources
  6. Why measure end-to-end latency rather than only model kernel time?
    • Kernel time always equals request time
    • Queueing, preprocessing and network time also affect user response
    • Preprocessing cannot add delay
    • Network time is included in model FLOPs
  7. What additional signal should be examined alongside latency percentiles?
    • Only the model parameter count
    • Only average training loss
    • The checkpoint file name
    • Timeout and failure rates
  8. Why should latency percentiles include a stated measurement window and population?
    • The estimate depends on which requests and time period were sampled
    • Percentiles are fixed constants for a model
    • The window determines the model weights
    • It removes all sampling variability