引導式測驗 · 中等 等級
模型服务测验中的尾部延迟
解釋 p95 和 p99 延遲並評估批次、排隊、逾時和對沖,而不會產生額外的負載或重複的副作用。
問題 1 的 8
p99 延遲對測量請求樣本意味著什麼?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- p99 延遲對測量請求樣本意味著什麼?
- The single slowest request
- The average latency of all requests
- 99 percent of requests completed at or below that value
- 99 requests per second
- 為什麼平均延遲可能掩蓋服務問題?
- A low mean proves every request met its latency objective
- A slow tail can affect a smaller share of requests without shifting the average much
- Averages always equal p99
- The mean uses the p99 value as its input
- 動態批次化可能帶來哪些權衡?
- Lower throughput with guaranteed lower latency
- More model accuracy without extra work
- Reduced memory with no scheduling effects
- Higher throughput but additional queueing delay
- 什麼時候考慮使用對沖推論最安全?
- For idempotent requests with controlled duplicate work
- For non-repeatable actions such as charging a customer
- When the service is already overloaded with no limits
- For requests that modify shared state
- 對沖請求如何使延遲事件變得更糟?
- They add concurrent attempts and more load when the fleet is already congested
- They convert the tail distribution into a fixed upper bound
- They eliminate slowdowns when replicas share a saturated backend
- They serialize each duplicate so only one request consumes resources
- Why measure end-to-end latency rather than only model kernel time?
- Kernel time always equals request time
- Queueing, preprocessing and network time also affect user response
- Preprocessing cannot add delay
- Network time is included in model FLOPs
- What additional signal should be examined alongside latency percentiles?
- Only the model parameter count
- Only average training loss
- The checkpoint file name
- Timeout and failure rates
- Why should latency percentiles include a stated measurement window and population?
- The estimate depends on which requests and time period were sampled
- Percentiles are fixed constants for a model
- The window determines the model weights
- It removes all sampling variability