Bài kiểm tra có liên kết hướng dẫn · Trung bình Cấp độ

Độ trễ đuôi trong bài kiểm tra phục vụ mô hình

Giải thích độ trễ p95 và p99, đồng thời đánh giá việc phân đợt, xếp hàng, hết thời gian chờ và phòng ngừa rủi ro mà không tạo thêm tải hoặc tác dụng phụ trùng lặp.

Đường dẫn hướng dẫn liên quanĐộ trễ đuôi trong việc phục vụ mô hình
Câu hỏi 1 của 8

What does p99 latency mean for a measured request sample?

Xem trước mọi câu hỏi8 câu hỏi

Câu trả lời và giải thích xuất hiện khi bạn làm bài kiểm tra.

  1. What does p99 latency mean for a measured request sample?
    • The single slowest request
    • The average latency of all requests
    • 99 percent of requests completed at or below that value
    • 99 requests per second
  2. Why can mean latency hide a service problem?
    • A low mean proves every request met its latency objective
    • A slow tail can affect a smaller share of requests without shifting the average much
    • Averages always equal p99
    • The mean uses the p99 value as its input
  3. What tradeoff can dynamic batching introduce?
    • Lower throughput with guaranteed lower latency
    • More model accuracy without extra work
    • Reduced memory with no scheduling effects
    • Higher throughput but additional queueing delay
  4. When is hedged inference safest to consider?
    • For idempotent requests with controlled duplicate work
    • For non-repeatable actions such as charging a customer
    • When the service is already overloaded with no limits
    • For requests that modify shared state
  5. How can hedged requests make a latency incident worse?
    • They add concurrent attempts and more load when the fleet is already congested
    • They convert the tail distribution into a fixed upper bound
    • They eliminate slowdowns when replicas share a saturated backend
    • They serialize each duplicate so only one request consumes resources
  6. Why measure end-to-end latency rather than only model kernel time?
    • Kernel time always equals request time
    • Queueing, preprocessing and network time also affect user response
    • Preprocessing cannot add delay
    • Network time is included in model FLOPs
  7. What additional signal should be examined alongside latency percentiles?
    • Only the model parameter count
    • Only average training loss
    • The checkpoint file name
    • Timeout and failure rates
  8. Why should latency percentiles include a stated measurement window and population?
    • The estimate depends on which requests and time period were sampled
    • Percentiles are fixed constants for a model
    • The window determines the model weights
    • It removes all sampling variability