Útmutatóhoz kapcsolódó kvíz · Közepes Szint

Tail Latency in Model Serving Quiz

Interpret p95 and p99 latency and evaluate batching, queueing, timeouts and hedging without creating additional load or duplicate side effects.

Kapcsolódó útmutató utakTail Latency In Model Serving
kérdés 1 -ból 8

What does p99 latency mean for a measured request sample?

Tekintse meg az összes kérdést8 kérdéseket

A válaszok és magyarázatok megjelennek a kvíz kitöltésekor.

  1. What does p99 latency mean for a measured request sample?
    • The single slowest request
    • The average latency of all requests
    • 99 percent of requests completed at or below that value
    • 99 requests per second
  2. Why can mean latency hide a service problem?
    • A low mean proves every request met its latency objective
    • A slow tail can affect a smaller share of requests without shifting the average much
    • Averages always equal p99
    • The mean uses the p99 value as its input
  3. What tradeoff can dynamic batching introduce?
    • Lower throughput with guaranteed lower latency
    • More model accuracy without extra work
    • Reduced memory with no scheduling effects
    • Higher throughput but additional queueing delay
  4. When is hedged inference safest to consider?
    • For idempotent requests with controlled duplicate work
    • For non-repeatable actions such as charging a customer
    • When the service is already overloaded with no limits
    • For requests that modify shared state
  5. How can hedged requests make a latency incident worse?
    • They add concurrent attempts and more load when the fleet is already congested
    • They convert the tail distribution into a fixed upper bound
    • They eliminate slowdowns when replicas share a saturated backend
    • They serialize each duplicate so only one request consumes resources
  6. Why measure end-to-end latency rather than only model kernel time?
    • Kernel time always equals request time
    • Queueing, preprocessing and network time also affect user response
    • Preprocessing cannot add delay
    • Network time is included in model FLOPs
  7. What additional signal should be examined alongside latency percentiles?
    • Only the model parameter count
    • Only average training loss
    • The checkpoint file name
    • Timeout and failure rates
  8. Why should latency percentiles include a stated measurement window and population?
    • The estimate depends on which requests and time period were sampled
    • Percentiles are fixed constants for a model
    • The window determines the model weights
    • It removes all sampling variability