Guide-linked quiz · Medium Level

Tail Latency in Model Serving Quiz

Interpret p95 and p99 latency and evaluate batching, queueing, timeouts and hedging without creating additional load or duplicate side effects.

Question 1 of 8

What does p99 latency mean for a measured request sample?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. What does p99 latency mean for a measured request sample?
    • The single slowest request
    • The average latency of all requests
    • 99 percent of requests completed at or below that value
    • 99 requests per second
  2. Why can mean latency hide a service problem?
    • A low mean proves every request met its latency objective
    • A slow tail can affect a smaller share of requests without shifting the average much
    • Averages always equal p99
    • The mean uses the p99 value as its input
  3. What tradeoff can dynamic batching introduce?
    • Lower throughput with guaranteed lower latency
    • More model accuracy without extra work
    • Reduced memory with no scheduling effects
    • Higher throughput but additional queueing delay
  4. When is hedged inference safest to consider?
    • For idempotent requests with controlled duplicate work
    • For non-repeatable actions such as charging a customer
    • When the service is already overloaded with no limits
    • For requests that modify shared state
  5. How can hedged requests make a latency incident worse?
    • They add concurrent attempts and more load when the fleet is already congested
    • They convert the tail distribution into a fixed upper bound
    • They eliminate slowdowns when replicas share a saturated backend
    • They serialize each duplicate so only one request consumes resources
  6. Why measure end-to-end latency rather than only model kernel time?
    • Kernel time always equals request time
    • Queueing, preprocessing and network time also affect user response
    • Preprocessing cannot add delay
    • Network time is included in model FLOPs
  7. What additional signal should be examined alongside latency percentiles?
    • Only the model parameter count
    • Only average training loss
    • The checkpoint file name
    • Timeout and failure rates
  8. Why should latency percentiles include a stated measurement window and population?
    • The estimate depends on which requests and time period were sampled
    • Percentiles are fixed constants for a model
    • The window determines the model weights
    • It removes all sampling variability

Keep testing yourself

More quizzes picked for your level.

View all quizzes