Guide-linked quiz · Medium Level
Tail Latency in Model Serving Quiz
Interpret p95 and p99 latency and evaluate batching, queueing, timeouts and hedging without creating additional load or duplicate side effects.
Question 1 of 8
What does p99 latency mean for a measured request sample?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- What does p99 latency mean for a measured request sample?
- The single slowest request
- The average latency of all requests
- 99 percent of requests completed at or below that value
- 99 requests per second
- Why can mean latency hide a service problem?
- A low mean proves every request met its latency objective
- A slow tail can affect a smaller share of requests without shifting the average much
- Averages always equal p99
- The mean uses the p99 value as its input
- What tradeoff can dynamic batching introduce?
- Lower throughput with guaranteed lower latency
- More model accuracy without extra work
- Reduced memory with no scheduling effects
- Higher throughput but additional queueing delay
- When is hedged inference safest to consider?
- For idempotent requests with controlled duplicate work
- For non-repeatable actions such as charging a customer
- When the service is already overloaded with no limits
- For requests that modify shared state
- How can hedged requests make a latency incident worse?
- They add concurrent attempts and more load when the fleet is already congested
- They convert the tail distribution into a fixed upper bound
- They eliminate slowdowns when replicas share a saturated backend
- They serialize each duplicate so only one request consumes resources
- Why measure end-to-end latency rather than only model kernel time?
- Kernel time always equals request time
- Queueing, preprocessing and network time also affect user response
- Preprocessing cannot add delay
- Network time is included in model FLOPs
- What additional signal should be examined alongside latency percentiles?
- Only the model parameter count
- Only average training loss
- The checkpoint file name
- Timeout and failure rates
- Why should latency percentiles include a stated measurement window and population?
- The estimate depends on which requests and time period were sampled
- Percentiles are fixed constants for a model
- The window determines the model weights
- It removes all sampling variability
Keep testing yourself
More quizzes picked for your level.