Викторина, связанная с руководством · Средний Уровень
Задержка хвоста в викторине по обслуживанию модели
Интерпретируйте задержку p95 и p99 и оценивайте пакетную обработку, организацию очередей, тайм-ауты и хеджирование без создания дополнительной нагрузки или дублирования побочных эффектов.
Вопрос 1 из 8
What does p99 latency mean for a measured request sample?
Предварительный просмотр каждого вопроса8 вопросы
Ответы и пояснения появляются по мере прохождения теста.
- What does p99 latency mean for a measured request sample?
- The single slowest request
- The average latency of all requests
- 99 percent of requests completed at or below that value
- 99 requests per second
- Why can mean latency hide a service problem?
- A low mean proves every request met its latency objective
- A slow tail can affect a smaller share of requests without shifting the average much
- Averages always equal p99
- The mean uses the p99 value as its input
- What tradeoff can dynamic batching introduce?
- Lower throughput with guaranteed lower latency
- More model accuracy without extra work
- Reduced memory with no scheduling effects
- Higher throughput but additional queueing delay
- When is hedged inference safest to consider?
- For idempotent requests with controlled duplicate work
- For non-repeatable actions such as charging a customer
- When the service is already overloaded with no limits
- For requests that modify shared state
- How can hedged requests make a latency incident worse?
- They add concurrent attempts and more load when the fleet is already congested
- They convert the tail distribution into a fixed upper bound
- They eliminate slowdowns when replicas share a saturated backend
- They serialize each duplicate so only one request consumes resources
- Why measure end-to-end latency rather than only model kernel time?
- Kernel time always equals request time
- Queueing, preprocessing and network time also affect user response
- Preprocessing cannot add delay
- Network time is included in model FLOPs
- What additional signal should be examined alongside latency percentiles?
- Учитываются только параметры модели
- Only average training loss
- The checkpoint file name
- Timeout and failure rates
- Why should latency percentiles include a stated measurement window and population?
- The estimate depends on which requests and time period were sampled
- Percentiles are fixed constants for a model
- The window determines the model weights
- It removes all sampling variability