Tail Latency in Model Serving

Latency is not one number. The mean describes average response time, while percentiles show thresholds below which a share of requests complete. At p99, 99 percent of observed requests are at or below that latency and the slowest one percent are above it. Tail latency matters when users notice timeouts, when services have strict response objectives, or when an application waits for several model calls to finish.

Model-serving tails can be caused by queueing near capacity, uneven request sizes, cold starts, data loading, accelerator contention, cache misses, garbage collection, network delays, or a slow downstream service. A model with fast kernel time may still have slow request latency if preprocessing or queue wait dominates. Record end-to-end timings and stage spans, and analyze by input size, model, device, and traffic condition.

Batching can improve hardware utilization by processing several requests together, but waiting to fill a batch adds latency. Use a maximum batching delay and observe both throughput and percentiles. Autoscaling also trades capacity cost against queueing during scale-up. Load shedding or bounded queues can protect the service from overload, while timeouts prevent callers from waiting indefinitely.

Hedged requests send a duplicate to another worker after a delay when an original request is unusually slow; the first valid response wins and remaining work is canceled when possible. This can reduce tail delays when the slow path is transient and replicas are independent. It can also increase traffic and worsen overload. Use it only for idempotent requests or operations with safe duplication, and control the rate.

Measure percentiles over representative traffic and sufficient windows, and define whether the statistic is per-request, per-batch, or per-user journey. Averages and percentiles can hide errors or groups with different needs. Track timeout rate and failure rate alongside latency, and verify that optimizations preserve model outputs.