Autoscaling Model Inference on Kubernetes

Autoscaling changes the number of serving replicas in response to demand or resource signals. Kubernetes Horizontal Pod Autoscaler (HPA) can scale workloads using resource metrics such as CPU and memory, or custom and external metrics when adapters provide them. Queue depth or in-flight requests may better represent ML serving pressure than CPU alone, especially when GPU kernels saturate accelerators while host CPU remains underused. GPU inference scaling involves more than replica count. Pods need GPU resource requests, compatible nodes, device plugins and sufficient accelerator capacity. A scheduler cannot create hardware that is unavailable or over quota. Node autoscaling may add GPUs, but provisioning and driver setup take time. Large model images and weight downloads add startup delay, and model loading can consume substantial memory. KEDA can scale Kubernetes workloads using event sources and custom triggers, such as queue length, and may support scaling to zero depending on the scaler and setup. Scaling to zero saves idle cost but creates a cold start when the next request arrives. A queue, warm pool or minimum replica count can balance cost and latency. HPA behavior settings, stabilization windows and scale-up/down policies prevent oscillation and overly abrupt changes. Choose metrics tied to user impact and workload. Request queue age and p95 latency can reveal service pressure; GPU utilization and memory help explain capacity; error rate may signal overload. Metrics need reliable exporters and careful aggregation. Set max replicas to respect cost and quota, and test bursts, model-load failures and scale-down behavior. Autoscaling can react to measured demand, but it cannot guarantee timely capacity under hardware scarcity or fix a model that is intrinsically too slow. Include load tests, readiness checks, graceful draining and backpressure in the serving design.