Load Testing Model Endpoints

Load testing sends controlled traffic to an endpoint to determine how it behaves at expected and elevated demand. For model serving, relevant measures include requests per second, end-to-end latency distributions, error and timeout rates, queue depth, CPU and GPU utilization, memory use, and scaling events. A single average or successful-response count does not reveal the full capacity or user experience. The workload should resemble intended use. Vary input dimensions, sequence lengths, batch sizes, and request frequency if those change compute or memory. Include authentication and preprocessing when they are part of the actual path. Decide whether the test represents a closed workload, where each virtual user waits for a response before sending another request, or an open arrival rate, where requests arrive independently of response time. A closed workload can reduce offered load as the service slows, hiding overload behavior. Use distinct test types. A baseline measures normal traffic. A load test checks expected demand. A stress test pushes beyond expected capacity to observe failure modes, while a soak test runs longer to reveal leaks or degradation. Increase load gradually, set stop conditions, and monitor dependencies so the test does not spill into unrelated services. Cold and warm paths should be evaluated separately. If the service scales to zero, include startup and model-loading behavior; for steady serving, prewarm as production would. Ensure repeated requests do not accidentally reuse cached responses if the real service will not. Measure from the client as well as inside the server to capture network and queue time. Tools such as Locust let teams define request scenarios in Python and observe response times, throughput, and errors. The tool does not make a test realistic by itself: request models, generators, network placement, and metrics matter. Run in a controlled environment, label synthetic traffic, avoid sensitive data, and document hardware and software versions. Compare results against an explicit service objective rather than declaring capacity from one test run.