Continuous Batching
Continuous batching is a serving technique that adds and removes requests from a running batch token-by-token, instead of waiting for a whole fixed batch to finish.
Overview
It keeps the GPU constantly busy and sharply increases how many users an AI model can serve at once.
Deep Dive
GPUs are fastest when they process many requests together in a batch. The naive approach, static batching, groups a fixed set of requests, runs them all to completion, then starts the next batch. The problem: language model outputs vary wildly in length, so short requests finish early and their slots sit idle while the batch waits for the longest one, wasting GPU cycles and delaying new arrivals. Continuous batching (also called in-flight or iteration-level batching, popularized by the Orca paper and used in vLLM, TensorRT-LLM, and TGI) operates at the granularity of a single decoding step. After each token is generated, finished sequences exit the batch and freshly arrived requests are slotted in immediately. This keeps the batch full and the GPU saturated, often boosting throughput several times over with lower latency for waiting users.
Technical Insight
The key shift is from batching whole requests to batching individual iterations. At every decode step the scheduler builds the active set: it runs one forward pass over all in-flight sequences, emits one token each, evicts any that hit an end-of-sequence token or length limit, and admits queued requests to fill the freed slots. Pairing this with PagedAttention's flexible KV memory makes inserting and removing sequences mid-flight cheap, since each sequence's cache lives in independent blocks.
Strategic Impact
Cost and budget
Architecture decisions drive performance and operating cost for years.
Clearer decisions
Technical education helps teams choose the right stack, not just the newest one.
Quality control
Better engineering choices reduce reliability incidents in production.
The Future of Continuous Batching
Continuous batching is now standard in production LLM serving. Future work refines the scheduler: separating the compute-heavy prefill phase from the lighter decode phase (disaggregation), chunked prefill to avoid stalling decoding, priority and fairness policies for mixed workloads, and tighter coupling with speculative decoding so multiple draft tokens are validated per step. The goal is squeezing maximum tokens-per-second per GPU while keeping individual response latency low and predictable.
Real-World Implementation
A chat API admitting newly arrived user messages into the running batch immediately instead of queuing them for the next batch
Evicting a short completed answer mid-batch and backfilling its slot so the GPU never idles waiting on a long generation
Combining continuous batching with vLLM's PagedAttention to insert and remove sequences cheaply at each decode step
A code-completion service sustaining high tokens-per-second under bursty, variable-length traffic by keeping the batch full
Risks & Guardrails
Optimizing one benchmark can hide broader system weaknesses.
Infrastructure and maintenance costs are often underestimated.
Security and observability gaps can grow as systems become more complex.
Implementation Roadmap
Define latency, quality, and cost targets before implementation.
Benchmark under realistic load and data conditions.
Instrument monitoring for errors, drift, and user impact.
Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Continuous Batching quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
Kubernetes for ML Workloads
Frequently asked questions
What is Continuous Batching?
Continuous batching is a serving technique that adds and removes requests from a running batch token-by-token, instead of waiting for a whole fixed batch to finish. It keeps the GPU constantly busy and sharply increases how many users an AI model can serve at once.
What is the main weakness of static (fixed) batching for LLM serving?
Because output lengths vary, finished sequences sit idle until the slowest one completes, wasting GPU cycles and delaying new requests.
At what granularity does continuous batching add and remove requests?
Continuous (iteration-level) batching updates the active set after every single decode step, so it operates token-by-token rather than per whole request.
When a sequence finishes mid-batch under continuous batching, what happens to its slot?
Finished sequences are evicted and waiting requests are slotted in immediately, keeping the batch full and the GPU busy.
Which technique pairs naturally with continuous batching to make inserting/removing sequences cheap?
PagedAttention stores each sequence's KV cache in independent blocks, so adding or removing a sequence mid-flight does not disturb others' memory.
What research paper is commonly credited with popularizing iteration-level (continuous) batching?
The Orca paper introduced iteration-level scheduling, and the idea was adopted by serving systems like vLLM, TGI, and TensorRT-LLM.