Up nextNext guide
Continuous Training and Automated Retraining
Technical
Technical GUIDE
Continuous Batching
Continuous batching updates the active request set between generation iterations instead of holding one fixed batch until every request finishes.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
It can improve serving utilization, while capacity limits and scheduling policy still govern latency.
Deep Dive
GPUs are fastest when they process many requests together in a batch. The naive approach, static batching, groups a fixed set of requests, runs them all to completion, then starts the next batch. The problem: language model outputs vary wildly in length, so short requests finish early and their slots sit idle while the batch waits for the longest one, wasting GPU cycles and delaying new arrivals. Continuous batching (also called in-flight or iteration-level batching, popularized by the Orca paper and used in vLLM, TensorRT-LLM, and TGI) operates at the granularity of a single decoding step. After each token is generated, finished sequences exit the batch and queued requests can be admitted when the scheduler has capacity. Iteration-level admission can improve utilization and reduce avoidable waiting compared with fixed batches; the measured effect depends on request lengths, capacity, and scheduling.
Real-World Implementation
A chat API admitting newly arrived user messages into the running batch immediately instead of queuing them for the next batch
Evicting a short completed answer mid-batch and backfilling its slot so the GPU never idles waiting on a long generation
Combining continuous batching with vLLM's PagedAttention to insert and remove sequences cheaply at each decode step
A code-completion service sustaining high tokens-per-second under bursty, variable-length traffic by keeping the batch full
The Future of Continuous Batching
Serving teams balance throughput with time to first token and time between generated tokens. Long prefill work can delay decoding unless the scheduler splits or otherwise manages it. Capacity budgets, request priorities, and fairness policies determine which work is admitted. Continuous batching can be combined with paged caches, chunked prefill, and speculative decoding, but each combination should be measured on the actual traffic and hardware.
Putting it into practice
Why it matters
- Cost and budget. Architecture decisions drive performance and operating cost for years.
- Clearer decisions. Technical education helps teams choose the right stack, not just the newest one.
- Quality control. Better engineering choices reduce reliability incidents in production.
Watch out for
- Optimizing one benchmark can hide broader system weaknesses.
- Infrastructure and maintenance costs are often underestimated.
- Security and observability gaps can grow as systems become more complex.
Next steps
- Define latency, quality, and cost targets before implementation.
- Benchmark under realistic load and data conditions.
- Instrument monitoring for errors, drift, and user impact.
- Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Continuous Batching quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Continuous Batching?
Continuous batching updates the active request set between generation iterations instead of holding one fixed batch until every request finishes. It can improve serving utilization, while capacity limits and scheduling policy still govern latency.
What is the main weakness of static (fixed) batching for LLM serving?
Because output lengths vary, finished sequences sit idle until the slowest one completes, wasting GPU cycles and delaying new requests.
At what granularity does continuous batching add and remove requests?
Continuous (iteration-level) batching updates the active set after every single decode step, so it operates token-by-token rather than per whole request.
A continuous-batching request finishes and frees capacity. What can the scheduler do next?
Iteration-level scheduling can refill available capacity subject to memory, token, and other limits.
Which technique pairs naturally with continuous batching to make inserting/removing sequences cheap?
PagedAttention stores each sequence's KV cache in independent blocks, so adding or removing a sequence mid-flight does not disturb others' memory.
What research paper is commonly credited with popularizing iteration-level (continuous) batching?
The Orca paper introduced iteration-level scheduling, and the idea was adopted by serving systems like vLLM, TGI, and TensorRT-LLM.
Keep learning
Related guides
More guides picked for this topic