概述
It can improve serving utilization, while capacity limits and scheduling policy still govern latency.
深入探讨
GPUs are fastest when they process many requests together in a batch. The naive approach, static batching, groups a fixed set of requests, runs them all to completion, then starts the next batch. The problem: language model outputs vary wildly in length, so short requests finish early and their slots sit idle while the batch waits for the longest one, wasting GPU cycles and delaying new arrivals. Continuous batching (also called in-flight or iteration-level batching, popularized by the Orca paper and used in vLLM, TensorRT-LLM, and TGI) operates at the granularity of a single decoding step. After each token is generated, finished sequences exit the batch and queued requests can be admitted when the scheduler has capacity. Iteration-level admission can improve utilization and reduce avoidable waiting compared with fixed batches; the measured effect depends on request lengths, capacity, and scheduling.
现实世界的实施
聊天 API 立即将新到达的用户消息接纳到正在运行的批次中，而不是将它们排队等待下一个批次
在批次中驱逐较短的已完成答案并回填其插槽，以便 GPU 永远不会闲置等待较长的一代
将连续批处理与 vLLM 的 PagedAttention 相结合，在每个解码步骤中以低成本插入和删除序列
代码完成服务通过保持批次完整，在突发、可变长度流量下维持每秒高令牌数
连续配料的未来
Serving teams balance throughput with time to first token and time between generated tokens. Long prefill work can delay decoding unless the scheduler splits or otherwise manages it. Capacity budgets, request priorities, and fairness policies determine which work is admitted. Continuous batching can be combined with paged caches, chunked prefill, and speculative decoding, but each combination should be measured on the actual traffic and hardware.
付诸实践
为什么这很重要
- 成本与预算. 多年来，架构决策决定着性能和运营成本。
- 更清晰的判决. 技术教育帮助团队选择正确的堆栈，而不仅仅是最新的堆栈。
- 质量控制. 更好的工程选择可以减少生产中的可靠性事故。
留意
- 优化一项基准测试可以隐藏更广泛的系统弱点。
- 基础设施和维护成本常常被低估。
- 随着系统变得更加复杂，安全性和可观察性差距可能会扩大。
后续步骤
- 在实施之前定义延迟、质量和成本目标。
- 在实际负载和数据条件下进行基准测试。
- 仪器监控错误、漂移和用户影响。
- 在扩展之前准备回滚和事件响应路径。
不断探索
常见问题
What is Continuous Batching?
Continuous batching updates the active request set between generation iterations instead of holding one fixed batch until every request finishes. It can improve serving utilization, while capacity limits and scheduling policy still govern latency.
LLM 服务的静态（固定）批处理的主要弱点是什么？
由于输出长度各不相同，因此完成的序列会处于空闲状态，直到最慢的序列完成为止，从而浪费 GPU 周期并延迟新请求。
连续批处理以什么粒度添加和删除请求？
连续（迭代级别）批处理会在每个解码步骤后更新活动集，因此它会逐个令牌而不是每个整个请求进行操作。
连续批处理请求完成并释放容量。调度器接下来可以做什么？
Iteration-level scheduling can refill available capacity subject to memory, token, and other limits.
哪种技术与连续批处理自然地结合在一起，可以使插入/删除序列变得便宜？
PagedAttention 将每个序列的 KV 缓存存储在独立的块中，因此在运行中添加或删除序列不会干扰其他序列的内存。
哪些研究论文被普遍认为推广了迭代级（连续）批处理？
Orca 论文引入了迭代级调度，该思想被 vLLM、TGI 和 TensorRT-LLM 等服务系统采用。