Disaggregated Prefill and Decode Serving
A serving architecture that splits large language model inference into two separate phases—prefill and decode—and runs them on different pools of GPUs.
Overview
It matters because these two phases have opposite hardware appetites, and forcing them onto the same machines wastes capacity and hurts latency.
Deep Dive
When an LLM answers, it works in two stages. Prefill reads the entire prompt at once and builds the key-value (KV) cache; this is a big, parallel, compute-bound burst that saturates the GPU's math units. Decode then generates tokens one at a time, each step reading the whole KV cache—a memory-bandwidth-bound, lightly-compute trickle. Run together, a long prefill stalls everyone's decode (head-of-line blocking), and batching the two creates interference. Disaggregation puts prefill on one GPU pool and decode on another, transferring the KV cache between them over fast interconnects like NVLink or InfiniBand. Each pool is tuned and scaled independently, improving goodput, smoothing tail latency, and letting operators hit tight time-to-first-token and time-per-output-token targets simultaneously.
Technical Insight
The two phases differ in their bottleneck. Prefill processes all prompt tokens in parallel, so its FLOPs scale with prompt length and it maxes out tensor cores. Decode is autoregressive: each new token needs one forward pass that re-reads the full KV cache from HBM, so throughput is gated by memory bandwidth, not compute. Disaggregation exploits this by sizing, batching, and even choosing different parallelism for each pool, then shipping the KV cache from prefill workers to decode workers.
Strategic Impact
Cost and budget
Architecture decisions drive performance and operating cost for years.
Clearer decisions
Technical education helps teams choose the right stack, not just the newest one.
Quality control
Better engineering choices reduce reliability incidents in production.
The Future of Disaggregated Prefill and Decode Serving
Expect disaggregation to become a default in production stacks. Systems like DistServe, Splitwise, and Mooncake popularized it, and vLLM and NVIDIA Dynamo now ship disaggregated modes. Research is pushing KV-cache transfer optimizations, cache pooling and reuse across requests, dynamic re-balancing of prefill/decode ratios under shifting traffic, and tighter integration with prefix caching and chunked prefill. As context windows grow into the millions of tokens, separating these phases becomes increasingly essential for cost-effective, low-latency serving.
Real-World Implementation
A chat assistant routes long document prompts to a compute-heavy prefill cluster, then streams replies from a memory-optimized decode cluster to keep typing latency smooth.
NVIDIA Dynamo and vLLM let operators deploy separate prefill and decode worker groups so a burst of long prompts doesn't freeze ongoing generations.
Mooncake (used by Moonshot AI's Kimi) disaggregates prefill and decode and adds a distributed KV-cache pool to cut redundant prompt recomputation at scale.
A code-completion service dedicates a small prefill pool for short prompts and a large decode pool, since most cost comes from streaming many output tokens.
Risks & Guardrails
Optimizing one benchmark can hide broader system weaknesses.
Infrastructure and maintenance costs are often underestimated.
Security and observability gaps can grow as systems become more complex.
Implementation Roadmap
Define latency, quality, and cost targets before implementation.
Benchmark under realistic load and data conditions.
Instrument monitoring for errors, drift, and user impact.
Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Disaggregated Prefill and Decode Serving quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
KServe and Model Serving on Kubernetes
Frequently asked questions
What is Disaggregated Prefill and Decode Serving?
A serving architecture that splits large language model inference into two separate phases—prefill and decode—and runs them on different pools of GPUs. It matters because these two phases have opposite hardware appetites, and forcing them onto the same machines wastes capacity and hurts latency.
What is the core hardware reason for separating prefill and decode onto different GPU pools?
Prefill processes the whole prompt in parallel and saturates compute, while decode reads the KV cache each step and is limited by memory bandwidth—opposite appetites that justify separate, independently tuned pools.
What data structure must be transferred from prefill workers to decode workers?
Prefill builds the KV cache for the prompt; decode needs that cache to continue generating, so the cache is shipped over a fast interconnect to the decode pool.
Which problem does disaggregation specifically reduce in a shared-GPU setup?
On shared GPUs a long prefill burst can block ongoing decode steps; separating them prevents that interference and stabilizes tail latency.
Why can prefill be batched aggressively but decode benefits from different tuning?
Prefill processes all prompt tokens together, so larger batches feed the tensor cores well; decode generates one token at a time and is gated by memory, so it scales differently.
Which interconnects are typically used to move the KV cache between disaggregated pools?
High-bandwidth, low-latency links like NVLink (intra-node) and InfiniBand (inter-node) are needed so KV-cache transfer doesn't become the new bottleneck.