Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Continuous batching and prefill-decode disaggregation solve different inference problems. Continuous batching is a scheduling method that keeps one serving engine productive as requests start and finish. Prefill-decode disaggregation is a placement architecture that runs prompt processing (prefill) and autoregressive generation (decode) in separate instances or pools. They are not mutually exclusive: each pool in a disaggregated design can use continuous batching.
Choose disaggregation when measurements show that prompt-heavy work is disturbing generation latency and you need independent controls for time to first token (TTFT) and inter-token latency (ITL). Keep a shared pool, often with chunked prefill, when it already meets your latency, throughput, and cost targets. Disaggregation adds KV-cache movement, routing, rate matching, reserved capacity, and operational complexity; it is not an automatic throughput upgrade.
The key distinction: scheduling versus placement
The two terms answer different questions:
- Continuous batching: How should the engine schedule requests over time so available compute stays busy while sequence lengths differ?
- Prefill-decode disaggregation: Where should each phase of inference run, and how should the intermediate KV cache move between those locations?
vLLM’s disaggregated-prefill documentation describes separate prefill and decode instances, while its engineering overview discusses continuous batching and disaggregated prefill as parts of the same serving system. A deployment can therefore use continuous batching inside a single shared engine, inside only the prefill pool, inside only the decode pool, or inside both pools.
What happens during prefill and decode?
Prefill creates the context
During prefill, the model processes the tokens already supplied in the prompt. Attention computations produce key/value (KV) state for that context. This phase usually performs substantial parallel work over many prompt tokens, so long prompts can consume significant GPU compute and memory bandwidth in a short interval. The request cannot emit its first generated token until enough prefill work has completed.
Decode generates one token at a time
During decode, the model generates output autoregressively. Each step uses the previously produced KV state, adds the new token’s state, and predicts the next token. A request may require dozens, hundreds, or thousands of decode steps. Decode work is comparatively incremental, but many concurrent sequences can keep the GPU occupied for a long period.
In a colocated engine, requests in these phases share schedulers, GPU memory, compute, and batching decisions. In a disaggregated design, a prefill worker finishes the prompt, then a decode worker receives the request and its KV state. The transfer path is implemented by a connector: the vLLM documentation describes a connector as the mechanism through which a KV consumer retrieves cache data from a producer.
How continuous batching works
Dynamic membership instead of fixed batches
Traditional static batching waits for a group of requests and processes that group together. That approach wastes capacity when one sequence finishes earlier than the others. Continuous batching treats the active set as changeable: completed sequences leave, and waiting requests enter at scheduling boundaries. The engine can continue issuing work for remaining sequences without waiting for the original batch to finish.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
This is particularly useful for decode, where output lengths vary. A short answer can leave the active set while a long answer continues. New arrivals can use the freed slots, subject to the engine’s memory, token, and fairness limits. The result is generally better utilization and more predictable queueing than fixed batches under mixed request lengths.
What it improves
- Utilization: Finished sequences do not hold a batch slot until the longest sequence completes.
- Queueing efficiency: New work can be admitted while existing requests are still generating.
- Mixed-length handling: The scheduler can account for different prompt and output lengths rather than assuming every request has the same shape.
- Operational simplicity: One pool avoids a separate inter-pool routing and KV-transfer pipeline.
What it does not isolate
Continuous batching does not create a separate resource boundary between prefill and decode. A long prompt admitted into the same engine can still compete with decode work for compute, memory bandwidth, and scheduler time. The engine can mitigate this with token budgets, priorities, or chunked prefill, but the phases remain colocated. Continuous batching also does not provide an independent way to scale prompt workers and generation workers when their demand grows at different rates.
How prefill-decode disaggregation works
Separate worker pools
Disaggregation runs prefill and decode in distinct instances or pools. A request first reaches a prefill worker, which processes the prompt and produces KV state. The request is then routed to a decode worker, which loads or retrieves that state and generates output tokens. The pools can be sized, scheduled, and scaled independently.
Independent latency tuning
TTFT is the elapsed time from request arrival until the first output token. ITL is the time between successive generated tokens; tail ITL refers to high-percentile values, such as the 95th or 99th percentile, rather than the average. Separate pools let a team choose different parallelism and scheduling strategies for these goals. Prefill capacity can be tuned for prompt-processing speed and TTFT, while decode capacity can be tuned for steady token production and ITL.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Separating the phases can also protect decode tails. If prefill jobs are not inserted into ongoing decode work, a burst of long prompts is less likely to create a sudden rise in decode ITL. This is one of the motivations documented by vLLM.
KV-cache transfer is part of the design
Disaggregation introduces a handoff between pools. The KV state must be transferred or made available to the decode worker, and the request must be coordinated across routing, metadata, and lifecycle events. Transfer time depends on the cache volume, serialization or transport method, network topology, link bandwidth, memory copies, and implementation. A connector failure, stale state, timeout, or decode-worker backpressure can prevent a request from progressing even when both pools are individually healthy.
Rank #2
- [Local AI Inference & 70B Model Ready] Equipped with the AMD Ryzen 7 PRO 8845HS processor, NEXUS is engineered for heavy local AI workloads. With a full-size GPU bay, it runs 70B LLMs natively without an internet connection. Ideal for AI developers and tech enthusiasts who need private environment for coding and model testing.
- [132TB Mass Storage with ZFS Integrity] Features a hybrid storage architecture (3×NVMe + 4×3.5" HDD) supporting up to 132TB. Utilizing the enterprise-grade ZFS file system and ECC memory, it prevents data corruption and bit rot—a must-have for professional photographers and video editors safeguarding 4K/8K RAW footage.
- [OpenClaw-Driven Automation Workflow] The built-in OpenClaw execution layer allows complex automated tasks to be processed locally. Even when offline, your backup schedules and AI file organization continue seamlessly. Say goodbye to monthly cloud subscriptions and high latency.
- [Dual 10GbE & USB4 Ultra-Connectivity] Experience server-class speeds with dual 10GbE ports and a 40Gbps USB4 interface. It enables multi-user real-time collaboration on large project files directly from the NAS, ensuring zero-lag editing for creative studios and production teams.
- [Open-Source ZimaOS for Total Privacy] Running on the fully open-source ZimaOS, NEXUS ensures your data stays physically on-premise with no backdoors. It acts as a "Digital Fortress" for privacy-conscious families and small businesses who demand absolute data sovereignty.
Why it is not a free throughput gain
The official vLLM documentation states, “Disaggregated prefill DOES NOT improve throughput.” That statement means the architectural split alone does not increase completed work. It can improve a particular latency or SLO outcome by preventing interference and allowing better resource matching, but the transfer and coordination overhead can offset gains. Throughput may improve only when the new arrangement lets the system use resources more effectively for the measured workload and service objective.
Side-by-side comparison
| Aspect | Continuous batching in a shared pool | Prefill-decode disaggregation |
|---|---|---|
| Primary abstraction | Scheduling and batch membership over time | Placement of prefill and decode in separate instances or pools |
| Resource boundary | Prefill and decode share the engine’s resources | Each phase has separately allocated resources |
| TTFT control | Managed through admission, token budgets, priorities, and prefill scheduling in the shared engine | Prefill capacity and parallelism can be tuned independently |
| ITL and tail ITL | Can be affected when prompt work competes with decode | Decode workers can be protected from new prefill jobs, subject to routing and capacity |
| KV movement | Usually remains local to the colocated engine | Requires a connector or equivalent handoff between producer and consumer |
| Scaling model | Scale the shared serving pool | Scale and rate-match prefill and decode pools independently |
| Operational complexity | Lower: one scheduler and fewer coordination paths | Higher: routing, connectors, pool sizing, autoscaling, and cross-pool observability |
| Best fit | Workloads that already meet TTFT, ITL, throughput, and cost targets, or that benefit from chunked prefill | Prefill-heavy or bursty workloads, larger models, and services with strict or separate TTFT and ITL objectives |
When disaggregation is a sensible choice
Use measured evidence rather than the presence of an LLM alone. Disaggregation is most defensible when several of the following conditions hold:
Recommended Free Tools
- Prefill is a large share of demand. Long or frequent prompts consume enough capacity to interfere with generation.
- Decode latency has a strict tail SLO. High-percentile ITL rises when prompt jobs enter the shared engine, and protecting decode is worth the extra infrastructure.
- TTFT and ITL need different tuning. The best parallel strategy for prompt processing is not the best strategy for token generation.
- The model is large or the traffic is bursty. NVIDIA Research’s June 2025 study reports that disaggregation is most effective for prefill-heavy traffic and larger models, and emphasizes dynamic rate matching and elastic scaling.
- The platform can absorb coordination overhead. You have a reliable connector, sufficient network capacity, request routing, health checks, and telemetry for both pools.
- Separate capacity is economically acceptable. Dedicated prefill workers reserve GPUs even when prompt demand is temporarily low. NVIDIA Dynamo’s versioned tuning guidance notes the latency benefit of dedicated prefill workers while warning that they reserve GPUs and require tuning.
Do not infer a universal gain from the study’s scope. The NVIDIA Research summary describes evaluations across hundreds of thousands of design points, but it does not provide one headline percentage that applies to every model, hardware layout, or traffic pattern.
When a shared pool or chunked prefill is better
Keep the shared design when targets are already met
If one continuously batched pool meets TTFT, ITL, throughput, and cost objectives at the required concurrency, splitting it can add failure modes without improving the user-visible result. A simpler topology also makes capacity planning, deployment, upgrades, and debugging easier.
Use chunked prefill to reduce interference
Chunked prefill divides a long prompt into smaller processing units and schedules those units alongside decode work. The goal is to prevent a single large prefill from monopolizing the engine or pausing ongoing generation. Sarathi-Serve’s USENIX OSDI 2024 presentation describes chunking and stall-free scheduling that admit new requests without stopping ongoing decodes. This keeps the phases on a shared serving path while addressing one of the main reasons teams consider disaggregation.
Chunk size, token budgets, fairness rules, and admission control still need tuning. Very small chunks can increase scheduling overhead; very large chunks can recreate the interference the technique is intended to limit. Measure tail ITL and TTFT after changing the policy rather than assuming that chunking is automatically beneficial.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →A practical decision and benchmark procedure
- Define the service objectives. Record target TTFT, median and tail ITL, throughput or goodput, concurrency, and the percentage of requests that must satisfy each SLO. “Latency” without phase-specific definitions is not enough.
- Characterize the traffic. Capture prompt-length and output-length distributions, arrival bursts, concurrency, prefill intensity, cancellations, and any separate tenant or priority classes. Include realistic long-context outliers.
- Measure a continuously batched baseline. Hold the model, tokenizer, quantization, hardware, software version, batching limits, and network conditions constant. Record GPU utilization, memory pressure, queue time, TTFT, ITL percentiles, completed requests, and generated tokens.
- Test shared-pool mitigations. Evaluate chunked prefill, token budgets, admission limits, and priority policies. Keep a configuration log so a change in scheduler policy is not mistaken for an architectural effect.
- Build the disaggregated variant. Use separate prefill and decode pools with an explicitly documented connector. Record cache-transfer method, cache volume, topology, bandwidth, transfer time, queue time in each pool, and failed or retried handoffs.
- Rate-match the pools. Vary prefill and decode capacity independently, then test bursts and sustained load. Observe idle GPUs in one pool while the other is saturated; that imbalance is a central cost of poor sizing.
- Compare equal conditions. Use the same prompts, output limits, concurrency, arrival pattern, model, precision, hardware class, and SLO definitions. Report both average and tail values, plus goodput at the stated SLO.
- Test failure and recovery. Drain a prefill worker, interrupt a connector, restart a decode worker, and apply backpressure. Verify whether requests are retried safely, whether KV state is duplicated or lost, and how long the system takes to return to its target rate.
- Choose the least complex design that passes. Adopt disaggregation only when its measured latency or SLO benefit justifies transfer overhead, reserved capacity, and operational burden.
Metrics that prevent misleading comparisons
| Metric or axis | What to report | Why it matters |
|---|---|---|
| Workload mix | Prompt and output length distributions, arrival pattern, prefill intensity, concurrency, and burst behavior | Disaggregation value changes materially with the request mix. |
| TTFT | Time from request arrival to first output token, with median and tail percentiles | Shows whether prompt processing and queueing meet the user’s first-token objective. |
| ITL and tail ITL | Time between generated tokens, including high percentiles | Exposes decode smoothness and interference that averages hide. |
| Throughput or goodput | Requests or tokens completed while the stated latency SLO is met | Raw tokens per second can rise while too many users violate the SLO. |
| KV transfer | Cache size, transport, topology, bandwidth, transfer duration, retries, and failures | Shows the actual cost of crossing the prefill/decode boundary. |
| Resource use | GPU allocation, utilization, memory headroom, idle time, rate matching, and autoscaling behavior for each pool | Dedicated pools can reserve capacity that is not immediately useful. |
| Operating complexity | Number of pools, routing and coordination components, connector reliability, scaling controls, and observability coverage | Production cost includes the control plane, not only model execution. |
Costs, failure modes, and mitigations
Idle or mismatched capacity
Prefill and decode demand rarely grow at exactly the same rate. Overprovisioning both pools raises cost; underprovisioning one creates queueing while GPUs in the other pool sit idle. Dynamic rate matching and elastic scaling are therefore central to a disaggregated design, not optional refinements. Use separate queue-depth, service-time, and utilization signals for each phase.
KV-transfer bottlenecks
Long prompts create larger KV state. A high-bandwidth accelerator fabric can still be limited by copies, serialization, contention, or connector implementation. Track transfer time as its own stage and set timeouts that distinguish a slow network from a failed worker. If transfer dominates TTFT, adding decode GPUs will not solve the bottleneck.
Tail-latency regressions
A split can reduce decode interference but introduce queueing at the handoff. Protect the decode pool with admission control and backpressure, and measure p95 or p99 TTFT and ITL during bursts rather than relying on steady-state averages.
Coordination and observability gaps
Every request needs a trace that follows arrival, prefill queueing, prefill execution, KV export, transfer, decode queueing, and token generation. Without phase-level timestamps, a team may blame the model for a router or connector delay. Alert separately on failed handoffs, stale KV entries, pool imbalance, and autoscaler lag.
Experimental implementation status
The current vLLM documentation labels disaggregated prefilling experimental and subject to change. It documents connectors and separate instances but does not establish a complete compatibility matrix for every software version, hardware backend, or connector. Check the documentation for the exact version and backend selected for deployment. The Dynamo guidance is likewise versioned at v0.7.1; do not assume its settings map unchanged to another release.
How to interpret published speedups
The DistServe paper reports up to 7.4× more requests or 12.6× tighter latency SLOs, with latency constraints met for more than 90% of requests in its evaluations across selected popular models, applications, and latency requirements. These are DistServe’s paper results against its stated baselines, not a guarantee for a modern deployment or a direct comparison with every continuous-batching engine. Before using the numbers in a capacity plan, read the full DistServe methodology, baselines, hardware configuration, and workloads.
The result is useful as evidence that separating phases can improve an SLO-constrained design under some conditions. It is not evidence that every disaggregated system will deliver the same multiplier, nor that disaggregation removes the need for batching, rate matching, or careful workload measurement.
Example architecture choices
Interactive assistant with short prompts
If prompts and answers are both short, traffic is steady, and a shared continuously batched pool meets TTFT and ITL targets, keep the colocated design. Add chunked prefill only if long-context outliers create measurable tail spikes. The second pool would add transfer and coordination paths without a demonstrated user benefit.
Document question answering with long prompts
Long retrieved contexts can make prefill bursts collide with ongoing generation. First test chunked prefill and admission controls. If p99 ITL remains above the SLO and prompt demand is substantial, a prefill pool sized for burst absorption plus a protected decode pool may be justified. Include cache-transfer time in the TTFT budget.
Large-model service with distinct latency tiers
When one tenant requires very fast first tokens and another requires smooth long responses, separate pools can provide clearer controls than a single scheduler. Use explicit routing and capacity reservations, then verify that the cost of reserved prefill and decode GPUs is covered by the service value.
Bottom line
Continuous batching is the default scheduling technique for keeping variable-length requests productive; prefill-decode disaggregation is an architectural split for independently managing prompt processing and generation. They can and often should be combined. Start with a continuously batched shared pool, measure TTFT, ITL, tails, goodput, and resource use, and test chunked prefill. Move to disaggregation only when prefill-heavy interference, strict latency objectives, or workload scaling justify the KV-transfer overhead and additional operations.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
