I recently tried disaggregated serving, a setup some production systems use to serve LLMs faster. Normally, each GPU both processes the prompt (prefill) and generates the answer (decode). In disaggregated serving, prefill and decode run on separate GPUs, so long new prompts do not slow down answers that are already streaming.
In my test, I compared three standard replicas, one per GPU, with a disaggregated setup on the same three GPUs, where two GPUs process the prompts and pass the KV cache to a third GPU that only generates answers. Both setups received the same traffic.
Median time per output token (TPOT) and time to first token (TTFT) as traffic grows on three A100s; dashed lines mark the latency limits.
Results
As traffic grew to 16 requests per second, the time per token (TPOT) with disaggregated serving stayed under 10 ms, while standard replicas slowed to 72 ms. The trade-off is that with disaggregation, the time to first token (TTFT) is a little longer, because each request waits for its KV cache to be passed from one GPU to another. At 12 requests per second, the first token took 0.54 s against 0.24 s for standard replicas.
Counting only the requests served within 1 s to the first token and 20 ms per token (goodput), as in DistServe [1], disaggregated serving handled 10.0 requests per second against 7.6 for three standard replicas.
Chunked Prefill
The vLLM documentation notes that disaggregation does not improve raw throughput, and that tuning chunked prefill, which splits long prompts into smaller pieces, can reach a similar goal, although the right chunk size is hard to find. So I tried that too. With a 512-token chunk budget, standard replicas slowed to 29 ms instead of 72 ms, but still served fewer requests within the latency limits than the disaggregated setup, 8.0 against 10.0 requests per second.
When Disaggregation Helps
Prefill and decode use the GPU differently, as prefill is compute-intensive and decode is memory-intensive [2]. Disaggregation is likely to help most when prompts are long, as in agentic workflows with a growing context, few-shot prompting, or questions over long documents. The DistServe authors [1] make the same argument for long contexts, since the KV cache to transfer grows linearly with the prompt, while prefill computation grows quadratically, so the transfer takes a shrinking share of the time. They also note that with only a few GPUs there is little room to rebalance prefill and decode, yet even on three GPUs, the split improved goodput for these long prompts. With short prompts, prefill takes little GPU time, so it barely slows the answers being generated. Passing the KV cache from one GPU to another then adds latency without much gain.
Setup
The model is Qwen3-4B-Instruct, served with vLLM 0.19.0 on three A100 GPUs. Each prompt is a 25-shot TeleQnA question of about 3.5K tokens, with different examples per request. The disaggregated setup runs two prefill instances and one decode instance with vLLM’s NixlConnector, and the KV cache is sent over NVLink within one node. All setups use 128-token KV blocks with prefix caching off. Chunked prefill is on for the standard replicas (the vLLM default) and off on the prefill instances in disaggregated serving.
Requests arrive at random (Poisson) at 2 to 16 requests per second, with 600 requests per rate and one run per rate. The requests-per-second figures count only the requests that meet both latency limits, at the best of the eight rates. Accuracy is the same in every setup, 67.7 to 68.5%. Real systems also turn on prefix caching, which lowers latency in either setup.
Disaggregation in vLLM
Each GPU runs its own vllm serve instance with the same NixlConnector configuration. In my runs, the prefill instances processed whole prompts, with chunked prefill off, and since all instances share one machine, each needs its own NIXL side-channel port.
KV='{"kv_connector": "NixlConnector",
"kv_role": "kv_both",
"kv_buffer_device": "cuda"}'
# Prefill, GPU 0 (GPU 1 is the same, with ports 8101 and 5601)
CUDA_VISIBLE_DEVICES=0 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve Qwen/Qwen3-4B-Instruct-2507 --port 8100 \
--block-size 128 --no-enable-prefix-caching \
--no-enable-chunked-prefill \
--kv-transfer-config "$KV"
# Decode, GPU 2
CUDA_VISIBLE_DEVICES=2 VLLM_NIXL_SIDE_CHANNEL_PORT=5602 \
vllm serve Qwen/Qwen3-4B-Instruct-2507 --port 8102 \
--block-size 128 --no-enable-prefix-caching \
--kv-transfer-config "$KV"
The standard replicas run the same vllm serve command on each GPU, without --kv-transfer-config, and the run with 512-token chunks adds --max-num-batched-tokens 512.
vLLM does not route requests between the prefill and decode instances by itself, so a small proxy sits in front of them. It sends each request to a prefill instance with max_tokens=1, then passes the kv_transfer_params that instance returns to the decode instance, which generates the answer. vLLM’s toy proxy server for NixlConnector is a good starting point.
References
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al., OSDI 2024)
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting (Patel et al., arXiv 2023)