Pull, Don't Recompute: Peer-to-Peer KV Cache Sharing in llm-d
A request arrives and its 48K-token prefix is already sitting in a KV cache, but the endpoint holding it - one schedulable serving instance: either a complete engine instance or a rank within a multi-rank deployment - is busy. Route the request to the cache owner and it queues behind that backlog; route it to an idle endpoint and it spends seconds recomputing work the cluster already did. Waiting and recomputing are both wrong answers. P2P (peer-to-peer) KV cache sharing adds the third option: send the request to the best endpoint for load, and copy the finished KV to it.
A busy owner is only one way locality breaks. Load balancing may select another endpoint, or the required KV may have been generated by a different serving role. In every such case the llm-d router already knows how much of a request's prefix each candidate endpoint holds. With P2P, that knowledge becomes a transfer instruction: route the request to the best endpoint overall, then tell it where to fetch the missing prefix.
On GLM-5.2-FP8 at concurrency 64, P2P increased successful throughput
(requests completing end-to-end within the fixed 300-second window) with
both prefix-cache routing methods: 4.6% with approximate routing and 9.7%
with precise routing. Precise routing with P2P led all four configurations
at 11.1% above the baseline. Across three repeat comparisons, the combined
policy improved successful throughput by 9.6% on average.
This is not a universal speedup. When routing already produces a local hit, P2P correctly does nothing. When locality conflicts with load balance or serving topology, it can replace seconds of repeated computation with a peer transfer. The operating principle is simple: local hits first; portable reuse when locality breaks.
The rest of this post answers four questions: how expensive is a pull; whether it improves load-balanced serving; whether it preserves session history across prefill/decode (P/D) roles; and when it should stay inactive.
The Gap: Prefix Caches Are Per-Endpoint
Shared system prompts, common documents, agentic loops, and multi-turn conversations all repeat long prefixes. Reusing their KV skips a large portion of prefill work, reduces time to first token (TTFT), and leaves more GPU cycles for decode. For more background, see KV-Cache Wins You Can See.
llm-d already improves reuse in two ways. Prefix-aware routing sends requests to the endpoints that hold their prefixes. KV offloading keeps copies in a larger CPU tier, and a filesystem backend can extend that tier to local or shared storage for persistence and larger capacity.
Routing alone cannot make every good placement local, while shared storage adds an infrastructure dependency and a storage data path. Often the required blocks are already in another endpoint's CPU tier, one network hop away. P2P cache sharing uses that middle path.
How P2P Works
P2P pulling is a small, opt-in scheduling step. It is off by default because the crossover and available CPU-tier and fabric capacity are deployment-specific; without calibration, a short-prefix pull can cost more than recomputation. Every participating vLLM instance can serve two roles, selected per request:
- Consumer. Pulls matching KV blocks from a peer instead of computing them locally.
- Producer. Serves blocks from its CPU offload tier when another peer requests them.
The consumer sends the producer the hashes of the blocks it needs. The producer reports which blocks remain available, then writes the matching blocks over NIXL, the NVIDIA Inference Xfer Library. The llm-d router describes the operation as a pull because the consumer requests it; after the lookup handshake, the producer performs the data-path write. Neither GPU performs the peer-to-peer copy, so serving a peer costs the producer CPU memory bandwidth and network capacity, not GPU compute. The transfer copies reusable blocks to the destination; the producer keeps its copy. An instrumented GLM run below verifies that the router selected a remote source and the engine completed the peer transfer. A normal miss falls back to computation.
The EPP selects the destination and source; the sidecar passes that decision to the engine; NIXL moves matching blocks CPU-to-CPU.
The router's Endpoint Picker (EPP) supplies the source decision from its
prefix index. It compares the peer with the most cached tokens against the
endpoint selected to run the request. If the peer holds enough more cached
prefix tokens than that destination (minCachedTokenDelta) to beat the
measured transfer crossover, the EPP names it as the KV source; a tie or
self-match stays local. Among near-tied holders, the EPP samples the source
by load, weighting peers inversely to their waiting-queue depth as a proxy
for producer pressure, so concurrent pulls spread across producers instead
of converging on one. The main deployment builds its source map from KV
events reported by the engines. The GLM comparison also tests the
placement-derived approximate index. Both aim to locate reusable state. The
approximate index infers that location from placement history, so its view
can drift from what the engines actually hold.
P2P also composes with P/D disaggregation. A prefill worker can pull history generated by a decoder, compute only the new portion, and continue the normal P/D flow. No application change is required.
| Situation | P2P behavior |
|---|---|
| Placement finds a local prefix | Stays idle; moving zero bytes is optimal |
| Load balancing selects another endpoint | Pulls the prefix instead of recomputing it |
| A later P/D turn needs decode-generated history | Copies that history to the prefill worker |
What We Measured
We evaluated four models, from an 8B dense model to a 753B wide-EP MoE, across aggregated and P/D-disaggregated deployments. The experiments make two kinds of claims. In an isolated A/B, P2P is the only policy difference. In a system-policy comparison, placement or the CPU offload stack changes with it, so the result belongs to the complete serving policy.
The three anchor results below establish the transfer economics, compare a complete wide-EP routing policy under saturation, and measure the P/D session-continuity payoff. The document Q&A and smaller-model experiments then show where the same mechanism appears under different serving policies.
1. Price the Transfer Before Using It
Pulling is useful only when it costs less than recomputing. The crossover depends on the model, KV representation, hardware, and network, so it must be measured rather than assumed. A fresh prefix in these experiments means a newly salted token sequence that is seeded on the source and absent from the consumer before the timed request.
Setup: a single pod pair, KV source injected directly, no router in the path
The driver seeds a fresh prefix on the source pod and requests it on the
consumer, injecting the KV source into the request rather than letting the
router decide. This measures the transfer itself rather than any placement
policy. Transfers run CPU-to-CPU over NIXL (UCX) with rdma/ib devices
exposed to both pods. The transport sets the crossover, so the
guide
recommends deriving the threshold from a crossover measured on your own
fabric.
On openai/gpt-oss-120b with H200 GPUs, the pull won at every measured prefix
length, so the crossover sits below the smallest measured prefix of 2K tokens.
The gap also widens with length: the transfer moves bytes at network
bandwidth, linear in prefix size, while recompute pays prefill FLOPs per token
plus an attention term that grows quadratically with context:
| Prefix tokens | Recompute | P2P pull | Delta |
|---|---|---|---|
| 2,048 | 78 ms | 35 ms | -56% |
| 8,192 | 250 ms | 57 ms | -77% |
| 16,384 | 510 ms | 86 ms | -83% |
| 32,768 | 1,173 ms | 165 ms | -86% |
| 49,152 | 1,988 ms | 235 ms | -88% |
Recompute grows far faster than the peer transfer; the advantage reaches 88% at 48K tokens.
The crossover moved on the 753B GLM testbed, whose KV footprint is about 93 KB per token. Pull and recompute were roughly tied near 8K tokens; at 12K the pull was 27% faster, and at 24K it was 61% faster. A 24K prefix is about 2.1 GiB of KV, while a 70K prefix is about 6.2 GiB. Break-even depends on the ratio of prefill cost to KV-transfer cost, not model size alone. On the GLM rig, short-prefix recompute stayed below the measured 1.2-1.3 second pull floor, so P2P began winning only around 8.7K tokens.
This is why the EPP uses a per-deployment minimum cached-token advantage rather than pulling every remote hit. The production threshold should sit above the measured crossover and include margin for fabric contention and producer load.
2. Reuse Cached State When Placement Changes
Approximate and precise routing try to answer the same question: where does reusable state live? Approximate routing infers the answer from prior placements and expected cache behavior. That inference often keeps a session on the same worker, but it can diverge from actual residency, including after evictions. Precise routing instead follows the cache state reported by the engines. It has a direct view of residency, but a cache-rich worker may not be the best place to run the next request.
P2P works with either view. It lets the llm-d router choose where the request should run, then reuse state held by another worker.
The GLM-5.2 agentic-serving study describes the production coding-agent trace shape and wide-EP serving architecture. This experiment crosses both routing methods with and without P2P under saturation.
Setup: GLM-5.2-FP8 on 32x H200 P/D-disaggregated; one four-configuration comparison plus three repeat comparisons; AIPerf trace replay at concurrency 64
The 753B MoE runs on two prefill and two decode instances, each 8-way data/expert parallel. AIPerf replays 48 entries from the SemiAnalysis Weka coding-agent trace corpus with seed 67. The four-configuration comparison covers approximate and precise routing, each with and without P2P. Each configuration starts on fresh engine pods and uses the same 300-second admission window plus a 120-second drain. Only requests that reach a terminal state by 300 seconds enter the result, so this is a matched saturation snapshot rather than a measurement through full workload completion.
The repeated baseline is calibrated approximate routing without P2P. The candidate combines DP-aware precise cache-location data with peer retrieval using a 2,048-token pull threshold. All four configurations use the same calibrated prefill capacity and 55-second affinity-penalty budget.
The 2,048-token threshold is below the separate 12,288-token crossover recommendation. It describes the measured policy, not a production tuning recommendation.
The four-way comparison is one saturation snapshot. The repeat comparisons test whether the combined policy holds.
P2P improves successful throughput with either routing method. The combination of precise routing and P2P delivers the best performance.
| Routing policy | Successful req/s | Change from approximate without P2P | Median TTFT |
|---|---|---|---|
| Approximate routing | 2.890 | baseline | 2.691 s |
| Approximate routing with P2P | 3.023 | +4.6% | 2.184 s (-18.9%) |
| Precise routing | 2.927 | +1.3% | 2.899 s (+7.7%) |
| Precise routing with P2P | 3.210 | +11.1% | 2.018 s (-25.0%) |
Without P2P, the two routing modes stayed close in this snapshot. That does not make them equivalent: approximate routing infers residency from placement history, while precise routing uses cache state reported by the engines. P2P improved both. The highest successful throughput came when it could pull from the reported cache holder.
| Across three repeat comparisons | Average improvement |
|---|---|
| Successful requests per second | +9.6% |
| Input-token throughput | +11.8% |
Successful throughput improved in every repeat. Tail latency varied, so the repeatable result is higher capacity.
| Mechanism check | Result |
|---|---|
| Approximate routing with P2P | 77.7 GiB submitted; no failed transfer rounds |
| Precise routing with P2P | 39.4 GiB submitted; no failed transfer rounds |
| Both configurations without P2P | Zero peer transfers |
| Prefill queue p90 across repeats | 12.8-13.7 requests at baseline; 8.0-9.0 with precise routing and P2P |
The byte counts come from P2P submission records.
A separate GLM test isolates P2P under identical placement:
| Metric | Without P2P | With P2P | Change |
|---|---|---|---|
| Mean TTFT | 7.85 s | 2.56 s | -67% |
| Throughput | 3.80 req/s | 10.10 req/s | 2.7x |
The result reproduced in a separately built fleet.
A smaller Llama-3.1-8B shared-prefix pool provides the clean P2P-only A/B.
At 8 requests per second, P2P reduced median request latency by 43%. Near
saturation, it raised the fleet ceiling by 22% and peak token throughput by
32%. The gain grew with load because the no-pull control consumed capacity
recomputing cross-pod misses:
| Offered rate | Without P2P | With P2P |
|---|---|---|
| 4 req/s | 1.12 s p50 | 0.93 s p50 |
| 8 req/s | 2.49 s p50 | 1.41 s p50 (-43%) |
| 12 req/s | 12.2 s p50, 9.9 req/s achieved | 2.1 s p50, 11.6 req/s achieved |
| 16 req/s | 21.3 s p50, 10.3 req/s achieved | 7.8 s p50, 12.6 req/s achieved (+22%) |
Setup: 4 aggregated H200 pods; identical load-balanced placement on both sides
A pool of 64 shared 16K prefixes, larger than any single pod's cache. Both runs use the same load-balanced placement; the treatment additionally enables P2P KV cache sharing.
Document Q&A: When Locality Becomes a Queue
This experiment compares precise affinity with load-aware placement plus P2P when active sessions queue behind their document owners.
Setup: gpt-oss-120b on 16 aggregated H200 pods; precise affinity versus load-aware placement with P2P
The baseline uses precise prefix affinity. The candidate combines load-aware placement with P2P using a 2,048-token pull threshold. Both use the same precise cache-location data.
The workload used 192 distinct 48K-token documents, six turns per document, and 128 active conversations.
| Metric | Precise affinity | Load-aware placement with P2P |
|---|---|---|
| Median TTFT | 0.3 s | 0.6 s |
| p99 TTFT | 25.2 s | 16.6 s |
| Throughput | baseline | +35% |
| Client timeouts on a cold fleet | 48 | 0 |
Affinity wins the cheapest local hit. Load-aware placement with P2P improves tail latency and throughput when document owners become queueing hotspots.
3. Preserve Session History Across P/D Roles
Multi-turn P/D sessions create a locality break that placement cannot eliminate. The decoder generates the newest KV history, but a prefill worker handles the next turn. Without a peer transfer, the prefiller rebuilds history that already exists on another serving role.
The small-model run established the mechanism before the larger workload showed a user-visible payoff:
| Model | Session history moved | Result |
|---|---|---|
Llama-3.1-8B | 477K-1.65M tokens | Reclaimed prefill capacity; TTFT stayed near parity |
Qwen3-30B-A3B-Thinking | 1.23M tokens | Lower TTFT and higher throughput |
Setup: 2 prefill and 4 decode pods; identical placement, the P2P side adds the offload tier and the pull
One H200 per pod. Both runs use NIXL for the P/D handoff and identical placement. The P2P run adds the CPU offload tier and enables peer retrieval using a 1,024-token pull threshold.
The agentic-serving workload uses 24 conversations, 10K-100K-token system prompts, 4-40 turns, and 1-20-second tool-call gaps. Prompts average 61.9K tokens.
| Metric | Without P2P | With P2P | Change |
|---|---|---|---|
| Median TTFT | 6.83 s | 1.09 s | 6.3x lower |
| Throughput | 0.82 req/s | 1.24 req/s | +50% |
Moving session history across serving roles reduces repeated prefill work.
P2P did not improve the first prefill of a cold context because no reusable KV existed yet. It removes repeated work, not the first computation.
Where P2P Should Stay Inactive
The negative controls define when P2P should remain quiet:
| Situation | Expected behavior |
|---|---|
| Placement already finds a local hit | No peer transfer |
| KV is restored from the same endpoint's CPU tier | Local restore, not P2P |
| The routing index has restarted | No peer source until new cache state arrives |
| The prefix has not been seen before | Compute it once; reuse it later |
These controls are why enabling P2P is not enough evidence. A valid benchmark must show that placement created a useful remote source and, when causal attribution matters, that matching blocks actually moved.
The Operational Rule
Calibrate before enabling pulls broadly. Measure recompute versus transfer on a warmed peer pair, choose a threshold above the crossover, and verify transferred bytes rather than inferring success from timing alone. The first connection between two peers may pay a one-time session-establishment cost that steady-state pulls do not, so a single cold probe prices the transient rather than the data path.
Every peer must use identical block-size and hash-seed settings. A mismatch produces different block hashes and silently degrades P2P to zero matching transfers.
The P2P KV Cache Sharing guide carries the deployment manifests, CPU-tier sizing, compatibility rules, verification gates, calibration workload, and complete benchmark reports.
What's Next
The results above establish the basic economics and two production-shaped payoff cases. The next step is to measure P2P under fleet changes and traffic patterns that create locality breaks dynamically:
- Hot-prefix skew. Drive a non-uniform prefix distribution that concentrates work on a few cache owners, then measure per-worker prefill balance and p99 TTFT with load-aware placement and P2P.
- Scale-out warmup. Add a cold replica under steady shared-prefix traffic and compare how quickly it reaches useful TTFT and cache-hit levels with and without peer pulls.
- Restart and preemption recovery. Restart a prefill fleet under live multi-turn sessions, where every conversation must recover its context at once. Without pulls this is a synchronized recompute storm; with them, the decode tier serves the history back. Measure whether the per-pull saving compounds into the fleet mean under that synchronized pressure.
- Prefetch ahead of arrival. Trigger the pull from local or remote CPU when a session's next request is predictable, so the transfer overlaps idle time instead of the request's critical path.
Local Hits First, Portable Reuse When Locality Breaks
The transfer does not create compute or network capacity. It decouples placement from cache locality, allowing the llm-d router to optimize for load or topology without automatically paying the full recompute cost. The crossover measurement prices each transfer; the fleet experiments show how that price compounds into the tail latency and throughput users feel.
Keep the request with the cache when that is the best placement. When load or topology requires a different worker, copy the KV instead of recomputing it.