The KV Cache · 4. When the pool fills
One table, real rows, and the resources behind each policyI count the bytes and the resource clocks behind eviction, swapping and a cache handoff.
Open video ↗Read the walkthrough
Pressure. Evict unused cached prefixes first. If that is insufficient, a running request can be preempted. Recompute retains token ids and rebuilds the rows later; swapping retains state in host memory. vLLM V1 defaults to recompute and removed the old GPU to CPU swap preemption path. Admission control limits repeated preemption.
Two resource clocks. For 4,096 tokens the cache is 1,342,177,280 bytes. An out-and-back transfer at an ideal 64 GB/s costs 41.94 ms. The approximate 2PT forward-work calculation with 70.55B parameters and one H100-equivalent at 989 TFLOP/s gives 584.37 ms. It omits attention and is not a deployed-model latency: 70B weights are sharded. Recompute can use otherwise available arithmetic and be chunked; swap uses shared links and host buffers. Neither number proves a universal policy.
Other tiers. LMCache moves blocks between GPU, CPU, disk and remote stores; it reports up to fifteen times throughput with vLLM on repeated question-answering and document workloads. Dynamo manages GPU, CPU, local storage and networked object storage. TensorRT-LLM exposes host cache capacity and priority-based eviction. Mooncake uses spare CPU memory and SSDs and reports 75 percent more requests on Kimi production traffic. These workload-specific results are retained from the original post. Prefix reuse and rescuing one preempted request are different reasons to keep a second tier.
Handoff. A 32k prompt means 32,768 tokens and 10,737,418,240 bytes, or 10.74 GB across all model shards. Ideal serialized times are 23.86 ms at 450 GB/s directional NVLink, 214.75 ms at 400 Gb/s (50 GB/s), and 858.99 ms at 100 Gb/s (12.5 GB/s). NVIDIA specifies 900 GB/s bidirectional H100 NVLink bandwidth; dividing by two supplies the directional ceiling used here. Actual sharded transfers depend on topology, concurrent paths, overhead and contention. NIXL transports state in vLLM, Dynamo and llm-d; SGLang supports NIXL and Mooncake. DistServe reports 7.4 times more requests or 12.6 times tighter latency targets with over ninety percent meeting constraints. Splitwise reports 1.4 times throughput at twenty percent lower cost, or 2.35 times at equal cost and power. These paper results do not predict this handoff.
Engines and scheduling. All three page, batch continuously, reuse prefixes and support FP8 cache. vLLM V1 uses block hashes and recompute preemption. Its current documentation says chunked prefill is enabled by default whenever possible, correcting the old post's unconditional statement. SGLang combines radix caching with scheduling, and is recommended by DeepSeek for its models. TensorRT-LLM uses PyTorch as its default backend since 1.0. Sarathi-Serve introduced chunked prefill and reports 2.6 times serving capacity on Mistral-7B and up to 3.7 times on Yi-34B. Chunking splits a prompt among batches already decoding; the chunk size trades prompt completion time against interruptions to decode.