How does a serving engine manage the cache?
Parts 2 and 3 counted bytes as if a sequence’s cache were one clean object of a known size. It is not. Requests arrive at random times, run for lengths nobody knows in advance, share large chunks of their prompts with each other, and sometimes have to be thrown out to make room.
Everything an inference engine does about that reduces to one design decision made in 2023, and every other feature in this part is a consequence of it.
This is also the part of the topic where interviewers stop testing arithmetic and start testing whether you have actually read the systems. It is worth being able to say what vLLM does differently from SGLang without reaching for marketing words.
The 30-second version
Before it, an engine reserved one contiguous slab per sequence, sized for the longest output it might produce, so most of the reserved memory held tokens that were never generated. The vLLM paper measured existing systems using only 20 to 38 percent of their cache memory for real token state. Paged attention cuts the cache into fixed blocks, sixteen tokens by default, gives each sequence a block table mapping logical blocks to physical ones, and allocates a block only when the previous one fills. Nothing is contiguous, nothing is reserved, waste is bounded by one partly-filled block per sequence, and the paper reports under four percent lost with two to four times the throughput at the same latency. The deeper consequence is the table: once a sequence does not own its memory, blocks can be shared between requests with a reference count, which is prefix caching; evicted when the pool fills; copied to host memory; or shipped over a network to another GPU, which is how disaggregated prefill works.Why does naive allocation waste most of the memory?
The obvious design is one contiguous slab per sequence. The problem is that you have to size it before you know the answer’s length, so you size it for the maximum.
The result is waste in two places. Inside the slab, everything past the last real token is reserved and empty, which is internal fragmentation. Between slabs, the leftovers are too small to hold another reservation, which is external fragmentation.
The vLLM paper put a number on it: in the systems they measured, only 20.4 to 38.2 percent of KV cache memory held actual token state. The first step of the figure runs that allocator on eight requests. Four of them fill the pool, using about half of what they reserved, while the other four wait for memory that is sitting there empty.
What does paging change?
Paged attention borrows the operating system’s answer to exactly this problem. Cut the cache into fixed blocks, sixteen tokens each by default, give every sequence a table mapping its logical blocks to physical ones, and hand it a new block only when it has filled the last.
Nothing is contiguous and nothing is reserved. The waste per sequence is bounded by one partly filled block, fifteen tokens at worst, whatever the length of the request. vLLM reports under four percent of memory lost to it, against two to four times the throughput at the same latency.
On the model from Part 2, one block is 16 tokens of cache, 5.2 MB, and the node’s 432 GB pool is about 82,000 of them. That is the unit of everything the scheduler does.
The cost is an indirection: the attention kernel has to gather its keys and values through the table rather than striding a contiguous tensor. That is why paged attention is a kernel and not just an allocator, and why every serving engine ships its own.
What is the block table actually for?
It is worth separating the memory saving from the capability, because the second is bigger.
Once a sequence does not own its memory, four things become possible that were not. Two sequences can point at the same physical block, which is prefix sharing. A block can be dropped without touching the rest of the sequence, which is eviction. It can be copied to host memory on its own. And it can be sent over a network by itself, which is how a prompt processed on one GPU is decoded on another.
Every remaining question in this part is one of those four. That is the argument for why paged attention mattered more than the fragmentation number suggests.
What happens when two users send the same prompt?
Real traffic repeats itself constantly: a system prompt, a few-shot preamble, a document that twenty people are asking questions about. Without sharing, those tokens are prefilled again for every request and stored again for every request.
vLLM hashes each block by its contents and the prefix before it, and reuses any block whose hash it already holds; prefix caching is on by default in its V1 engine. SGLang keeps the cached prefixes in a radix tree, evicting least-recently-used leaves so that shared ancestors outlive their children, and schedules by longest shared prefix so that requests which can hit the cache run while it is still warm.
That scheduling decision is what turns a cache into a hit rate. SGLang report 52.4 and 74.1 percent hit rates on two production traces and an average 1.7 times lower time to first token, and up to 6.4 times higher throughput on their full benchmark suite against the systems of the time.
The catch is exactness. A block is reusable only if every token before it matches, so one different token anywhere in the prefix invalidates everything after it. That is the real reason production prompts are written with the stable parts first and the variable parts last, and it is a good question to ask a candidate who says prefix caching is free.
What happens when the pool fills?
It always fills. An engine needs a policy for the moment there is no free block, and there are two levers with very different costs.
Cached prefixes that no running sequence is using go first, least recently used. They are pure optimisation, so the only cost is the prefill that will have to be redone if someone asks for that prefix again.
If that is not enough, a running sequence is preempted. vLLM’s V1 engine drops its blocks and reruns its prefill when it is rescheduled, and it removed the alternative path entirely.
The arithmetic here is worth doing, because it does not say what people expect. Swapping a 4,096-token sequence out to host memory and back over PCIe is about 42 milliseconds of link time. Rerunning its prefill is about 584 milliseconds of tensor-core time at the H100’s peak.
Swapping is faster in raw wall clock, and engines still choose recompute, because it spends arithmetic that a memory-bound decode step was wasting anyway, it needs no second memory tier or pinned host buffers, and it can be chunked into batches that are already running.
Where else can the cache live?
Host memory and disk, and in 2026 that is a real tier rather than a fallback.
The distinction that matters is what the tier is for. Rescuing one preempted sequence is not worth it, for the reasons above. Keeping a prefix that a hundred later requests will hit is worth a great deal, because it turns a prefill into a transfer.
LMCache is the layer most people mean: it moves KV blocks between GPU, CPU, disk and remote stores, and reports up to fifteen times the throughput when combined with vLLM on multi-round question answering and document workloads, where the hit rate is high.
NVIDIA’s Dynamo has its own block manager with four tiers, GPU, CPU, local storage and networked object storage. TensorRT-LLM exposes a host cache size and priority-based eviction. Moonshot’s Mooncake went furthest, building the whole serving architecture around a disaggregated cache pool in the cluster’s spare CPU memory and SSDs, and reporting 75 percent more requests handled on Kimi’s production traffic.
The pattern is the same everywhere: the cache stops being a per-GPU resource and becomes a cluster-level one, with a hit rate you can measure and a routing decision attached to it.
Can the cache move between GPUs?
This is the 2026 shape of a large deployment, and it follows straight from Part 1’s observation that prefill and decode are different machines.
Run them in separate pools, and hand the cache over when the prompt is done. A 4,096-token prompt of this model carries 1.3 GB, which crosses NVLink in a couple of milliseconds and a 400 gigabit network in about 27, against roughly 600 milliseconds of prefill arithmetic that the decode pool no longer has to interleave with its steps.
DistServe, the paper that made the case, reports 7.4 times more requests or 12.6 times tighter latency targets while keeping over ninety percent of requests within their constraints. Splitwise, from the same period, reports 1.4 times the throughput at twenty percent lower cost, or 2.35 times at the same cost and power.
NVIDIA’s NIXL moves the blocks GPU to GPU over RDMA and is the transport in vLLM’s disaggregated prefill path, in Dynamo and in llm-d; SGLang supports both NIXL and Mooncake’s transfer engine.
It is not free. Below a few sequences per GPU the coordination costs more than it saves, and the whole thing assumes a fabric fast enough that a gigabyte is a rounding error. That is a large-deployment technique, not a default.
What do the three engines actually do?
They have converged, and the honest answer to “which one” is that the features are largely the same and the workload decides.
All three page the cache, batch continuously, reuse prefixes and support an FP8 cache. vLLM’s V1 engine is the reference implementation, always schedules chunked prefill, and defaults to recompute on preemption. SGLang’s radix tree makes prefix reuse a scheduling decision rather than a lookup, which is why it is the deployment DeepSeek recommends for its own models. TensorRT-LLM is NVIDIA’s, tuned kernel by kernel, and since its 1.0 release its default backend is PyTorch rather than a compiled engine.
Chunked prefill deserves its own sentence because it is the scheduling idea that ties this part to the next. Sarathi-Serve introduced it: split a prompt into near-equal chunks and add them to batches that are already decoding, so a long prefill never stalls the users behind it. They report 2.6 times the serving capacity on Mistral-7B and up to 3.7 times on Yi-34B. vLLM’s V1 scheduler is built around it and cannot turn it off.
Rapid fire: can you do these from memory?
- Name the two kinds of fragmentation in a slab allocator and give the measured range of useful memory it left.
- What is the bound on wasted memory per sequence under paged attention, and why is it a bound rather than an average?
- List four things the block table makes possible beyond saving memory.
- Why must a prefix cache hash the tokens before a block as well as the block itself?
- Compare swapping a preempted sequence against recomputing it, in both time and resource, and say which engines choose which.
- What is chunked prefill for, and what does it cost the request being chunked?
- How many bytes does a 4k prompt of Llama 3 70B transfer between a prefill and a decode GPU, and how long does that take on a 400 gigabit link?
- When is prefill and decode disaggregation a bad idea?
Part 5 takes all of it back to the number the business cares about: tokens per second, the frontier a server sits on, and why the same change helps one user and hurts the node.