Why does a KV cache exist at all?
The GPU series ended on a memory budget with one line I kept coming back to. Weights are a fixed cost you pay once and amortise across everyone in the batch. The KV cache is not fixed, and nobody shares it.
It is also the object almost every inference interview circles back to, because it sits exactly where the model meets the machine. Ask about long context and you are asking about the cache. Ask about batch size, throughput, cost per million tokens, why the second GPU did not double anything: same answer, different question.
So this series takes it apart. Five parts on the cache itself and one question set at the end. This part is the first question anyone asks, which is not “how big is it” but “why is it there”.
The 30-second version
Generation is sequential: each new token runs a full forward pass and attends over every token before it. The keys and values of those earlier tokens depend only on the tokens themselves, not on the token being generated, so they can be computed once and kept. That is the cache: the K and V projections of every past token, for every layer and every key-value head. Without it, each step would rebuild the entire prefix, so producing n tokens costs work quadratic in the sequence length instead of linear. Queries are not cached, because each step brings a new one and never asks for an old one. What you buy with the cache is a huge reduction in arithmetic. What you pay is a table that has to stay resident and be read in full on every single step, which is what turns decoding from a compute problem into a memory problem.What is actually sequential about generation?
An LLM generates one token per forward pass. Token $t+1$ cannot begin until token $t$ exists, because token $t$ is part of its input. That is the whole shape of the problem, and everything else follows from it.
For Llama 3 70B, which is the model I will use for every number in this series, one pass reads all 141 GB of BF16 weights out of HBM to do two arithmetic operations per weight. On an H100 at 3.35 TB/s that is 42 ms of pure traffic against 143 microseconds of tensor-core time.
The step is memory bound by a factor of 295 before the cache is even counted. That number is not a coincidence: it is the H100’s ridge point from Part 2 of the GPU series, the ratio of its peak FLOP/s to its bandwidth.
The figure below runs the loop. Watch the row underneath the model: every pass leaves something behind.
Which half of attention is reusable?
Attention at step $t$ is one row of the causal matrix. A single query, scored against the keys of every token up to $t$, used to average their values:
\[o_t = \sum_{j \le t} \alpha_{tj} v_j, \qquad \alpha_{tj} = \frac{\exp(q_t \cdot k_j / \sqrt{d_h})}{\sum_{j' \le t} \exp(q_t \cdot k_{j'} / \sqrt{d_h})}\]Look at where $t$ appears. The query carries it. The keys and values do not: $k_j$ and $v_j$ are functions of token $j$ and its position, computed when token $j$ went through the layer, and nothing later changes them.
So the matrix splits along its two axes. Rows are throwaway, because each is computed once and never asked for again. Columns are reusable, because column $j$ is read by every row below it.
A cache is what you build out of the reusable half. The queries get no cache, and a candidate who says “the QKV cache” has just told the interviewer they have never written the loop.
What does recomputing the prefix actually cost?
Say the prompt is 4,096 tokens and the model generates 512 more.
Without a cache, step $i$ rebuilds the whole prefix: about $2P(p+i)$ FLOPs for the weights plus the full quadratic attention. Summed over 512 steps that is 327 PFLOP, which at the H100’s full peak, a rate no real kernel reaches, is five and a half minutes of arithmetic.
With a cache, step $i$ is $2P$ FLOPs plus attention over the cached tokens. Summed, 78 TFLOP. Four thousand times less work.
The wall clock tells a different story, and it is the interesting one. The cached run takes about 22 seconds, not 4,000 times less, because the arithmetic it saved was never the thing it was waiting on. Step 3 of the figure runs both clocks side by side.
What exactly is stored, per token?
One token’s cache entry is a fixed grid. For each layer and each key-value head, one key vector and one value vector, each $d_h$ numbers wide:
\[\text{bytes per token} = 2 \cdot L \cdot n_{kv} \cdot d_h \cdot \text{bytes per number}\]For Llama 3 70B in BF16: $2 \times 80 \times 8 \times 128 \times 2 = 327{,}680$ bytes. That is 320 KiB, the number everyone quotes, and it is the same number the GPU series used.
Nothing about the batch, the attention pattern or the 64 query heads enters it. Depth and key-value head count are the only two model choices in the formula, plus the bytes per number, which is why every compression idea in Part 3 attacks one of exactly those three.
The same formula on the other Llama 3 shapes: 8B is 32 layers and 128 KiB per token, 405B is 126 layers and 504 KiB. Depth is what makes a cache expensive, not width.
Why is prefill a different machine from decode?
The prompt does not go through the loop. Every prompt token can be projected at once, so prefill is one large matmul per layer that fills $p$ columns of the cache in a single pass.
That difference is entirely about bytes, not arithmetic. Prefill and decode do almost the same FLOPs per token. But prefill moves 35 MB per token, because 4,096 tokens share one read of the weights, and decode moves 142 GB per token, because one token does.
A factor of 4,096, from the same weights on the same GPU. Prefill lands on the compute roof and decode on the memory slope, which is why 2026 serving stacks increasingly run them on different GPUs. Part 4 and Part 5 are about living with that.
The cache is the object handed from one phase to the other. Prefill writes it, decode reads it, and if you disaggregate the two, the cache is the thing that goes over the wire.
What did the cache actually cost you?
It is not free, it is a different currency. You removed work that grew with the square of the context and installed a table that grows linearly with it, must stay resident, and is read in full on every step, for every sequence in the batch.
At batch 1 and 4k of context the table is 1.3 GB against 141 GB of weights: noise. At batch 32 it is 43 GB, a quarter of the traffic. At batch 128 and 4k it is bigger than the model.
There is a number worth memorising for this model: 430,000 cached tokens weigh exactly as much as the weights do. Batch times context is the only thing that matters, and how you got there does not. A thousand short chats and eight long documents can occupy the same memory to the byte.
That product, and what it does to a server, is the whole of Part 2.
What does this change about how you answer?
Three habits, and they are worth practising out loud.
When someone asks why decoding is slow, do not start with attention. Start with the weights: at batch 1 the token rate is bandwidth divided by model bytes, and that alone explains most single-stream numbers people quote.
When someone asks why the cache exists, give the causality argument, not the speed argument. Keys and values do not depend on the token being generated. That is the reason a cache is possible at all, and the reason queries are not in it.
And when someone asks what the cache costs, name both columns: capacity, because it has to be resident, and bandwidth, because it is read again on every step. Most candidates name the first and stop, and the second is the one that decides throughput.
Rapid fire: can you do these from memory?
- Why can keys and values be cached while queries cannot? Answer with the dependence, not with the speedup.
- Write the bytes per token for a model given layers, KV heads, head dimension and dtype, then do it for Llama 3 70B.
- Generating n tokens with no cache costs work that grows how, and with a cache?
- Why is the wall-clock speedup from caching far smaller than the FLOP saving?
- Prefill and decode do about the same arithmetic per token. What differs, and by what factor at a 4k prompt?
- How many cached tokens of Llama 3 70B weigh the same as the model's BF16 weights?
- Two servers hold the same number of cached tokens, one as many short chats and one as few long ones. What is different about their memory, and what is different about their throughput?
Part 2 does the arithmetic properly: bytes per token, per sequence, per batch, and the point where one node of eight H100s runs out of room.