How do I make the cache smaller, and what does it cost?

I have a long conversation that fits the model but fills its memory. The old keys and values cannot change during causal decoding, so keeping them saves computation. Keeping all of them at full precision is another decision.

I want to separate five bargains: share heads, compress their representation, use fewer bits, retain fewer positions, or keep the positions and read fewer. I compare every storage scheme on Llama 3 70B’s shape: one sequence, 80 layers, head width 128, at 128k context, meaning 131,072 tokens128k, expanded128 × 1,024 = 131,072 tokens. These are payload estimates across the model, before sharding and allocation overhead.

They will ask Name the ways to shrink a KV cache and what each one costs.
The 30-second version Sharing KV heads reduces independent learned projections; a latent replaces the head rows with a trained bottleneck. Quantization trades precision for space and needs scales and suitable kernels. A window bounds storage but removes direct access to older content; attention sinks can help streaming stability without restoring that content. Sparse attention keeps the main cache and selects fewer reads, paying for selection and potentially extra state. Compatible reductions compose, but architecture changes need a model trained or adapted for them.

The first film changes the Transformer toy, “Diary of Jane is a song by”, in explicitly illustrative ways. I compute the changed answers before counting saved bytes. The split follows The Transformer, Part 2.

What do sharing and a latent actually remove?

The ordinary cache counts:

\[b_{\text{token}}=2\cdot L\cdot n_{kv}\cdot d_h\cdot b\]

Read it as: key and value, times layers, stored heads, numbers per head, and bytes per number. The query-head count is absent. Llama’s 64 query heads share 8 KV heads, giving 320 KiB per token320 KiB per token, worked out2 × 80 × 8 × 128 × 2 = 327,680 bytes327,680 ÷ 1,024 = 320 KiB.

BF16 scheme KV heads Per token At 128k
Multi-head 64 2.5 MiB2.5 MiB, worked out2 × 80 × 64 × 128 × 2 = 2,621,440 bytes2,621,440 ÷ 1,048,576 = 2.5 MiB 343.60 GB343.60 GB, worked out131,072 × 2,621,440 ÷ 10⁹ = 343.59738368 GB
Grouped-query 8 320 KiB 42.95 GB42.95 GB, worked out131,072 × 327,680 ÷ 10⁹ = 42.94967296 GB
Multi-query 1 40 KiB40 KiB, worked out2 × 80 × 1 × 128 × 2 = 40,960 bytes40,960 ÷ 1,024 = 40 KiB 5.37 GB5.37 GB, worked out131,072 × 40,960 ÷ 10⁹ = 5.36870912 GB

In the toy, sharing head 1’s values changes head 2’s average from [0.571, 0.429] to [0.429, 0.429][0.571, 0.429] to [0.429, 0.429], worked outThe zero query gives each of seven tokens weight 1 ÷ 7.Private value columns sum to [4, 3]; shared columns sum to [3, 3].Divide each sum by 7.. The query survives; its private representation does not. Real GQA learns that compromise. The GQA paper reports scores 47.2 for multi-head, 47.1 for GQA-8 and 46.6 for multi-query on its T5-XXL comparison. Its conversion mean-pools projections, then uptrains with 5% of original pretraining compute.

A latent caches a shorter row and reconstructs keys and values:

\[c=xD,\qquad k=cU_K,\qquad v=cU_V\]

Input times down-projection gives the latent; latent times each up-projection gives the corresponding row. My hand-built film example compresses the key instead and deliberately loses a value feature. Learned MLA trains around the bottleneck. For content keys, absorption moves reconstruction onto the query:

\[q(cU_K)^\top=(qU_K^\top)c^\top,\qquad \alpha(CU_V)=(\alpha C)U_V\]

In words: project the query before scoring latents; mix latents before rebuilding the output. DeepSeek-V2 keeps a 512-wide latent plus 64 rotary numbers. On our 80-layer comparison that is 90 KiB per token, or 12.08 GB90 KiB per token, or 12.08 GB, worked out80 × (512 + 64) × 2 = 92,160 bytes = 90 KiB131,072 × 92,160 ÷ 10⁹ = 12.07959552 GBThis uses the comparison shape, not V2’s actual layer count.. The positional component stays separate. This is an architectural choice, not a flag for an ordinary Llama checkpoint.

What do fewer bits and a window give up?

The second film keeps the arithmetic visible: integer codes decode to slightly changed keys, then a nine-token mask shows exactly what a window forgets.

For the illustrative integer quantizer:

\[z_j=\operatorname{round}(k_j/s),\qquad \widehat{k}_j=s z_j\]

Divide by the row’s scale, round to an integer, then multiply by the scale to recover an approximation. The film reserves 10% range headroom to make error visible on these unusually simple keys. This is an INT8 illustration, not FP8. I round its stored scale to BF16 too.

At unchanged GQA shape, FP8’s ideal payload is 21.47 GB21.47 GB, worked out42.94967296 × 8 ÷ 16 = 21.47483648 GB. Packed four, three and two-bit payloads would be 10.74, 8.05 and 5.37 GB10.74, 8.05 and 5.37 GB, worked out42.94967296 × 4 ÷ 16 = 10.73741824 GB42.94967296 × 3 ÷ 16 = 8.05306368 GB42.94967296 × 2 ÷ 16 = 5.36870912 GB, before scales and residual buffers. KIVI uses per-channel keys to isolate outliers and per-token values to localise error. KVQuant quantizes keys before RoPE and reports under 0.1 perplexity loss at three bits. vLLM’s April 2026 tests recovered 97 to 98% of baseline MRCR AUC at 128k with FP8 KV and attention, using uncalibrated per-tensor scales. I still evaluate my workload.

A window changes the number of retained positions:

\[B_{\text{window}}=\min(T,W)\,b_{\text{token}}\]

Keep at most the window width, counting the current token, then multiply by bytes per token. An all-layer 4,096-token window on our shape holds 1.34 GB1.34 GB, worked outmin(131,072, 4,096) × 327,680 ÷ 10⁹ = 1.34217728 GB. This is hypothetical for Llama. StreamingLLM measured Llama-2-13B at 65k with a 1,024-token window: perplexity 5158 versus 5.40 when four initial sink tokens were preserved with its window. Stability is not retrieval: evicted rows remain unavailable. Hybrid local/global layers keep some long-range access; their storage must be counted layer by layer.

Can I keep everything and still read less?

I can leave the main cache resident and choose a subset per query. DeepSeek-V3.2’s config selects 2,048 positions with 64 indexer heads. At 128k that selects 64 times64 times, worked out131,072 ÷ 2,048 = 64Only the selected main KV payload is counted, not the indexer. less KV payload. The indexer still scores the history, so this is not a total speedup or a constant-work claim. Native Sparse Attention combines compressed, selected and local branches instead.

Compatible reductions multiply. Start with multi-head, share into eight KV heads, halve precision, then select those positions: 21.47 GB resident and about 336 MB selected per step21.47 GB resident and about 336 MB selected per step, worked out343.59738368 ÷ 8 ÷ 2 = 21.47483648 GB resident21.47483648 ÷ 64 × 1,000 = 335.54432 MB selectedHypothetical compatible mechanisms; excludes selector state and traffic.. MLA replaces the representation; I cannot blindly multiply an MLA saving by a GQA saving.

For a checkpoint I already serve, I start with supported cache precision and evaluate quality. When choosing a model, I compare its GQA or MLA representation and trained attention pattern. When training, I can choose which information to share or forget. The films’ folded walkthroughs retain the longer benchmark comparisons; none substitutes for measuring my workload.

Rapid fire: can I defend each trade?

  1. Why do I count KV heads rather than query heads?
  2. What changed in the toy when two queries shared one value head?
  3. State the absorption identity. Why keep the rotary component separate?
  4. What does a quantization scale cost, and what does a window forget?
  5. Which reduction changes selected reads without shrinking the main cache?