The KV Cache · 3. Sharing heads and caching latents
Actual rows, then the same 128k comparisonI count what stays resident separately from what a query reads. Toy modifications are illustrative; real-model sizes use the same 80-layer shape.
Open video ↗Read the walkthrough
The fixed comparison. One sequence, 131,072 tokens, 80 layers, head width 128. Sizes are logical payloads across the model; per-token units are binary and aggregate GB are decimal.
An illustrative split. I split the Transformer toy into two heads of width two. The first output for by is [1.458, 0.235], the second [0.571, 0.429]. Sharing head 1 keys and values preserves the first output but changes the second to [0.429, 0.429]. This is a changed architecture, not the original model forward pass.
Real head counts. With 64, 8 or 1 stored heads, BF16 payloads are 2.5 MiB, 320 KiB or 40 KiB per token: 343.60, 42.95 or 5.37 GB at 128k. The 64 query heads remain in all three comparisons. Llama 3 70B uses 8 KV heads.
Quality evidence retained from the post. The GQA paper reports T5-XXL average scores 47.2, 46.6 and 47.1 for MHA, MQA and GQA-8; inference times 1.51, 0.24 and 0.28 seconds. Its conversion mean-pools KV projections then uptrains for 5% of pretraining compute. The MQA paper reports dev perplexity 1.424 to 1.439 and beam-one BLEU 27.7 to 27.5 on WMT14 English to German.
The illustrative latent. The key row multiplies a 4 by 2 down-projection. Two stored numbers reconstruct four key numbers and four approximate value numbers. Diary reconstructs exactly; Jane loses its value entry minus one. BF16 payload falls from 16 to 4 bytes. This hand-built bottleneck is deliberately lossy and is not DeepSeek MLA.
Learned MLA. DeepSeek-V2 caches 512 latent plus 64 rotary numbers per layer per token. On the comparison shape of 80 layers this is 90 KiB per token and 12.08 GB at 128k, not the total cache of V2 itself. Content-key and value up-projections can be absorbed; the rotary component remains separate. V2 reports 93.3% smaller cache and 5.76 times maximum generation throughput against its 67B baseline; these are the paper's comparison, not predictions for Llama.