The KV Cache · 3. Fewer bits, shorter windows, fewer reads
Actual rows, then the same 128k comparisonI count what stays resident separately from what a query reads. Toy modifications are illustrative; real-model sizes use the same 80-layer shape.
Open video ↗Read the walkthrough
An illustrative quantizer. Symmetric INT8 codes use a per-row scale equal to 1.1 times the maximum magnitude divided by 127, rounded to BF16. The 10% range reserve is a teaching choice. Before scale rounding, max/127 would represent these unusually simple toy keys exactly. Zero rows use scale 1. This is not an FP8 codec.
The actual error. Diary [1, 1, 0, 0] becomes integer codes [115, 115, 0, 0], then [0.996704, 0.996704, 0, 0]. Each nonzero error is minus 0.003296. Four INT8 entries plus one BF16 scale take 6 bytes, against 8 for the original BF16 key row.
Precision evidence retained. The ideal 8, 4, 3 and 2 bit GQA payloads are 21.47, 10.74, 8.05 and 5.37 GB. Real allocations include scales, packing and any residual buffers. KIVI quantizes keys per channel and values per token, reporting 2.35 to 3.47 times throughput at two bits from larger batches. KVQuant reports under 0.1 perplexity loss at three bits with per-channel keys before RoPE. Accuracy depends on the model and evaluation.
The FP8 evidence, checked again. The vLLM April 22, 2026 report uses uncalibrated per-tensor scales, not per-head scales, for its evaluations. It reports 97 to 98% recovery of baseline MRCR AUC at 128k for FP8 KV plus attention and model-dependent reasoning degradation. Source: https://vllm.ai/blog/2026-04-22-fp8-kvcache
A window. The four-position window includes the current token. At Benjamin, positions 6 to 9 remain: song, by, Breaking, Benjamin. The toy keeps 64 rather than 144 bytes. A hypothetical all-layer 4,096-token window on the comparison shape keeps 1.34 GB at 128k. This is not Llama's native attention pattern.
Sinks and hybrid layers. Mistral 7B reported an eightfold cache reduction with a 4,096 window at 32k. StreamingLLM reports Llama-2-13B perplexity 5158 versus 5.40 at 65k with a 1,024 window when four initial tokens are retained, and stable streaming to four million tokens. This preserves stability, not arbitrary retrieval of evicted content. The original post also notes Gemma 3's five 1,024-wide local layers per global layer and gpt-oss's alternating local/global layers with a 128-wide local window and learned per-head sink.
Sparse attention. DeepSeek-V3.2 uses a lightning indexer with 64 indexer heads and top 2,048 selection. Native Sparse Attention combines compressed, selected and local branches, reporting 11.6 times faster decoding at 64k. Sparse reads preserve the main cache payload, but selectors add work and can add storage. The indexer still scores the full history.
Composition. Hypothetical compatible mechanisms on the fixed shape: MHA to GQA divides payload by eight, FP8 by two, leaving 21.47 GB resident. Reading 2,048 of 131,072 positions selects 335.54 MB per step, a 64-fold reduction in selected KV payload only. The estimate excludes the indexer, metadata, and implementation traffic. MLA replaces the head-based representation; it is not an additional free GQA multiplier.