The KV Cache · 2. The decode bill
How much work does each byte buy?After Diary of Jane is a song by, the model feeds Breaking back in. Its query [1, 2, 0, 0] reads eight key rows, producing scaled scores [1.5, 0, 2, 0, 0, 1, 0, 2]. Softmax supplies the eight mixing weights.
Open video ↗Read the walkthrough
The toy. After Diary of Jane is a song by, the model feeds Breaking back in. Its query [1, 2, 0, 0] reads eight key rows, producing scaled scores [1.5, 0, 2, 0, 0, 1, 0, 2]. Softmax supplies the eight mixing weights.
The second product. The weighted value rows sum to [0.382, 1.310, −0.092, 0.382], rounded. Each of the two products uses 32 multiply-adds. With two FLOPs per multiply-add, that is 128 FLOPs for 128 bytes of logical BF16 KV reads: 1 FLOP per byte. Scaling, softmax and other traffic are omitted.
Grouped queries. Eight query heads reuse each KV head in Llama 3 70B. Ideal cache reuse raises the attention product intensity to 8 FLOPs per byte. The whole-model large-batch ceiling includes roughly 2P FLOPs per token: 60.5 at 8,192 tokens, 11.3 at 131,072. Both are below the H100 BF16 ridge of about 295. These are long-context results, not a claim about every possible decode workload.
The clock. Eight times 3.35 TB/s gives 26.8 TB/s. At 4k and batch 321, the ideal 571.84 GB read takes 21.34 ms: 15,044 aggregate tokens/s, or 46.9 per user. At 128k and batch 10 it is about 470 aggregate tokens/s. Communication, kernel overhead, bandwidth efficiency and scheduling make real service slower.