The KV Cache · 5. More tokens per read

One past, two serving budgets

The target verifies Breaking, Benjamin and a full stop with the actual toy matrices. Then I count useful tokens per read and check where speculation stops paying.

Open video ↗
Read the walkthrough

The scheduling boundary. I cache Diary through song, leaving by as the boundary token. The three input rows by, Breaking and Benjamin predict Breaking, Benjamin and the full stop in one causal pass. If prefill already processed by, the first target distribution is already available. EOS is not fed back.

Actual matrices. The three queries share the saved six-row prefix, plus their own new keys and values. The mask hides later draft rows. Softmax and weighted values exactly match sequential decoding, using full precision internally.

The decision. The unchanged residuals, norms, FFN and vocabulary projection yield target probabilities 0.612, 0.393 and 0.532. All proposals match the target argmax, so greedy verification accepts all three. At a mismatch only the earlier matching prefix survives; a target correction replaces the first mismatch.

Payload accounting. In BF16 the toy KV pair costs 16 bytes. A fused read of nine rows is 144 bytes; serial reads of seven, eight and nine rows total 384. With W weight bytes the comparison is W + 144 versus 3W + 384. Writes, activations, draft costs and actual kernel rereads are excluded. For Llama at a 4,096-token first read, the analogous target payloads are 142.343 GB versus 427.028 GB.

The expectation. Assuming conditional acceptance probability a at every position, a draft of k has expected accepted prefix a + ... + a^k. Total emitted tokens include a correction or bonus: 1 + a + ... + a^k. At k = 5 and a = 0.8 these are 2.68928 and 3.68928. EOS truncates the run; this formula assumes continuing generation. Exact stochastic sampling requires rejection correction, not greedy matching.

The crossover. Verification of five drafts plus the extra target position uses six query rows. In the ideal roofline model at context 1k it becomes compute bound near batch 55. Ignoring draft time, speedup is 3.69 at batch 1, 1.82 at 128 and 0.61 at the 1,286-sequence pool limit. At 128k the ten-sequence limit remains memory bound: ideal gain 3.69. These are model outputs, not implementation promises.

What changes the bill. Cache compression reduces capacity and read traffic. Sparse reads reduce traffic without freeing stored rows. Paging and prefix reuse recover waste. Better drafts raise useful tokens per expensive read. I choose based on the measured bottleneck and the latency target.