The KV Cache · 5. The batch dial

One past, two serving budgets

One shared weight read, one cache per sequence: I compute the step time, both token rates and the wall that longer context moves.

Open video ↗
Read the walkthrough

The setup. Llama 3 70B in BF16, eight H100s, rounded 141 GB weights, 327,680 bytes per cached token and 26.8 TB/s aggregate bandwidth. Distinct equal-length requests and ideal payload reuse; these rates are not benchmarks.

One division. At batch 32 and context 4,096, the cache is 42.94967296 GB. The step reads 183.94967296 GB, taking 6.8638 ms in the bandwidth estimate: 145.7 tokens/s per user and 4,662 total.

The wall. The retained 431.8 GB example cache pool holds 321 sequences at 4k, 40 at 32k or 10 at 128k. Approximate total rates are 15,044, 1,879 and 470 tokens/s. This pool is a budget assumption, not a serving framework default.

Both resources. Arithmetic counts approximately 2P plus dense attention work per query. The H100 ridge is about 295 FLOPs/byte. Ordinary 4k decode reaches 85 at the pool limit; capacity stops it before compute does.

The frontier. Writing u for per-user tokens/s and R for total tokens/s gives W u + (T c) R = bandwidth. Real kernels, communication and scheduling push rates below this ideal line.