The KV Cache · 2. The node budget

How many full conversations fit?

Eight 80 GB H100s give 640 decimal GB. I assume 92% for the runtime and reserve another 2 GB per GPU for activations, graphs and workspace. Subtract the series' rounded 141 GB of weights: 431.8 GB remains for KV payload. This is a chosen budget, not a library default.

Open video ↗
Read the walkthrough

The ledger. Eight 80 GB H100s give 640 decimal GB. I assume 92% for the runtime and reserve another 2 GB per GPU for activations, graphs and workspace. Subtract the series' rounded 141 GB of weights: 431.8 GB remains for KV payload. This is a chosen budget, not a library default.

The capacity. 320 KiB means 327,680 bytes per token. At 32,768 tokens, one user holds 10.73741824 GB. Floor division gives 40 users; the same pool gives 321 at 4,096 or 10 at 131,072. These are capacity probes for the architecture, not claims about a checkpoint's supported context window. Distinct prefixes, equal lengths, no extra allocator overhead.

The crossover. Cache equals the 141 GB weights at about 430,298 total cached tokens: about 3,362 per request at batch 128. The original post's 3,364 used 141.1 GB; this film consistently uses the series' rounded 141 GB.

FP8. Halving cache element bytes gives 160 KiB per token and 80 users at 32k before scale metadata. Weights stay BF16. Quality, kernels and allocation overhead still need checking.

Reads. A step reads the shared weights plus all requests' private KV payloads. At 4k, batch 32 reads 183.95 GB, about 23% cache; batch 321 reads 571.84 GB, about 75% cache.