A 70B model on eight H100s: how many 32k users fit?
I want a number I can defend at a whiteboard. The model fits across the GPUs, but that does not tell me how many conversations fit beside it. Each user brings a history, and every generated token makes that history longer.
I use the Llama 3 70B shape throughout: 80 layers, 64 query heads, 8 KV heads, head width 128, and the series’ rounded 141 GB of BF16 weights. Here 32k means 32,768 cached tokens. The long contexts below are memory probes for this architecture, not claims about a checkpoint’s supported window.
The 30-second version
Subtract the BF16 weights and an explicit runtime reserve from the node's memory. Divide the remaining cache pool by one conversation's cache: this budget fits 40 usersWhole conversations, worked outPool: 8 × (0.92 × 80 − 2) − 141 = 431.8 GBOne user: 32,768 × 327,680 bytes = 10.73741824 GBFloor of 431.8 ÷ 10.73741824 = 40. That assumes distinct prefixes, equal lengths and evenly sharded storage. I then check bandwidth: fitting the histories does not tell me how quickly I can read them every step.The first film fills the node’s memory ledger, then changes the conversation length and the cache precision. Watch what stays fixed.
Which division gives me the user count?
A cached token keeps a key and a value in every KV head of every layer:
\[b_{\text{token}}=2L n_{kv}d_h b,\qquad B_{\max}=\left\lfloor\frac{C_{\text{pool}}}{T b_{\text{token}}}\right\rfloor\]Read the first as two vectors times layers, stored heads, row width and bytes per number. Read the second as cache pool divided by one user’s bytes, rounded down. Queries are absent from storage. Part 1 explains why.
BF16 gives 320 KiB per tokenBytes per token2 × 80 × 8 × 128 × 2 = 327,680 bytes327,680 ÷ 1,024 = 320 KiB. Storing all 64 heads instead would cost 2.5 MiBWithout grouped queries327,680 × 64 ÷ 8 = 2,621,440 bytes2,621,440 ÷ 1,048,576 = 2.5 MiB. Per-token sizes use binary units; aggregate GB below are decimal.
I choose 92% of HBM for the runtime, then reserve another 2 GB per GPU for activations, graphs and workspace. That leaves 431.8 GBThe node ledgerHBM: 8 × 80 = 640 GBHeadroom: 0.08 × 640 = 51.2 GBWorkspace: 8 × 2 = 16 GB640 − 51.2 − 16 − 141 = 431.8 GB, about 1.3 million cached tokensThe token pool431.8 × 10⁹ ÷ 327,680 = 1,317,749.02Payload budget before allocation overhead; leave room for generation.. This is my budget assumption, not a library default: vLLM v0.8.4 documents 90%.
A 32k user takes 10.74 GBOne conversation32,768 × 327,680 ÷ 10⁹ = 10.73741824 GB. The same division gives 321 users at 4k or 10 at 128kChange only the context4k: floor(431.8 ÷ 1.34217728) = 321128k: floor(431.8 ÷ 42.94967296) = 10. These are occupancy limits, not admission targets for growing conversations.
The cache equals the weights when batch times context reaches about 430,000 tokensCache equals weights141 × 10⁹ ÷ 327,680 = 430,297.85 tokensAt batch 128: 430,297.85 ÷ 128 ≈ 3,362 tokens each. Busy servers get there at ordinary conversation lengths.
FP8 halves the payload to 160 KiB per tokenOne byte instead of two320 ÷ 2 = 160 KiB, giving 80 users at 32kFP8 capacityOne user: 10.73741824 ÷ 2 = 5.36870912 GBfloor(431.8 ÷ 5.36870912) = 80 before scale metadata. Weights stay BF16; numerical quality still needs checking.
Why does a bigger batch still wait for memory?
Each decode step shares a weight read across users, but each user reads their own cache. I want to see that asymmetry in actual arithmetic. The next film feeds “Breaking” into the toy, computes its attention over eight cached rows, and counts the work beside the bytes.
The two toy matrix products use 128 FLOPs for 128 bytesThe toy’s intensityQuery times keys: 8 × 4 = 32 multiply-addsWeights times values: 8 × 4 = 32 multiply-adds2 × (32 + 32) = 128 FLOPsBF16 keys and values: 2 × 8 × 4 × 2 = 128 bytes128 ÷ 128 = 1 FLOP per byte. I count a multiply-add as two FLOPs and omit softmax, scaling and extra traffic. Ideal reuse across Llama’s grouped queries raises attention’s ratio to 8 FLOPs per byteReuse within a query group64 ÷ 8 = 8 queries per KV headEight times the toy’s work per cached byte: 8 × 1 = 8.
For the whole model, the large-batch limit is:
\[I_{\infty}\approx\frac{2P+4L n_h d_h T}{T b_{\text{token}}}\]The numerator is approximate projection and FFN work, plus the two attention products, per generated token. The denominator is that user’s cache read. Here P counts parameters and the head count is query heads: queries do arithmetic even though I do not store them.
Using the rounded weight size consistently gives ceilings of 60.5 at 8k and 11.3 at 128kWhole-model FLOPs per byteAttention: 4 × 80 × 64 × 128 ÷ 327,680 = 88k: 141 × 10⁹ ÷ (8,192 × 327,680) + 8 = 60.53128k: 141 × 10⁹ ÷ (131,072 × 327,680) + 8 = 11.28. Both miss the H100 BF16 ridge of 295. No batch reaches the compute roof for these long-context cases. Short contexts need a fresh calculation.
What token rate would I actually promise?
I first compute an optimistic bandwidth clock:
\[D=W+BTb_{\text{token}},\qquad \tau=\frac{D}{\beta},\qquad R=\frac{B}{\tau}\]Read these as bytes per step, seconds per step, then aggregate tokens per second. W is weight bytes and beta is aggregate memory bandwidth: 26.8 TB/sEight memory interfaces8 × 3.35 = 26.8 TB/sAssumes evenly sharded reads and ideal bandwidth..
At 4k and batch 321, that gives 21.34 ms per stepThe bandwidth clockCache: 321 × 4,096 × 327,680 = 430.83890688 GBTraffic: 141 + 430.83890688 = 571.83890688 GB571.83890688 ÷ 26,800 = 0.0213373 seconds: roughly 15,000 tokens/s in total, 47 per userThroughput and each user’s rate321 ÷ 0.0213373 ≈ 15,044 tokens/s1 ÷ 0.0213373 ≈ 46.9 tokens/s per user. At 128k and ten users, it is about 470 tokens/s totalSame node, longer historiesTraffic: 141 + 10 × 42.94967296 = 570.4967296 GBStep: 570.4967296 ÷ 26,800 = 0.0212872 seconds10 ÷ 0.0212872 ≈ 470 tokens/s.
I would not promise those rates. Communication between GPUs, kernel overhead, scheduling and imperfect bandwidth all take time. Even budgeting half the ideal rate is a planning guess until I measure the stack. Prefix sharing, allocation and unequal request lengths change the budget too.
For scale, NVIDIA reported 33,072 tokens/s on eight H200s for Llama 2 70B in MLPerf v5.0’s server scenario. Its quantized model, shorter sequences and different hardware cannot validate this estimate. I trust the byte arithmetic for comparisons; I benchmark before promising latency.
Rapid fire: can you defend the budget?
- Which head count belongs in cache storage, and which in attention FLOPs?
- Reconstruct the node's cache pool. Which deductions are assumptions?
- Why does doubling context roughly halve the users that fit?
- Why can batching share weight reads but not unrelated cache reads?
- What does FP8 change, and which measurements would you still need?