Why keep a sheet?
When someone proposes a layout in a meeting, or an interviewer asks whether a model fits, there is no time for a spreadsheet. Every number in this chapter came from a few multiplications, and almost all of them come down to a dozen rules. This page collects them: what to multiply, the 100B run as the worked example, and the part where each one first appeared.
Two habits make them fast. Round to one significant figure first and fix the answer afterwards: 989 TFLOP/s is about 10¹⁵, a day is about 10⁵ seconds, 1,024 is about a thousand. And say the units out loud, bytes per parameter, FLOPs per token, so a wrong multiplication shows up as a wrong unit.
Which numbers should I know by heart?
| quantity | value | what it is for |
|---|---|---|
| bytes per value | BF16 2, FP32 4, FP8 1, FP4 0.5 | every memory estimate starts here |
| H100 memory | 80 GB | the lid every budget is checked against |
| H100 dense BF16 peak | 989 TFLOP/s, about 10¹⁵ | the compute clock |
| H100 memory bandwidth | 3.35 TB/s | the HBM clock |
| NVLink, per GPU, one direction | about 450 GB/s | the link inside a node |
| network, per GPU | 400 Gb/s, which is 50 GB/s | the link between nodes, 9× slower |
| seconds in a day | 86,400, about 10⁵ | turns seconds into a calendar |
| powers of two | 2¹⁰ ≈ 10³; 8K = 2¹³; 128K = 2¹⁷; 4M ≈ 2²² | sequence lengths and batches are powers of two |
Network rates are quoted in bits. Divide by 8 before comparing them with anything measured in bytes.
How much memory does it need?
| what | rule | the 100B run | from |
|---|---|---|---|
| weights to serve | 2 bytes × P in BF16 | 200 GB | Part 1 |
| training state | 16 bytes × P: weight 2, gradient 2, FP32 master 4, two Adam moments 4 + 4 | 1.6 TB | Part 1 |
| activations, one sequence | 34 × s × h × L bytes, with FlashAttention and no recompute | 228 GB at 8K | Part 1 |
| GPUs’ worth of memory | total bytes ÷ 80 GB | 1,828 GB ÷ 80 GB = 23 | Part 1 |
| state per GPU under ZeRO | DDP 16P; ZeRO-1 4P + 12P/d; ZeRO-2 2P + 14P/d; ZeRO-3 16P/d | ZeRO-2 floor 200 GB; ZeRO-3 1.56 GB at d = 1,024 | Part 2 |
| activations with TP t and SP | 34 × s × h × L ÷ t | 28.5 GB at t = 8 | Part 3 |
| activations, TP without SP | s × h × L × (10 + 24/t) | 87 GB at t = 8 | Part 3 |
| longer context, CP over c | scale with s, divide by c | 456 GB at 128K; 28.5 GB with c = 16 | Part 5 |
| one borrowed layer (ZeRO-3) | 2P ÷ L, then ÷ t with TP | 2.5 GB; 0.3125 GB at t = 8 | Part 6 |
| a whole per-GPU budget | state + activations + workspace + one borrowed layer | candidate A: 1.56 + 28.5 + 4 + 0.31 = 34.4 GB; B at t = 4: 63.2 GB | Part 6 |
| weights kept whole (ZeRO-2) | 2P ÷ t of weights + 14P ÷ N of the rest + activations + workspace | candidate D: 25 + 1.37 + 28.5 + 4 = 58.9 GB | Part 6 |
Two checks catch most mistakes. Training state is eight times the BF16 weights, so a model that barely fits for serving is nowhere near fitting for training. And activations do not care how many parameters there are: they grow with tokens, width and depth, so a long-context run can run out of memory on activations alone.
How long will it take?
| what | rule | the 100B run | from |
|---|---|---|---|
| FLOPs per token | 6P: 2P forward, 4P backward | 6 × 10¹¹ | Part 1 |
| training FLOPs | 6 × P × D | 1.2 × 10²⁴ | Part 1 |
| cluster speed | N × peak × MFU | 1,024 × 989 × 10¹² × 0.4 = 4.05 × 10¹⁷ FLOP/s | Part 1 |
| wall-clock time | 6PD ÷ (N × peak × MFU) | 34.3 days; 68.6 at 20% MFU | Part 1 |
| optimizer steps | D ÷ tokens per step | 2 × 10¹² ÷ 4.19 × 10⁶ ≈ 477,000 | Part 1 |
| time per step | 6P × tokens per step ÷ cluster speed | 6.2 s | Part 1 |
| one micro-batchMicro-batchThe part of the batch one GPU pushes through a single forward and backward pass. A training step’s batch is too big to run at once, so each GPU runs several micro-batches one after another and adds up their gradients before the one update (gradient accumulation). In our 100B run a micro-batch is one 8,192-token sequence., per GPU | 6P × s ÷ (t × peak) | 621 ms at t = 8, at peak | Part 3 |
| attention against the rest | s ÷ 6h | 13% at 8K; 2.1× at 128K | Part 5 |
| a step with exposed additions | each GPU’s arithmetic + the additions it waits for | A: 6.21 + 0.84 = 7.05 s; B: 6.21 + 0.36 = 6.57 s | Part 7 |
| the whole run at a measured step | steps × step time | 477,000 × 6.57 s ≈ 36 days | Part 7 |
If you remember one line, remember 6PD. Everything else on this page is either bytes, which decide whether it fits, or 6PD divided by something, which decides whether it finishes.
How do the GPUs and the batch fit together?
| what | rule | the 100B run | from |
|---|---|---|---|
| GPU count | tensor × pipeline × context × groups = N | 8 × 1 × 1 × 128 = 1,024 | Part 6 |
| tokens per step | groups × sequences per micro-batch × micro-batches × tokens | A: 128 × 1 × 4 × 8,192 = 4,194,304; B: 256 × 1 × 2 × 8,192 | Part 6 |
| micro-batches per group | m = sequences per step ÷ groups | 512 ÷ 128 = 4 at PP 1; 8 at PP 2; 16 at PP 4 | Part 4 |
Every layout has to multiply back to the GPU count, and every change to one factor moves another. Deepen the pipeline and the groups shrink; keep the micro-batches per group fixed and the batch shrinks with them.
How long does the communication take?
First name the collective with two questions. Do the GPUs hold versions of the same numbers (reduce) or pieces of one tensor (gather)? Does every GPU need the whole result (all) or only its slice (scatter)? Gradients are versions and sharded weights are pieces, so gradients are reduced and weights are gathered.
| what | rule | the 100B run | from |
|---|---|---|---|
| any transfer | bytes ÷ bandwidth | 94 GB at 450 GB/s = 209 ms | Part 3 |
| all-reduce, per GPU | 2(d − 1)/d × M, about 2M | 100B BF16 gradients: 400 GB, 8 s at 50 GB/s | Part 2 |
| reduce-scatter or all-gather | (d − 1)/d × M, about M | half an all-reduce | Part 2 |
| ZeRO-1 or ZeRO-2, per GPU | reduce-scatter + all-gather, the same 2M as DDP | 400 GB | Part 2 |
| one activation message | 2 × s × b × h bytes | 167.8 MB | Part 3 |
| TP traffic per micro-batch | 4L collectives × 2(t − 1)/t × one message | 94.0 GB at t = 8 | Part 3 |
| borrowed weights under ZeRO-3 | the GPU’s weight slice, borrowed twice per micro-batch | t = 1, one micro-batch: 2 × 200 GB, plus 200 GB of gradients = 600 GB; candidate A: 8 × 25 GB ≈ 198 GB, about 4 s at 50 GB/s | Part 2, Part 6 |
Part 2 counted the last row for one micro-batch with no tensor parallelism: the weights borrowed twice and the gradients added up once, three passes of about 200 GB. Part 6 counted it for candidate A. Each GPU’s slice of the weights is 2P ÷ 8 = 25 GB, borrowed before the forward and again before the backward of each of its 4 micro-batches: 8 passes, about 198 GB and 4 s at 50 GB/s against a 6.2 s step, with the gradients on top. Whether that hides behind the arithmetic is what Part 7 measures; candidate D keeps its 25 GB slice and borrows nothing.
What about pipelines, long context and experts?
| what | rule | the 100B run | from |
|---|---|---|---|
| bubble, extra over ideal | (p − 1) ÷ m | 75% at p = m = 4; 18.75% at PP 4 with m = 16 | Part 4 |
| bubble, share of the wall clock | (p − 1) ÷ (m + p − 1) | 42.9% at p = m = 4 | Part 4 |
| interleaved, v chunks per GPU | (p − 1) ÷ (v × m) | 37.5% at p = m = 4, v = 2 (27.3% of the wall clock) | Part 4 |
| live micro-batches on stage 0 | all forward first: m; 1F1B: p | 8 against 4 at m = 8, p = 4: 57 GB against 28.5 GB | Part 4 |
| what crosses a stage boundary | one activation forward, its gradient back | 2 × 167.8 MB = 335.5 MB per micro-batch, against TP’s 94 GB | Part 4 |
| causal work, contiguous split | GPU r does 2r + 1 parts | 1 : 3 : 5 : 7 over four GPUs; zig-zag makes it 4 : 4 : 4 : 4 | Part 5 |
| groups left under context parallelism | 1,024 ÷ (tensor × context) | TP 8 × CP 16: 8 groups, each running 4 sequences of 128K | Part 5 |
| expert utilisation | mean load ÷ max load | 40% for loads 5, 1, 1, 1 | Part 5 |
What about experiments and recovery?
| what | rule | example | from |
|---|---|---|---|
| exposed communication | only the part the step waits for | a 300 ms transfer beside 800 ms of compute: 0; started at 650 ms: 150 ms; after the compute: 300 ms | Part 7 |
| speedup | old time ÷ new time, at the same tokens per step | 7.05 ÷ 6.57 ≈ 1.07 | Part 7 |
| scaling efficiency | speedup ÷ how many times more GPUs | 1.8 ÷ 2 = 90% | Part 7 |
| strong scaling with a fixed cost C | T(k) = T ÷ k + C | 1,600 + 250 ms: 5.3× at 16 workers; even perfect overlap stops at 1,600 ÷ 250 = 6.4× | Part 7 |
| failures over the run | scale a measured rate by the GPU count | Llama 3: one every 3.1 h on 16,384 GPUs, about every 2 days on 1,024; plan one a day, about 36 in 36 days | Part 8 |
| a checkpoint | 14 bytes × P: everything but the gradients | 1.4 TB; 1.37 GB from each of 1,024 GPUs | Part 8 |
| checkpoint interval | τ ≈ √(2cM), with c the save pause and M the mean time between failures | c = 2 min, M = 1 day: 76 min, 5.3% of the run, about 1.9 days of 36 | Part 8 |
| work lost per failure | about τ ÷ 2, plus the restart | 38 min at τ = 76 min | Part 8 |
| no spare node | groups = GPUs left ÷ tensor degree | 1,016 ÷ 4 = 254 groups; 512 ÷ 254 ≈ 2.02 sequences each, so the batch no longer divides | Part 8 |
Can you do these in your head?
Each one takes under a minute with the rules above. Say the answer out loud before opening it.
1. A teammate says a 70B model trains with plain DDP on 80 GB GPUs. Does it fit?
No. Training state is 16 bytes × 70 × 10⁹ = 1.12 TB on every GPU under DDP, fourteen times the lid. ZeRO-3 over 64 GPUs brings it to 1.12 TB ÷ 64 = 17.5 GB, before a single activation.
2. A 7B model, 15 trillion tokens, 256 H100s at 40% MFU. How long?
6 × 7 × 10⁹ × 15 × 10¹² = 6.3 × 10²³ FLOPs. The cluster does 256 × 989 × 10¹² × 0.4 ≈ 1.0 × 10¹⁷ FLOP/s, so it takes 6.2 × 10⁶ s, and dividing by 86,400 gives about 72 days.
3. The 8K model moves to 32K context. Activations per GPU at TP 8 with sequence parallelism?
Four times the tokens, so 28.5 GB × 4 = 114 GB: over the 80 GB lid on activations alone. Splitting the sequence over two GPUs (context parallelism, CP 2) brings it to 57 GB, before state and workspace.
4. PP 8 with TP 8 on 1,024 GPUs, the same 4,194,304-token batch, one sequence per micro-batch. What does the bubble cost?
Each group is 8 × 8 = 64 GPUs, so 1,024 ÷ 64 = 16 groups remain, and each runs 512 ÷ 16 = 32 micro-batches. Extra time is (8 − 1) ÷ 32 = 22%; the idle share of the wall clock is 7 ÷ 39 = 18%.
5. A 25 GB gradient all-reduce across 128 GPUs on 50 GB/s links. How long?
About 2 × 25 GB = 50 GB sent per GPU (exactly 2 × 127/128 × 25 = 49.6 GB), so about 1 s before latency. If it overlaps a 6 s backward pass it may cost nothing; if it runs after the backward pass, it costs the full second.
6. Saving a checkpoint stalls training for 8 minutes, and the job fails about every 6 hours (a mean time between failures of 6 hours). How often should it save, and what does that cost?
τ ≈ √(2 × 8 × 360) = √5,760 ≈ 76 min, the same interval as our run’s 2 minutes and one day, because 2cM is the same product. The cost is not: 8 ÷ 76 + 76 ÷ 720 ≈ 10.5% + 10.5% = 21%, a fifth of the run, against our 5.3%. The fix is a faster save, or one that runs in the background while training continues, not a different interval.
None of these replaces a profiler. They decide which experiment is worth the profiler’s time.