Why keep a sheet?

When someone proposes a layout in a meeting, or an interviewer asks whether a model fits, there is no time for a spreadsheet. Every number in this chapter came from a few multiplications, and almost all of them come down to a dozen rules. This page collects them: what to multiply, the 100B run as the worked example, and the part where each one first appeared.

Two habits make them fast. Round to one significant figure first and fix the answer afterwards: 989 TFLOP/s is about 10¹⁵, a day is about 10⁵ seconds, 1,024 is about a thousand. And say the units out loud, bytes per parameter, FLOPs per token, so a wrong multiplication shows up as a wrong unit.

Which numbers should I know by heart?

quantity value what it is for
bytes per value BF16 2, FP32 4, FP8 1, FP4 0.5 every memory estimate starts here
H100 memory 80 GB the lid every budget is checked against
H100 dense BF16 peak 989 TFLOP/s, about 10¹⁵ the compute clock
H100 memory bandwidth 3.35 TB/s the HBM clock
NVLink, per GPU, one direction about 450 GB/s the link inside a node
network, per GPU 400 Gb/s, which is 50 GB/s the link between nodes, 9× slower
seconds in a day 86,400, about 10⁵ turns seconds into a calendar
powers of two 2¹⁰ ≈ 10³; 8K = 2¹³; 128K = 2¹⁷; 4M ≈ 2²² sequence lengths and batches are powers of two

Network rates are quoted in bits. Divide by 8 before comparing them with anything measured in bytes.

How much memory does it need?

what rule the 100B run from
weights to serve 2 bytes × P in BF16 200 GB Part 1
training state 16 bytes × P: weight 2, gradient 2, FP32 master 4, two Adam moments 4 + 4 1.6 TB Part 1
activations, one sequence 34 × s × h × L bytes, with FlashAttention and no recompute 228 GB at 8K Part 1
GPUs’ worth of memory total bytes ÷ 80 GB 1,828 GB ÷ 80 GB = 23 Part 1
state per GPU under ZeRO DDP 16P; ZeRO-1 4P + 12P/d; ZeRO-2 2P + 14P/d; ZeRO-3 16P/d ZeRO-2 floor 200 GB; ZeRO-3 1.56 GB at d = 1,024 Part 2
activations with TP t and SP 34 × s × h × L ÷ t 28.5 GB at t = 8 Part 3
activations, TP without SP s × h × L × (10 + 24/t) 87 GB at t = 8 Part 3
longer context, CP over c scale with s, divide by c 456 GB at 128K; 28.5 GB with c = 16 Part 5
one borrowed layer (ZeRO-3) 2P ÷ L, then ÷ t with TP 2.5 GB; 0.3125 GB at t = 8 Part 6
a whole per-GPU budget state + activations + workspace + one borrowed layer candidate A: 1.56 + 28.5 + 4 + 0.31 = 34.4 GB; B at t = 4: 63.2 GB Part 6
weights kept whole (ZeRO-2) 2P ÷ t of weights + 14P ÷ N of the rest + activations + workspace candidate D: 25 + 1.37 + 28.5 + 4 = 58.9 GB Part 6

Two checks catch most mistakes. Training state is eight times the BF16 weights, so a model that barely fits for serving is nowhere near fitting for training. And activations do not care how many parameters there are: they grow with tokens, width and depth, so a long-context run can run out of memory on activations alone.

How long will it take?

what rule the 100B run from
FLOPs per token 6P: 2P forward, 4P backward 6 × 10¹¹ Part 1
training FLOPs 6 × P × D 1.2 × 10²⁴ Part 1
cluster speed N × peak × MFU 1,024 × 989 × 10¹² × 0.4 = 4.05 × 10¹⁷ FLOP/s Part 1
wall-clock time 6PD ÷ (N × peak × MFU) 34.3 days; 68.6 at 20% MFU Part 1
optimizer steps D ÷ tokens per step 2 × 10¹² ÷ 4.19 × 10⁶ ≈ 477,000 Part 1
time per step 6P × tokens per step ÷ cluster speed 6.2 s Part 1
one micro-batchMicro-batchThe part of the batch one GPU pushes through a single forward and backward pass. A training step’s batch is too big to run at once, so each GPU runs several micro-batches one after another and adds up their gradients before the one update (gradient accumulation). In our 100B run a micro-batch is one 8,192-token sequence., per GPU 6P × s ÷ (t × peak) 621 ms at t = 8, at peak Part 3
attention against the rest s ÷ 6h 13% at 8K; 2.1× at 128K Part 5
a step with exposed additions each GPU’s arithmetic + the additions it waits for A: 6.21 + 0.84 = 7.05 s; B: 6.21 + 0.36 = 6.57 s Part 7
the whole run at a measured step steps × step time 477,000 × 6.57 s ≈ 36 days Part 7

If you remember one line, remember 6PD. Everything else on this page is either bytes, which decide whether it fits, or 6PD divided by something, which decides whether it finishes.

How do the GPUs and the batch fit together?

what rule the 100B run from
GPU count tensor × pipeline × context × groups = N 8 × 1 × 1 × 128 = 1,024 Part 6
tokens per step groups × sequences per micro-batch × micro-batches × tokens A: 128 × 1 × 4 × 8,192 = 4,194,304; B: 256 × 1 × 2 × 8,192 Part 6
micro-batches per group m = sequences per step ÷ groups 512 ÷ 128 = 4 at PP 1; 8 at PP 2; 16 at PP 4 Part 4

Every layout has to multiply back to the GPU count, and every change to one factor moves another. Deepen the pipeline and the groups shrink; keep the micro-batches per group fixed and the batch shrinks with them.

How long does the communication take?

First name the collective with two questions. Do the GPUs hold versions of the same numbers (reduce) or pieces of one tensor (gather)? Does every GPU need the whole result (all) or only its slice (scatter)? Gradients are versions and sharded weights are pieces, so gradients are reduced and weights are gathered.

what rule the 100B run from
any transfer bytes ÷ bandwidth 94 GB at 450 GB/s = 209 ms Part 3
all-reduce, per GPU 2(d − 1)/d × M, about 2M 100B BF16 gradients: 400 GB, 8 s at 50 GB/s Part 2
reduce-scatter or all-gather (d − 1)/d × M, about M half an all-reduce Part 2
ZeRO-1 or ZeRO-2, per GPU reduce-scatter + all-gather, the same 2M as DDP 400 GB Part 2
one activation message 2 × s × b × h bytes 167.8 MB Part 3
TP traffic per micro-batch 4L collectives × 2(t − 1)/t × one message 94.0 GB at t = 8 Part 3
borrowed weights under ZeRO-3 the GPU’s weight slice, borrowed twice per micro-batch t = 1, one micro-batch: 2 × 200 GB, plus 200 GB of gradients = 600 GB; candidate A: 8 × 25 GB ≈ 198 GB, about 4 s at 50 GB/s Part 2, Part 6

Part 2 counted the last row for one micro-batch with no tensor parallelism: the weights borrowed twice and the gradients added up once, three passes of about 200 GB. Part 6 counted it for candidate A. Each GPU’s slice of the weights is 2P ÷ 8 = 25 GB, borrowed before the forward and again before the backward of each of its 4 micro-batches: 8 passes, about 198 GB and 4 s at 50 GB/s against a 6.2 s step, with the gradients on top. Whether that hides behind the arithmetic is what Part 7 measures; candidate D keeps its 25 GB slice and borrows nothing.

What about pipelines, long context and experts?

what rule the 100B run from
bubble, extra over ideal (p − 1) ÷ m 75% at p = m = 4; 18.75% at PP 4 with m = 16 Part 4
bubble, share of the wall clock (p − 1) ÷ (m + p − 1) 42.9% at p = m = 4 Part 4
interleaved, v chunks per GPU (p − 1) ÷ (v × m) 37.5% at p = m = 4, v = 2 (27.3% of the wall clock) Part 4
live micro-batches on stage 0 all forward first: m; 1F1B: p 8 against 4 at m = 8, p = 4: 57 GB against 28.5 GB Part 4
what crosses a stage boundary one activation forward, its gradient back 2 × 167.8 MB = 335.5 MB per micro-batch, against TP’s 94 GB Part 4
causal work, contiguous split GPU r does 2r + 1 parts 1 : 3 : 5 : 7 over four GPUs; zig-zag makes it 4 : 4 : 4 : 4 Part 5
groups left under context parallelism 1,024 ÷ (tensor × context) TP 8 × CP 16: 8 groups, each running 4 sequences of 128K Part 5
expert utilisation mean load ÷ max load 40% for loads 5, 1, 1, 1 Part 5

What about experiments and recovery?

what rule example from
exposed communication only the part the step waits for a 300 ms transfer beside 800 ms of compute: 0; started at 650 ms: 150 ms; after the compute: 300 ms Part 7
speedup old time ÷ new time, at the same tokens per step 7.05 ÷ 6.57 ≈ 1.07 Part 7
scaling efficiency speedup ÷ how many times more GPUs 1.8 ÷ 2 = 90% Part 7
strong scaling with a fixed cost C T(k) = T ÷ k + C 1,600 + 250 ms: 5.3× at 16 workers; even perfect overlap stops at 1,600 ÷ 250 = 6.4× Part 7
failures over the run scale a measured rate by the GPU count Llama 3: one every 3.1 h on 16,384 GPUs, about every 2 days on 1,024; plan one a day, about 36 in 36 days Part 8
a checkpoint 14 bytes × P: everything but the gradients 1.4 TB; 1.37 GB from each of 1,024 GPUs Part 8
checkpoint interval τ ≈ √(2cM), with c the save pause and M the mean time between failures c = 2 min, M = 1 day: 76 min, 5.3% of the run, about 1.9 days of 36 Part 8
work lost per failure about τ ÷ 2, plus the restart 38 min at τ = 76 min Part 8
no spare node groups = GPUs left ÷ tensor degree 1,016 ÷ 4 = 254 groups; 512 ÷ 254 ≈ 2.02 sequences each, so the batch no longer divides Part 8

Can you do these in your head?

Each one takes under a minute with the rules above. Say the answer out loud before opening it.

1. A teammate says a 70B model trains with plain DDP on 80 GB GPUs. Does it fit?

No. Training state is 16 bytes × 70 × 10⁹ = 1.12 TB on every GPU under DDP, fourteen times the lid. ZeRO-3 over 64 GPUs brings it to 1.12 TB ÷ 64 = 17.5 GB, before a single activation.

2. A 7B model, 15 trillion tokens, 256 H100s at 40% MFU. How long?

6 × 7 × 10⁹ × 15 × 10¹² = 6.3 × 10²³ FLOPs. The cluster does 256 × 989 × 10¹² × 0.4 ≈ 1.0 × 10¹⁷ FLOP/s, so it takes 6.2 × 10⁶ s, and dividing by 86,400 gives about 72 days.

3. The 8K model moves to 32K context. Activations per GPU at TP 8 with sequence parallelism?

Four times the tokens, so 28.5 GB × 4 = 114 GB: over the 80 GB lid on activations alone. Splitting the sequence over two GPUs (context parallelism, CP 2) brings it to 57 GB, before state and workspace.

4. PP 8 with TP 8 on 1,024 GPUs, the same 4,194,304-token batch, one sequence per micro-batch. What does the bubble cost?

Each group is 8 × 8 = 64 GPUs, so 1,024 ÷ 64 = 16 groups remain, and each runs 512 ÷ 16 = 32 micro-batches. Extra time is (8 − 1) ÷ 32 = 22%; the idle share of the wall clock is 7 ÷ 39 = 18%.

5. A 25 GB gradient all-reduce across 128 GPUs on 50 GB/s links. How long?

About 2 × 25 GB = 50 GB sent per GPU (exactly 2 × 127/128 × 25 = 49.6 GB), so about 1 s before latency. If it overlaps a 6 s backward pass it may cost nothing; if it runs after the backward pass, it costs the full second.

6. Saving a checkpoint stalls training for 8 minutes, and the job fails about every 6 hours (a mean time between failures of 6 hours). How often should it save, and what does that cost?

τ ≈ √(2 × 8 × 360) = √5,760 ≈ 76 min, the same interval as our run’s 2 minutes and one day, because 2cM is the same product. The cost is not: 8 ÷ 76 + 76 ÷ 720 ≈ 10.5% + 10.5% = 21%, a fifth of the run, against our 5.3%. The fix is a faster save, or one that runs in the background while training continues, not a different interval.

None of these replaces a profiler. They decide which experiment is worth the profiler’s time.