The cluster is yours. What would you change?
Part 8 finished the plan. One GPU cannot hold a training step, so the state is shared across the groups (Part 2). Each layer is split across the GPUs of a node, and its in-between work by tokens (Part 3). A pipeline waits in reserve for a bigger model (Part 4), and the sequence itself is split when one outgrows a GPU (Part 5). The layout is screened (Part 6), then measured (Part 7), and kept alive with checkpoints and spare nodes (Part 8).
Now the cluster is yours, and so are the questions an interviewer or a reviewer would ask about it. A strong answer names three things: the constraint, the mechanism that changes it, and the measurement that could prove the prediction wrong. The lab is Part 6’s: use it to check a layout before you answer.
Three levels follow. The first asks you to state a mechanism precisely, the second to choose the next experiment, the third to defend a claim against a sceptical reviewer. Every number in the answers is one the series has worked out; if one surprises you, the part it came from is the one to reread.
Level 1: can you state the mechanism?
-
A 100B model is 200 GB in BF16. Why does training it ask for 1.6 TB before a single activation is stored?
answer sketch
Training keeps 16 bytes per parameter, not 2: the BF16 weight (2), its BF16 gradient (2), an FP32 master copy (4) and Adam’s two FP32 moments (4 + 4). 16 × 100 × 10⁹ is 1.6 TB, eight times the weights. The activations come on top, 228 GB for one 8K sequence, and they grow with tokens, width and depth rather than with the parameter count, so the two bills have to be checked separately.
-
ZeRO-2 runs out of memory on our 100B model. A teammate proposes doubling the number of groups. Does that fix it?
answer sketch
No. ZeRO-2 shares the gradients and optimizer states, 14 of the 16 bytes, but every GPU still keeps all the BF16 weights: 200 GB, whatever the number of groups. More groups only shrink the shared part, which was already small. The fix has to divide the weights too: share them as well (ZeRO-3, which borrows each layer while it runs) or split every layer across GPUs (tensor parallelism). First check which allocation actually failed: the weights, a borrowed layer, or the activations.
-
After the backward pass under ZeRO-2, each GPU needs the summed gradient only for the slice it will update, and after the update every GPU needs all the new weights. Which two collectives, and how do their bytes compare with DDP’s?
answer sketch
Gradients are versions of the same numbers, and each GPU needs only its slice: add them up and scatter, a reduce-scatter. The new weights are pieces of one tensor that everyone needs: collect them, an all-gather. Each sends about (d − 1)/d of the model, so together they send what DDP’s all-reduce sends, about 400 GB per GPU for the 100B model, because an all-reduce is exactly a reduce-scatter followed by an all-gather. ZeRO-2 just puts the update in the middle.
-
A ZeRO-3 GPU stores only a 1.56 GB slice of the state. Why can its forward pass still run out of memory?
answer sketch
Because it cannot compute with a slice. Just before each layer runs, it borrows that layer’s missing slices, 2.5 GB of weights per layer without tensor parallelism, and frees them right after. Borrowing the next layer early to hide the transfer, keeping layers after the forward pass, and above all the activations (228 GB for one 8K sequence without tensor parallelism) all sit on top of the 1.56 GB. The persistent number is not the peak; profile the peak around each borrow.
-
Why are the two matrices of an MLP split by columns first and by rows second, and not the other way round?
answer sketch
A column cut gives each GPU complete hidden features, so the activation can run locally, and it leaves those features split across the GPUs exactly as a row cut of the next matrix wants its input. The row cut then gives each GPU a partial output, and one addition per MLP finishes it. Two column cuts would force the GPUs to collect the features between the multiplies; two row cuts would force an addition before the activation, because ReLU(−3 + 2) is 0 while ReLU(−3) + ReLU(2) is 2.
-
Tensor parallelism at t = 8 cuts every multiply eight ways. Why does each GPU still hold about 87 GB of activations, and what fixes it?
answer sketch
In Megatron’s count a layer stores about 34 bytes per token per hidden unit: 24 inside the multiplies, which TP divides by t, and 10 in the layer norms, dropout and residual additions, which every GPU still runs on the whole sequence. That leaves 10 + 24 ÷ 8 = 13 bytes, 87 GB. Those in-between steps treat each token on its own, so sequence parallelism splits them by tokens: the additions become a reduce-scatter and an all-gather, the same bytes as before, and the activations fall to 28.5 GB.
-
Four pipeline stages, four micro-batches. One dashboard says the bubble is 75%, another says 42.9%. Which one is wrong?
answer sketch
Neither. Every stage waits 3 slots at the start and the end together. Measured against the useful work, 3 ÷ 4 is 75%, how much longer the step is than a perfect line; measured against the whole step, 3 ÷ 7 is 42.9%, the share of the wall clock a GPU sits idle. Profilers report the second, scheduling papers often quote the first. Always say which denominator a number uses.
-
Under context parallelism, why do the GPUs pass keys and values round a ring instead of the activations, and how does the result stay exact?
answer sketch
A token’s query only needs the earlier tokens’ keys and values, not their activations, so each GPU keeps its queries and passes its block of keys and values to the next GPU; after c − 1 passes every block has been seen. The softmax over a row is built block by block with a running max and a running sum: when a bigger score arrives, the old sum is rescaled instead of recomputed (online softmax). It is an exact rearrangement, not an approximation.
Level 2: can you choose the next experiment?
-
TP 16 leaves more memory free than TP 8 but trains slower. What is your first hypothesis, and what would you look at?
answer sketch
TP 16 spans two nodes, so the four additions in every layer leave the fast links and cross the network. The bytes barely change, 94.0 GB to 100.7 GB per micro-batch, but at 50 GB/s instead of 450 GB/s the time goes from 209 ms to 2.01 s, over six times the 311 ms of arithmetic left on each GPU. Narrower multiplies also keep a GPU less busy. Look at a trace at the same tokens per step: how much of the additions’ time is exposed, and how fast the multiplies run.
-
We add a four-stage pipeline to the 8K run and keep 4 micro-batches per group. What changed besides the speed?
answer sketch
The experiment. A group now needs 8 × 4 = 32 GPUs, so only 32 groups remain, and 32 × 1 × 4 × 8,192 is 1,048,576 tokens, a quarter of the batch. Keeping 4,194,304 tokens means 16 micro-batches per group, which costs (4 − 1) ÷ 16 = 18.75% extra time in waiting. And at 8K it buys nothing, because the plan without a pipeline already fits with room to spare.
-
Can switching from every-forward-first to one-forward-one-backward (1F1B) make a pipelined step faster?
answer sketch
Not by itself. With the same stages and micro-batches both orders wait the same slots and finish at the same time. What 1F1B changes is memory: GPU 0 never holds more than p micro-batches instead of m, 4 instead of 8 in Part 4’s example, 28.5 GB instead of 57 GB. That room can buy more micro-batches, and more micro-batches shrink the bubble. Compare the two orders at a fixed configuration, then separately compare the configuration the freed memory allows.
-
The context grows to 128K. A teammate suggests a deeper pipeline. Why will it not help, and what will?
answer sketch
Activations grow with the tokens, 456 GB per GPU at 128K, and a pipeline does not touch that: under 1F1B the first stage holds one micro-batch’s activations through the whole model however many stages there are. Tensor parallelism is capped at a node, and the micro-batch is already one sequence. Split the sequence itself: context parallelism over 16 GPUs gives each a block of 8,192 tokens, back to 28.5 GB, with the zig-zag order to balance the causal work. The cost is GPUs taken from the groups: TP 8 × CP 16 leaves 8 groups, each running 4 sequences.
-
Should the 128K branch add expert parallelism?
answer sketch
No. Our model is dense, so there are no experts to spread. Long context strains the activations and attention’s work, which is what context parallelism and recomputation address. Expert parallelism arrives with a mixture-of-experts architecture, which is a different model with its own compute, memory and quality budgets, and its own constraint: tokens travelling to their expert and back, and the hot expert that left four GPUs only 40% busy.
-
Candidate A’s trace shows long gaps where GPUs wait for borrowed layers. Which candidate do you try next, and why?
answer sketch
D. Under A each GPU borrows its 25 GB weight slice before the forward and again before the backward of each of its 4 micro-batches, about 198 GB a step over the network, around 4 s at 50 GB/s against a 6.2 s step. D shares only the gradients and optimizer states (ZeRO-2) and keeps the 25 GB slice, so nothing is borrowed and one pass of new weights goes out after the update. It still fits at 58.9 GB. If the trace had shown exposed tensor additions instead, B would be the next run.
-
We predicted B to be about 7% faster than A. What result would prove that wrong, and what would each outcome teach us?
answer sketch
Wrong if B is not faster at the same 4,194,304 tokens a step, or its peak memory does not fit. A tie at about 6.21 s means A’s 0.84 s of additions were already hidden behind the arithmetic. A gain well beyond 0.48 s means B’s wider multiplies run faster per FLOP, which the screen cannot see. An out-of-memory error means the 63.2 GB screen missed something real, such as a buffer or the allocator.
-
A first test on 16 GPUs runs 5.3 times faster than on one. Is something broken?
answer sketch
Not necessarily. Communication that does not shrink with the number of GPUs puts a floor under the step: with 1,600 ms of compute and 250 ms of talking, 16 workers take 100 + 250 = 350 ms against 1,850 ms, 5.3 times as fast, 33% efficient. Even perfect overlap stops at 1,600 ÷ 250 = 6.4 times. Scale in stages and find what does not shrink before trusting a small result at full size.
Level 3: can you defend the claim?
-
A new layout has a 20% shorter step. What must you know before calling it faster?
answer sketch
That it trains the same experiment: the same tokens per step, data and order, loss normalization, optimizer, precision and schedule, with loss curves from the same start that track each other. A step can look shorter by carrying fewer tokens, so compare useful tokens per second, not step time. Measure warm, steady steps with the first compile and warm-up steps left out, report the spread over a stated window, and check that the peak memory still fits.
-
A trace shows 500 ms of communication in a step. Can you promise 500 ms of savings by removing it?
answer sketch
Only the part on the critical path, the longest chain of things that each wait for the one before. Communication that runs while the GPU computes costs nothing: a 300 ms transfer alongside 800 ms of compute leaves the step at 800 ms, while the same transfer started after the compute makes it 1,100 ms. Removing one transfer can also expose the next wait. Find what each transfer overlaps, change one thing, and measure the new critical path.
-
Why benchmark A first, when B is the one predicted to be faster?
answer sketch
A is the reference the screen trusts most: tensor additions inside a node, the most memory to spare (34.4 GB of 80), the batch exactly as planned. Its trace tells us which delay is real, the tensor additions or the borrowed weights, and each points to a different next run: B for the additions, D for the borrowing. Running B first would test one guess; running A first tells us which guess to test.
-
B’s loss curve does not match A’s to the last digit. Is that a bug?
answer sketch
Not by itself. Adding the same numbers in a different order rounds differently, and B splits its sums differently from A, so the last digits drift. The curves should track each other closely from the same starting weights. A loss that drifts apart is a bug, usually a different batch, a loss averaged over different tokens, or a different data order, and a faster wrong answer is not a speedup. Check a few updates against a small reference first.
-
The checkpoint loaded after a failure. Have you proved that recovery works?
answer sketch
No. A checkpoint counts only if every GPU’s slice is from the same step and the final marker was written. Then check the parts that fail quietly: the step number, which sets the learning rate, the data position of every group, and the random state. Compare a resumed short run with one that was never interrupted, and measure the restart, the load and the redone steps. A file that loads is one step of that experiment.
-
Saves pause the run for 2 minutes and the job fails about once a day. A teammate wants to save every hour instead of every 76 minutes. Do you object?
answer sketch
Not much. Young’s rule puts the cheapest interval at √(2 × 2 × 1,440) ≈ 76 minutes, 5.3% of the run in pauses and redone work. Hourly saves cost 2 ÷ 60 + 60 ÷ 2,880 = 3.3% + 2.1% = 5.4%, almost the same, because the total is flat near its minimum. What matters more is measuring the real pause and the real failure rate, and shrinking the pause with faster storage or background saves.
-
A node dies and there is no spare. Can the run simply carry on with 1,016 GPUs?
answer sketch
It can run, but not as the same experiment for free. With B’s tensor groups of 4 there are 254 groups, and the 512 sequences of a step no longer divide evenly (about 2.02 each). Every GPU would also need a 1,016th of the state instead of a 1,024th, so the saved slices must be cut again for the new owners (resharding). Libraries can re-cut the slices, but keeping the batch, the data order and the schedule is our job. That is why the plan keeps spare nodes.
-
Layout X is 3% faster per step than B but takes twice as long to save. Which one do you run for the month?
answer sketch
Compare useful progress per hour that survives failures, not step time. With a 4-minute pause and one failure a day, the best interval becomes √(2 × 4 × 1,440) ≈ 107 minutes and the overhead 4 ÷ 107 + 107 ÷ 2,880 ≈ 3.7% + 3.7% = 7.5%, against 5.3% for B. That swallows most of the 3%, before counting slower restarts over about 36 failures. Measure the save and restart of both before choosing.
Can you write the one-page launch decision?
Here is ours, as the series built it.
The layout to benchmark first is A: tensor parallelism 8 with sequence parallelism inside each node, the state shared over 128 groups (ZeRO-3), no pipeline and no context split, each group running 4 micro-batches of one 8K sequence. That is 4,194,304 tokens a step and about 34.4 GB per GPU on the screen. Its rival is B, tensor parallelism 4 over 256 groups with 2 micro-batches, 63.2 GB, predicted up to about 7% faster if A’s additions are exposed. C and D wait for what A’s trace shows.
If B wins at 6.57 s a step, the run is about 477,000 steps, 36 days. We plan for one failure a day: a checkpoint of 1.4 TB, 1.37 GB from each GPU, about every 76 minutes, 5.3% of the run, with spare nodes so that a restart never changes the layout.
The launch still needs measurements, not more arithmetic: the peak memory of a warm step, loss curves that track a reference, useful tokens per second with a trace that shows where each layout waits, and a rehearsed failure whose loss carries on where it stopped. If a result contradicts a prediction, change the model of the system before adding another split.
That is what this series set out to build: a reason for every split, and an experiment for every reason.
Every estimate behind that plan is collected, worked out, on one page in Part 10: the rules to run in your head when there is no time for a spreadsheet.