How do all the splits fit together?

Part 5 ended with every split the run might need. Now they have to share one machine. Each one divides something different and sends a different kind of traffic, so the real question is which split gets which link.

split what it divides what crosses the wire where it belongs
sharing the state (ZeRO-3, Part 2) weights, gradients, optimizer states borrowed weights, gradients added up across nodes: it can fetch a layer ahead
tensor parallelism (TP, Part 3) every matrix multiply partial outputs added up, four times a layer inside a node, on the fast links
sequence parallelism (SP, Part 3) the norms and dropout between multiplies the same bytes as TP, in two halves with TP
pipeline (PP, Part 4) the layers one activation per stage boundary across nodes, once the model outgrows one
context parallelism (CP, Part 5) the tokens of one sequence keys and values round a ring when one sequence outgrows a GPU
experts (EP, Part 5) the experts of an MoE layer tokens to their expert and back only in an MoE

Whatever we pick has to account for every GPU. Multiply the splits together and you must get the machine: tensor × pipeline × context × groups = 1,024, where the groups are the copies of the whole arrangement that each take different examples.

Our candidate for the 8K run: TP 8 with SP inside each node, no pipeline, no context split, and the state shared over the groups: 8 × 1 × 1 × 128 = 1,024the candidate’s counttensor 8 × pipeline 1 × context 1 = 8 GPUs per group1,024 GPUs ÷ 8 per group = 128 groupsEvery layout must multiply back to the GPU count.. The figure places it on the machine.

So the busiest traffic, the additions inside every layer, stays on the fast links, and the traffic that can wait, the borrowed weights and the gradients, crosses the network. That is the whole idea of a layout: put each kind of traffic on the link it can afford.

Does the candidate fit in one GPU?

Part 3 screened two things, the state and the activations. A real step needs a little more. Each GPU holds its 1.56 GB of state and the activations of one micro-batchMicro-batchThe part of the batch one GPU pushes through a single forward and backward pass. A training step’s batch is too big to run at once, so each GPU runs several micro-batches one after another and adds up their gradients before the one update (gradient accumulation). In our 100B run a micro-batch is one 8,192-token sequence., 28.5 GB for one 8K sequence with TP 8 and SP.

It also holds the layer it has just borrowed, only 0.3125 GBone gathered layerone layer’s BF16 weights: 2 bytes × 100 × 10⁹ ÷ 80 layers = 2.5 GBwith TP 8 each GPU gathers only its eighth: 2.5 GB ÷ 8 = 0.3125 GB because tensor parallelism already cut every layer eight ways, and some room for the framework’s own workspace, which we set at 4 GB. Together that is about 34.4 GBthe budgetstate 1.5625 + activations 28.52 + workspace 4 + one gathered layer 0.3125= 34.4 GB, leaving 45.6 GB of the 80A coarse screen: profile a warm step before trusting the headroom., about 45.6 GB free.

That headroom is a screen, not a measurement. Borrowing the next layer early, a larger embedding, the optimizer’s temporary buffers and the memory allocator can all raise the real peak. Only a profiled, warmed-up step settles it.

Does the candidate run the same experiment?

Fitting is not enough. A faster layout that quietly changes the batch is a different experiment, and its speed proves nothing. Our batch is fixed at 4,194,304 tokens per step: 128 groups, each running 4 micro-batches of one 8K sequence.

\[\underbrace{128}_{\text{groups}}\times\underbrace{1}_{\text{sequence}}\times\underbrace{4}_{\text{micro-batches}}\times\underbrace{8192}_{\text{tokens}}=4{,}194{,}304\;\text{tokens per step}.\]

Change one factor and another must move. Add a four-stage pipeline and keep 4 micro-batches: each group now needs 32 GPUs, only 321,024 ÷ (8 × 4)groups = GPUs ÷ (tensor × pipeline) = 1,024 ÷ 32 = 32with the same accumulation: 32 × 1 × 4 × 8,192 = 1,048,576 tokens, a quarter of 4,194,304 groups remain, and the batch falls to a quarter. Part 4 met this already. The fix is 16 micro-batches per group, which brings the batch back, and with it the pipeline’s waiting.

Try it in the lab: pick a pipeline of 4 and watch the batch turn red, then pick 16 micro-batches. Or pick tensor 16: memory drops, but the additions leave the node, and the batch shrinks too.

Which candidates are worth a benchmark?

The screen says the candidate fits. It does not say it is the fastest. Three other layouts fit as well, and each might win for a reason the screen cannot weigh, so each is worth a short run.

A: the baselineTP 8, no pipeline, everything shared (ZeRO-3), 4 micro-batches: the reference. Its risk is the borrowing. Each GPU borrows its 25 GB slice of the weights twice per micro-batch, about 200 GBA’s borrowed weights, per GPU per stepone GPU’s slice of the weights: 2 bytes × 100 × 10⁹ ÷ 8 = 25 GBborrowed before the forward and again before the backward pass, for each of 4 micro-batches: 8 passeseach pass brings 127 of the 128 pieces: 8 × 25 GB × 127/128 ≈ 198 GB198 GB ÷ 50 GB/s ≈ 4 s against a 6.2 s step, so it must hide behind the arithmetic. per step over the network, unless it hides behind the arithmetic.
B: tensor 4Each GPU’s multiplies are twice as wide, which keeps it busier, and its additions shrink to 179 msB’s additions and arithmetic4 additions per layer × 80 layers = 320 per micro-batch; at t = 4 each sends 2 × 3/4 = 1.5 messagesone message is 2 bytes × 8,192 × 10,240 = 167.8 MB: 320 × 1.5 × 167.8 MB = 80.5 GB80.5 GB ÷ 450 GB/s = 179 msarithmetic: 6 × 10¹¹ × 8,192 ÷ (4 × 989 × 10¹²) = 1.24 sA at t = 8: 209 ms against 621 ms. against 1.24 s of arithmetic per micro-batch. The price is twice the activations: 63.2 GBB’s memory per GPUstate: still 1.56 GB, 16 bytes per parameter shared over 1,024 GPUsactivations: 228 GB ÷ 4 = 57.0 GBa borrowed layer: 2.5 GB ÷ 4 = 0.63 GB, and the 4 GB workspace1.56 + 57.0 + 0.63 + 4 = 63.2 GB per GPU.
C: a short pipelineTP 8 and four stages, with 16 micro-batches to keep the batch. It buys nothing at 8K and waits 18.75% longer, but it measures the waiting Part 4 predicted: the number we will need the day the model outgrows a node.
D: whole weightsTP 8 with only the gradients and optimizer states shared (ZeRO-2). Each GPU keeps its 25 GB slice of the weights, so nothing is borrowed; one pass of new weights crosses the network after the update. The price is room: 58.9 GBD’s memory per GPUweights kept whole: 2 bytes × 100 × 10⁹ ÷ 8 = 25 GBgradients and optimizer states shared over the 128 groups: 14 bytes × 100 × 10⁹ ÷ 1,024 = 1.37 GBactivations 28.5 GB and the 4 GB workspace; nothing borrowed25 + 1.37 + 28.5 + 4 = 58.9 GB. Keeping the gradients whole too (ZeRO-1) would reach 83.7 GBwhole gradients too (ZeRO-1)gradients: 2 bytes × 100 × 10⁹ ÷ 8 = 25 GB, instead of 0.2 GB58.9 GB + 24.8 GB = 83.7 GB, over the 80 GB and stop fitting.

Do not change everything at once. Benchmark A first, find the biggest exposed delay in its trace, then pick the candidate that targets it: B if the tensor additions show, D if the borrowing does.

What the lab calculates, and what it leaves out

For 1,024 GPUs, tensor degree t and pipeline degree p, there are d = 1,024 ÷ (t × p) groups. Each GPU owns P ÷ (t × p) parameters, and the state rule then divides some of their 16 bytes by d, as in Part 2.

Activations are one micro-batch’s 228.17 GB ÷ t, assuming sequence parallelism and no recomputation. Under a pipeline the first stage still holds that much (Part 4), so the lab does not divide it by p. The borrowed layer is 2P ÷ (L × t) bytes under full sharding.

The tensor additions use Part 3’s count for L ÷ p layers per stage, on the fast links up to t = 8 and on the network beyond. A real collective across two nodes mixes both links, so this is a coarse penalty, not a network simulator.

Waiting is (p − 1) ÷ m, from Part 4. Weights sent counts a slice borrowed twice per micro-batch under full sharding, or one pass of new weights after the update under ZeRO-1 and 2; gradient traffic is left out. No estimate here replaces measuring the exposed communication in a trace.

What would convince us?

A candidate wins only if the comparison is fair and the measurement is real. Four things make it so, and the next two parts make each one concrete.

The same experiment. Use the same data, loss and its normalization, batch of tokens, optimizer, precision and schedule. Check a few updates against a small reference and compare short loss curves. Floating-point sums may differ in the last bits, but a loss that drifts is a bug.

A warm, steady step. The first steps compile code, set up the optimizer and load data, so leave them out. Record useful tokens per second, how much step times vary, and the peak memory actually allocated. A forward pass that ran once proves nothing about memory.

The critical path. Long communication is harmless if it runs while the GPU computes; short communication that stalls every layer is not. Look in the trace for gaps where a GPU waits for a borrowed layer, late gradient additions, idle pipeline slots, slow GPUs and a starved data loader.

A restart. Save and resume the model, the optimizer, the schedule, the random state and the position in the data, then check that the loss carries on where it stopped. A month-long run is as much about surviving failures as about speed, which is Part 8.

What is our answer, and what comes next?

The layout we can defend is the one whose measured memory fits, whose loss matches the experiment we meant to run, and whose useful tokens per second survive the communication and the restarts. Today that is still a hypothesis: A, TP 8 with SP and the state shared over 128 groups, earns the first benchmark, not a launch.

Part 7 runs that first comparison properly: one prediction, A against B, everything else held fixed, and the measurement that would prove the prediction wrong.

A larger experimental reference: the Ultrascale Playbook. Parallelism composition background: Megatron-LM’s large-scale training study. All numerical results in our interactive lab are analytical estimates; no cluster benchmarks were run for this chapter.