Why would we cut the model by layers?

Part 3 left us with a plan that fits. Tensor parallelism (TP) cuts every layer’s matrices across the eight GPUs of a node, written TP 8 for short, and sequence parallelism (SP) splits the rest of each layer’s activations by tokens across the same eight. The model state is shared out across the 128 nodes, and each GPU has about 50 GB to spare.

Two things could break it. A longer context grows the activations: at 32K tokens they no longer fit. A bigger model needs more GPUs per layer, and tensor parallelism cannot grow past one node, because its four additions per layerfour additions per layerattention ends with one: its output projection is cut by rows, so the GPUs add up their partial outputsthe MLP ends with one: its down projection is cut by rows toogoing forward: 1 + 1 = 2; coming back, the gradients pass through the same cuts: 2 more2 + 2 = 4 per layer; × 80 layers = 320 per micro-batchAt t = 8 each GPU sends about 94 GB for them per micro-batch (Part 3). would then cross the slow network. This part takes on the second threat. Part 5 takes on the first.

What we want is a split that asks little of the links between nodes. The obvious one: do not cut the layers at all, cut the stack of them. GPU 0 keeps layers 1 to 20, GPU 1 layers 21 to 40, and so on. Each GPU stores a quarter of the weights and never moves them. (In our cluster each of these would be a whole node, still running tensor parallelism inside; the figures draw one GPU each.)

Between neighbours only one thing crosses: a micro-batchMicro-batchThe part of the batch one GPU pushes through a single forward and backward pass. A training step’s batch is too big to run at once, so each GPU runs several micro-batches one after another and adds up their gradients before the one update (gradient accumulation). In our 100B run a micro-batch is one 8,192-token sequence.’s activations going forward, and their gradient coming back. For one 8K sequence that is about 168 MB each waywhat crosses a stage boundaryone micro-batch’s activations: 2 bytes × 8,192 tokens × 10,240 hidden units = 167.8 MBforward, then its gradient back: 2 × 167.8 = 335.5 MB per boundary per micro-batchtensor parallelism sends about 94 GB per micro-batch (Part 3): about 280 times moreA pipeline sends one activation per boundary; tensor parallelism sends four additions per layer., once per boundary, instead of tens of gigabytes of additions inside every layer. This way of splitting is pipeline parallelism (PP), and each group of layers is a stage. The number of stages goes after the name, as with TP 8: our four GPUs here are PP 4, and PP 1 means the layers are not cut at all, no pipeline.

It sounds simpler until you ask what GPU 3 does while GPU 0 runs the first micro-batch. Nothing. Step through it:

Why does a pipeline sit idle?

With one micro-batch, only one GPU works at a time: each is busy a quarter of the step. The cure is to send more work in behind the first. A training step already runs several micro-batches one after another before its single update (Part 2’s gradient accumulation), so let them follow each other down the line: while GPU 1 runs micro-batch 1, GPU 0 starts micro-batch 2.

The line fills, runs with every GPU busy, then drains. The empty slots at the start and the end never go away: every GPU still waits for its first micro-batch to arrive and for the last backward to leave. That waiting is the pipeline bubble.

The bubble costs every GPU the same p − 1 slots however many micro-batches there are, while the useful work grows with them. So the waiting, measured against the work, is (p − 1)/m(p − 1)/m, worked outwith p stages, every GPU waits p − 1 slots at the start and the end together, however many micro-batches there areits useful work grows with the micro-batches: m slots4 stages: 3 ÷ 4 = 75% with 4 micro-batches, 3 ÷ 8 = 37.5% with 8, 3 ÷ 16 ≈ 19% with 16Double the micro-batches and the bubble’s share halves.: more micro-batches, a smaller bubble. Profilers measure it against the whole step instead, which gives a smaller number for the same waiting.

Where (p − 1)/m comes from, and the profiler’s number

Give each micro-batch a forward that takes F and a backward that takes B on every stage. Each GPU has m(F + B) of real work. The first micro-batch needs p − 1 forwards to reach the last GPU, and the last backward needs p − 1 backwards to climb back to the first, so the step lasts about

\[T\approx(m+p-1)(F+B).\]

The extra time over a perfect line is

\[\frac{T-T_{\mathrm{ideal}}}{T_{\mathrm{ideal}}}\approx\frac{p-1}{m},\]

while a profiler reports the share of the wall clock spent waiting,

\[\frac{T-T_{\mathrm{ideal}}}{T}\approx\frac{p-1}{m+p-1}.\]

With four stages and four micro-batches these are 75% and 42.9%the same 3 slots, two denominatorsfill and drain cost p − 1 = 3 slots per stageextra over ideal: 3 ÷ 4 micro-batches = 75%share of the wall clock: 3 ÷ (4 + 3) = 42.9%Profilers report the second; scheduling papers often quote the first.: the same waiting, two denominators. Check which one a paper or a dashboard is quoting.

What does a full pipeline cost in memory?

Every forward leaves something behind. A GPU keeps a micro-batch’s activations from its forward until that micro-batch’s backward comes back, because the backward needs them. With eight micro-batches in the order we just used, every forward first, GPU 0 runs all eight forwards before the first backward returns, so at its peak it holds all eight.

That is a lot of memory. With TP 8 and SP, one micro-batch’s activations for 20 of the 80 layers are 7.13 GBone micro-batch on one stageone micro-batch through all 80 layers, with TP 8 and SP: 28.5 GB (Part 3)one stage holds 20 of the 80 layers: 28.5 GB ÷ 4 = 7.13 GB, so eight of them are 57 GBeight micro-batches held at once8 × 7.13 GB = 57.0 GB on GPU 0 at its peakOver two thirds of an 80 GB GPU, before the weights and the borrowed layers., most of an 80 GB GPU before a single weight is counted.

A different order fixes it, with one rule: whenever a backward is ready, run it before starting another forward. Each backward frees one micro-batch’s activations, so the memory stops growing. The order is called one forward, one backward (1F1B), and on our four GPUs with eight micro-batches it runs in three stretches.

Warm-up. The first backward cannot start until micro-batch 1 has gone forward through all four GPUs and the loss is computed. Rather than sit idle while that happens, GPU 0 keeps sending micro-batches in: F1, F2, F3, F4. Those four are in flight when B1 comes back.

Steady state. From then on GPU 0 alternates: B1, F5, B2, F6, B3, F7, B4, F8. Every backward frees one micro-batch and every forward adds one, so the count stays at four.

Drain. With no forwards left, GPU 0 finishes the backwards: B5, B6, B7, B8.

Count what GPU 0 holds. Every forward first climbs 1, 2, 3 and on to 8 micro-batches; 1F1B climbs 1, 2, 3, 4 and stays at 4. The other GPUs need even less: GPU k holds at most p − k, so the last GPU, which runs each backward right after its forward, holds just one.

Both orders do the same work, and in both each GPU still waits while the line fills at the start and drains at the end; 1F1B only moves each backward earlier. So both finish at the same time: 1F1B buys memory, not speed. The memory is what matters, because it lets a step afford more micro-batches, and more micro-batches shrink the bubble.

Look at GPU 0’s peak once more, because it hides a surprise. It holds 4 micro-batches, each only through its own 20 layers. Four times 20 is 80 layers: the same as one micro-batch through the whole model, 28.5 GBfour micro-batches of 20 layersunder 1F1B, GPU 0 holds 4 micro-batches × 7.13 GB = 28.5 GBthat is one micro-batch through all 80 layers: p micro-batches × L/p layers = L layersA deeper pipeline does not shrink the first stage’s peak..

Now make the pipeline deeper. With 8 GPUs each one keeps only 10 layers, but GPU 0 now sends 8 micro-batches in before the first backward comes back. Eight times 10 is still 80. A deeper pipeline makes each GPU’s share of the layers thinner and its queue longer by the same factor, so GPU 0’s peak never moves: it is always one micro-batch through the whole model.

That is why a pipeline cannot answer the first threat, a longer context. At 32K tokens one micro-batch through the model is 114 GBone 32K micro-batch through 80 layersat 8,192 tokens, with TP 8 and SP: 28.5 GB32K is 4 × 8,192 tokens, and activations grow with the tokens4 × 28.5 GB = 114 GBMore than a whole 80 GB GPU, before any weights., more than a whole GPU, and GPU 0 must hold all of it however deep the pipeline. Part 5 cuts the sequence itself.

Can the waiting be filled?

Two later ideas squeeze the bubble. The first gives each GPU two smaller pieces of the model instead of one: GPU 0 keeps layers 1 to 10 and 41 to 50, GPU 1 layers 11 to 20 and 51 to 60, and so on. Each piece is quicker, so the line fills sooner, but micro-batches now cross between GPUs twice as often (interleaving).

The second splits each backward in two. The part the previous GPU is waiting for, the gradient it needs to carry on, runs at once. The part nobody waits for, the gradient for this GPU’s own weights, can run later and fill an empty slot (zero bubble).

Neither makes a pipeline free. The waits at the very start stay, uneven stages and slow transfers still cost time, and the update at the end of the step still waits for every backward. When you compare schedules, compare traces, not names.

Should our 100B run use a pipeline?

Not yet. Part 3’s plan already fits with about 50 GB to spare, and a pipeline would only add waiting. To see how much, count GPUs.

Today one node holds a whole copy of the 80 layers, cut across its 8 GPUs: TP 8, PP 1. Call the GPUs that together hold one copy of the layers a group. Our 1,024 GPUs make 128 groups, and each group trains on its own sequences, sharing gradients and state with the others as in Part 2. The batch stays at 4,194,304 tokens, 512 sequencesthe batch, in sequences4,194,304 tokens ÷ 8,192 tokens per sequence = 512, per step, so each group runs 512 ÷ 128 = 4 micro-batches.

Now try PP 2: layers 1 to 40 on one node, 41 to 80 on the next. One group now spans two nodes, 16 GPUs, so only 64 groups remain. The batch has not changed, so each group must run 8 micro-batches instead of 4. Each GPU does the same arithmetic as before, twice the micro-batches through half the layers. What the pipeline adds is the waiting, (p − 1)/m: 12.5% extra timePP 2 at a fixed batchgroups: 1,024 GPUs ÷ (TP 8 × PP 2) = 64512 sequences ÷ 64 groups = 8 micro-batches eachwaiting ÷ work: (p − 1)/m = (2 − 1) ÷ 8 = 12.5%A deeper pipeline means fewer groups, so each pipeline gets more micro-batches..

PP 4 cuts the layers into four stages of 20: a group is 32 GPUs, 32 groups remain, and each runs 16 micro-batches: 18.75%PP 4 at a fixed batch1,024 ÷ (8 × 4) = 32 groups512 ÷ 32 = 16 micro-batches each(4 − 1) ÷ 16 = 18.75%Keep 4 micro-batches instead and the batch falls to 32 × 4 × 8,192 = 1,048,576 tokens.. The deeper pipeline does get more micro-batches, but its fill and drain grow faster. Keeping 4 micro-batches per group instead would cut the batch to a quarter, a different experiment.

So PP 1 stays our baseline. When does a pipeline earn its place?

A pipeline helps whenthe model outgrows what tensor parallelism inside one node can hold, the links between nodes are too slow for the traffic of the other splits, the model is deep enough to cut into even stages, and each step has enough micro-batches to keep the line full.
Be careful whenthe batch is small, the stages are uneven, the transfers between stages are not hidden behind compute, or the model already fits and runs fast without one. A pipeline adds waiting you must earn back.

Measure useful tokens per second before believing a deeper pipeline helps. That leaves the first threat from Part 3 standing: longer context. A 32K sequence needs 114 GB of activations per GPU, and neither sharing the state, nor splitting the layers, nor the pipeline fixes it. Part 5 splits the sequence itself, then looks at the other way models grow: experts.

Scheduling and composition background: Efficient Large-Scale Language Model Training on GPU Clusters. The interactive schedules here are generated from dependencies, not drawn as arbitrary colored boxes.