Why do the activations still not fit?
Part 2 brought the model state down to 1.56 GB per GPU, but each GPU still builds a whole sequence’s activations, 228 GB, and an 80 GB GPU cannot hold them. We now ask a different question: can several GPUs compute one example together without any one of them building every intermediate feature?
The answer starts with a two-by-two matrix, not a 100B model. Choose a row or column cut below, then calculate what each GPU actually knows.
Why does the direction of the cut matter?
A transformer’s MLP does two multiplies with an activation in between: project up to a wide hidden layer, apply the activation (GELU or ReLU) to each hidden feature, project back down. The direction of the cut decides what each GPU ends up holding, and that decides whether the activation can run without talking to the other GPU.
Cut by columns and each GPU computes some of the hidden features completely. It can apply the activation to them on its own, because the activation works one feature at a time and those features are finished.
Cut by rows and each GPU holds a piece of every feature: partial sums. Nothing can be applied yet, because the activation of a sum is not the sum of the activations (the box below has the numbers). The pieces must be added first.
So which cut, and when? Both, in a fixed order. Cut the up projection by columns: each GPU gets complete hidden features and applies the activation locally. Those features are now split across the GPUs, which is exactly the input a row cut expects: GPU 0 holds the first hidden features, and the row cut gives it the rows of the down projection that multiply them.
Cut the down projection by rows, and each GPU ends with a partial output. Add the partials once (an all-reduce), and every GPU has the layer’s output. That is one addition per MLP and none between the two multiplies.
Why not the other two orders? Cutting both by columns would force the GPUs to collect the hidden features before the second multiply; cutting both by rows would force an addition before the activation. Column, then row, is the order that talks least.
Derive the two-matrix construction
With row-vector inputs, split the first matrix along output columns and the second along input rows:
\[A=[A_0\;A_1],\qquad B=\begin{bmatrix}B_0\\B_1\end{bmatrix}.\] \[Y=f(XA)B=f(XA_0)B_0+f(XA_1)B_1.\]Each rank computes one complete hidden-feature partition, then one partial output. The final sum can be all-reduced. In backward, the shared input receives contributions from each branch, so its gradient needs a reduction too. Forward and backward communication must both be counted.
This is the central construction in Megatron-LM’s tensor parallelism. Vocabulary-parallel embeddings and loss, grouped-query attention replication, and fused kernels add implementation-specific details beyond this two-matrix example.
Attention splits the same way. Each GPU takes some of the heads, which is a column cut of the query, key and value projections, and runs those heads completely; the output projection is cut by rows and ends with one addition. So a transformer layer adds up partial outputs twice in the forward pass, after attention and after the MLP, and twice more in the backward pass.
Cutting every layer’s matrices like this is tensor parallelism (TP). The number of GPUs that share each layer is the tensor degree, t. With t = 8, each GPU holds an eighth of every weight matrix and computes an eighth of each layer’s big intermediate activations.
Why not use all 1,024 GPUs for every multiply?
If cutting each layer across 8 GPUs divides its activations by 8, why not cut it across all 1,024? Because the talking grows while the work shrinks. Every layer ends with the GPUs adding up their partial outputs, four times per layer for every micro-batchMicro-batchThe part of the batch one GPU pushes through a single forward and backward pass. A training step’s batch is too big to run at once, so each GPU runs several micro-batches one after another and adds up their gradients before the one update (gradient accumulation). In our 100B run a micro-batch is one 8,192-token sequence., and each addition is a conversation among all t GPUs.
As t grows, each GPU’s share of the arithmetic shrinks as 1/t, but what it must send per addition does not: about two copies of the 167.8 MBone activation message2 bytes (BF16) × 8,192 tokens × 10,240 hidden units= 1.678 × 10⁸ bytes ≈ 167.8 MBEvery tensor-parallel collective moves a tensor of this shape. message, whatever t is. Across the group the traffic grows with t: every addition moves about 2(t − 1) messagesthe whole group’s traffic per additioneach of the t GPUs sends about 2 × (t − 1)/t of the message: add up, then sharet × 2(t − 1)/t = 2(t − 1) messages in allt = 8: 2 × 7 = 14; t = 1,024: 2 × 1,023 = 2,046Each GPU’s share stays near two messages; the group’s total grows with t. in all, 14 at t = 8 and 2,046 at t = 1,024, passed hop by hop around the ring, each hop paying its own delay.
Put numbers on one micro-batch at t = 1,024. Each GPU computes for about 5 msone micro-batch’s arithmetic at t = 1,0246 × 100 × 10⁹ parameters × 8,192 tokens ≈ 4.9 × 10¹⁵ FLOPs÷ (1,024 GPUs × 989 × 10¹² FLOP/s) ≈ 4.9 ms per GPU, at peakAt t = 8 the same micro-batch takes 621 ms per GPU. but must send about 107 GBwhat one GPU sends per micro-batch at t = 1,024per addition: 2 × (1,023 ÷ 1,024) × 167.8 MB ≈ 335 MB4 additions per layer × 80 layers = 320 additions320 × 335 MB ≈ 107 GB107 GB ÷ 50 GB/s ≈ 2.1 s on the networkAbout 440 times the 4.9 ms of arithmetic., over 2 seconds on the 50 GB/s network. The GPUs would spend more than 400 times longer talking than computing. And past 8 GPUs, every one of those additions leaves the node’s fast links. Very narrow slices of a matrix also keep a GPU less busy.
Our nodes have eight GPUs joined by fast links, 450 GB/s each, while the network between nodes gives each GPU about 50 GB/s. TP = 8 keeps every addition inside one node, so it is our starting hypothesis, not a law of nature. TP = 4 could win if it leaves enough memory and gives better kernels; TP = 16 could be necessary on another workload, but needs evidence to justify the extra communication.
Put bytes and arithmetic on the same clock
This fold answers one question: for one micro-batch, does a GPU spend longer computing or talking? To compare the two, count each in seconds, one step at a time.
The bytes each GPU sends.
- One message is one layer’s output for one micro-batch: 8,192 tokens × 10,240 numbers × 2 bytes = 167.8 MB.
- One addition across t GPUs makes each GPU send 2(t − 1)/t messages2(t − 1)/t, worked outadding up partial outputs across t GPUs runs in two halvesadd up, each GPU keeping one slice (reduce-scatter): it sends its piece of the t − 1 slices it does not own, (t − 1)/t of the messageshare the totals (all-gather): it sends its own total slice to the t − 1 others, (t − 1)/t againboth halves: 2(t − 1)/t; at t = 4, 2 × 3/4 = 1.5 messages; at t = 8, 2 × 7/8 = 1.75The same two halves as Part 2’s “name it” figure: close to 2 messages for large t. of it: 1.75 at t = 8, about 294 MBone addition, per GPU, at t = 82 × 7/8 = 1.75 messages1.75 × 167.8 MB = 293.6 MB.
- Each layer adds up 4 times, after attention and after the MLP, going forward and coming back, so 80 layers make 320 additions.
- 320 × 293.6 MB = 94.0 GB94 GB per micro-batch4 additions per layer × 80 layers = 320 additions320 × 293.6 MB = 94.0 GB sent by each GPUThe bytes barely change with t: 1.75 messages per addition at t = 8, 1.875 at t = 16. sent by each GPU for every micro-batch.
The same count in symbols, with L layers, b sequences per micro-batch, s tokens and width h:
\[V_{\mathrm{TP}}\approx4L\cdot2\frac{t-1}{t}\cdot(2sbh).\]Read it from the right: 2sbh is one message in bytes, 2(t − 1)/t is how many messages one addition sends, and 4L is how many additions there are.
The time to send them is bytes divided by the link’s speed: 94.0 GB ÷ 450 GB/s is 209 msbytes ÷ bandwidth94.0 GB ÷ 450 GB/s (NVLink) = 0.209 s94.0 GB ÷ 50 GB/s (network) = 1.88 sNVLink carries 9× the network’s bytes per GPU, so a group that leaves the node pays about 9×. inside one node, and 94.0 GB ÷ 50 GB/s is 1.88 s if the group spans nodes.
The arithmetic for the same micro-batch takes about 621 msthe arithmetic on the same clock6 × 10¹¹ FLOPs per token × 8,192 tokens = 4.9 × 10¹⁵ FLOPs per sequencesplit over 8 GPUs: 6.1 × 10¹⁴ each÷ 989 × 10¹² FLOP/s = 0.621 s at peakLike with like: 209 ms of traffic against 621 ms of arithmetic per micro-batch. per GPU at t = 8, from Part 1’s six FLOPs per parameter per token.
Side by side. Inside one node, 209 ms of talking against 621 ms of computing: the talk is a third of the work and can partly hide behind it. At t = 16 the arithmetic halves to 311 ms, but the group now spans two nodes and the talk takes 2.01 st = 16, across two nodesthe arithmetic halves: 621 ms ÷ 2 = 311 msthe bytes grow a little: 320 × 2 × 15/16 × 167.8 MB = 100.7 GB100.7 GB ÷ 50 GB/s = 2.01 sMore than six times the arithmetic.. That is the layout to avoid.
These are not predictions of the step time. Talking can overlap computing, real links run below their rated speed, and every message also pays a fixed delay. The count flags a dangerous layout before a cluster run tests it.
What missing piece does sequence parallelism fill in?
Tensor parallelism at t = 8 cuts every big multiply eight ways, so it should cut the activations eight ways too, from 228 GB to 28.5 GB228 GB ÷ 8tensor parallelism splits the activations inside the matmuls by tsequence parallelism splits the rest, the layer norms and dropout, by t too228 GB ÷ 8 = 28.5 GB per GPUWithout sequence parallelism it would be s × h × L × (10 + 24/t) bytes: 87 GB at t = 8.. Count again and it does not: each GPU still holds about 87 GBwhat TP 8 leaves on each GPUMegatron’s count: a layer stores about 34 bytes per token, per hidden unit; 24 of them inside the matrix multiplies, 10 outsideTP 8 divides only the 24: 10 + 24 ÷ 8 = 13 bytes13 ÷ 34 × 228 GB ≈ 87 GB, more than the 80 GB GPUThe 10 bytes that stay whole are what sequence parallelism splits., more than the 80 GB it has.
The leftover is the work between the multiplies. Every layer also runs layer norms, dropout and the residual additions, and under TP every GPU runs these on the whole sequence, because they are not matrix multiplies and nothing cut them. In Megatron’s count, 24 of the 34 bytes a layer stores per token per hidden unit sit inside the multiplies and are divided by t; the other 10 sit in these in-between steps and stay whole on every GPU.
Those in-between steps have a useful property: each treats every token on its own. A layer norm normalizes one token’s vector without looking at any other token. So they can be split a different way, by tokens: with 8 GPUs, each takes an eighth of the sequence for these steps. That is sequence parallelism (SP).
The price is a change in how the GPUs talk, not in how much. Under plain TP, the partial outputs are added up and the total goes to every GPU (an all-reduce). With SP they are added up and each GPU keeps only its own tokens (a reduce-scatter); just before the next multiply, which needs every token, the tokens are collected again (an all-gather). Part 2 showed that an all-reduce is exactly these two halves, so the bytes are the same. In between, each GPU holds only an eighth of the tokens, and the activations fall to 28.5 GB.
This is not yet context parallelism. Here the goal is to stop every GPU repeating the in-between steps on the whole sequence. Part 5 splits the attention itself across more GPUs when the context grows.
What did we learn for the 100B run?
Two parts ago, one training step of our 100B model asked a single GPU for 1.83 TBone step on one GPU1.6 TB of model state + 228 GB of activations for one 8K sequence= 1,828 GB ≈ 1.83 TB. Here is how each idea has cut that down, per GPU:
| plan | model state | activations, one 8K sequence | total | fits in 80 GB? |
|---|---|---|---|---|
| one GPU (Part 1) | 1.6 TB | 228 GB | 1.83 TB | no |
| share the state across 1,024 GPUs (Part 2) | 1.56 GB | 228 GB | 230 GB | no |
| add tensor parallelism, t = 8 | 1.56 GB | 87 GB | 89 GB | no, just over |
| add sequence parallelism | 1.56 GB | 28.5 GB | 30 GB | yes |
Each line fixed the biggest thing left. Sharing the state removed the copies, but every GPU still built a whole sequence’s activations. Tensor parallelism split the multiplies, but not the norms and dropout between them. Sequence parallelism split those by tokens.
The layout behind the last line: the 1,024 GPUs form 128where 128 comes from1,024 GPUs ÷ 8 per tensor group = 128 groups1.6 TB ÷ (8 × 128) = 1.56 GB, the same as sharding over all 1,024Every layout must multiply back to the GPU count: 8 × 128 = 1,024. groups of eight, one group per node. Inside a group, the eight GPUs split every layer (TP 8 with SP) over the fast links. Across the groups, each takes different examples and the model state is shared out, as in Part 2: 1.6 TB ÷ (8 × 128) = 1.56 GB per GPU.
That leaves about 50 GBthe room that is left80 GB − (1.56 GB of state + 28.5 GB of activations)= 80 − 30.1 ≈ 50 GBA first screen only: Part 6 counts the rest. of the 80 for everything this budget leaves out: the layer a GPU has borrowed, the buffers the additions need, the framework’s own workspace. Part 6 counts them.
It is a plausible plan, not a finished one, and two things could break it. Longer text: activations grow with the sequence, so a 32K context needs four times 28.5 GB, 114 GBa 32K contextactivations grow with the tokens: 32,768 ÷ 8,192 = 44 × 28.5 GB = 114 GB per GPU, more than the 80 GB GPU, and the room is gone. A bigger model: tensor parallelism cannot grow past one node without its additions crossing the slow network, as the figure on adding up across GPUs showed. Both call for a way to split the model that does not need fast links between the pieces. The next part tries the other obvious cut: give whole layers to different machines.
Further derivation: sequence parallelism and selective recomputation.