What happens to our plan when the context grows?

Part 4 left one threat standing. Our plan, tensor parallelism 8 with sequence parallelism inside each node and the state shared across the groups, fits an 8,192-token sequence with about 50 GB to spare. Now the request changes: train on longer documents, first 32K tokens, then 128K.

Activations are stored for every token, so they grow with the length. At 32K each GPU would hold 114 GB. At 128K, sixteen times128K ÷ 8K131,072 ÷ 8,192 = 16in powers of two: 2¹⁷ ÷ 2¹³ = 2⁴ the original length, it would hold about 456 GB28.5 GB × 16activations grow linearly with tokens: 28.5 GB × 16 = 456 GB per GPUFlashAttention never stores the s × s score matrix, so no quadratic term is savedSplitting the sequence over 16 GPUs divides it back to 28.5 GB. per GPU, almost six GPUs’ worth. The work grows even faster: every token looks back at every earlier one, so attention’s arithmetic grows quadraticallyhow fast attention growsattention’s matmuls: about 4 s² h FLOPs per layer, forwardthe rest of the layer: about 24 s h²their ratio is s ÷ 6h: 0.13 at 8K, 2.1 at 128KPast s = 6h, about 61,000 tokens here, attention costs more than the rest of the layer. with the length.

None of our splits touches this. Sharing the state shrinks the state, not the activations. Tensor parallelism is already at 8, and past one node its additions would cross the slow network. A pipeline’s first stage still holds a whole sequence’s worth of activations. And the micro-batchMicro-batchThe part of the batch one GPU pushes through a single forward and backward pass. A training step’s batch is too big to run at once, so each GPU runs several micro-batches one after another and adds up their gradients before the one update (gradient accumulation). In our 100B run a micro-batch is one 8,192-token sequence. is already a single sequence: it cannot get any smaller without cutting a sequence in half.

So that is exactly what we do. Cut the sequence itself into blocks and give each GPU one: 131,072 tokens become 16 blocks of 8,192, and each GPU’s activations fall back to 28.5 GBone block per GPU131,072 tokens ÷ 16 GPUs = 8,192 tokens each, the length we started with456 GB ÷ 16 = 28.5 GB per GPUActivations follow the tokens a GPU holds, not the length of the whole sequence.. This is context parallelism (CP), and its degree c is the number of GPUs that share one sequence.

How can attention see tokens that live on another GPU?

Cutting the sequence is easy for most of a layer. The norms, the multiplies and the MLP work on each token separately, as sequence parallelism showed in Part 3. Attention does not: every token must look at every token before it, and after the cut most of those tokens live on other GPUs.

Look at what attention actually needs from the other tokens. Each token computes three vectors: a query, a key and a value. Its query is compared with every earlier token’s key, and the scores decide how much of each earlier token’s value it mixes in. So a GPU never needs the other tokens’ activations, only their keys and values.

That suggests the trick. Each GPU keeps its own queries and passes its block of keys and values to the next GPU, round a ring. After c − 1 passes, every block has visited every GPU (ring attention). Each pass moves one block of keys and values, and a GPU can work on the block it has while the next one travels.

One subtlety remains. Attention’s weights come from a softmax over every score in a row, but a GPU sees the scores one block at a time. So it keeps a running max and a running sum, and when a bigger score arrives it rescales the sum instead of starting over (online softmax). The result is exact, not an approximation.

Use context parallelism when: one sequence’s activations or attention work are what no longer fit. It adds traffic for keys and values and changes which GPU owns which tokens; it does not replace sharing the state or tensor parallelism.
Why online softmax makes blockwise attention possible

For a query, let a score block have maximum m, exponential sum l, and unnormalized weighted value sum u, all relative to that maximum. To merge blocks a and b,

\[\begin{aligned} m&=\max(m_a,m_b),\\ l&=e^{m_a-m}l_a+e^{m_b-m}l_b,\\ u&=e^{m_a-m}u_a+e^{m_b-m}u_b,\\ \mathrm{output}&=u/l. \end{aligned}\]

This rescales both blocks to a common reference before adding them. It is an exact algebraic decomposition, with usual floating-point differences in implementation. Passing blocks round a ring is one way to move them; gathering all the keys and values at once, or an all-to-all, can be better on other hardware and shapes.

Computation for a large attention tile can hide transfer of another tile. Small tiles or slow links may leave communication exposed. Compare measured tile-compute time against transferred bytes/effective bandwidth instead of assuming the ring is free.

Why does causality unbalance the work?

The ring has a catch, visible in its last phase. Under causal attention a token looks only backwards, so GPU 0, holding the start of the sequence, needs only its own block, while GPU 3 needs all four. Equal numbers of tokens do not mean equal work.

And everyone waits for the busiest GPU. The exchanges in a layer happen together, so the layer runs at the pace of the GPU with the most work.

The fix is to change who owns which tokens. Cut the sequence into twice as many chunks and give each GPU one early chunk and one late chunk (the zig-zag order). The loads change from 1, 3, 5, 7 to 4, 4, 4, 4counting the trianglecut the causal triangle into 8 chunks of rows; chunk i holds about 2i + 1 units of query-key pairs: 1, 3, 5, … 15contiguous: GPU r takes chunks 2r and 2r + 1, which is 4, 12, 20, 28, in the ratio 1 : 3 : 5 : 7zig-zag: GPU r takes chunks r and 7 − r, 16 each, 4 : 4 : 4 : 4The total, 64 units, never changes; only who holds it., and the total stays the same. We changed the ownership, not the mathematics of attention.

What if the model becomes a mixture of experts?

Long context is one way the workload changes. The other is the model itself. Our 100B model is dense: every token passes through every weight. A mixture of experts (MoE) replaces each MLP with several smaller MLPs, the experts, and a small router picks one or two of them for each token.

The model can then hold far more parameters while each token’s arithmetic stays close to a smaller dense model’s. That makes it a different model, not a faster way to train ours: a 100B-parameter MoE does much less arithmetic per token than our dense 100B, so its compute, memory and quality need their own budgets.

It also creates a constraint none of our splits has met. The experts are most of the parameters, too many to copy onto every GPU, so each GPU holds a few of them. A token on GPU 0 whose router picks an expert on GPU 2 has to travel there, and its result has to travel back.

Every GPU sends tokens to every other GPU at once (an all-to-all), once out and once back in each layer’s forward pass, and the same again backward. Spreading the experts across GPUs like this is expert parallelism (EP): the weights stay put and the tokens move.

Pick the hot expert in the figure. When most tokens choose one expert, its GPU works while the others wait, so an MoE’s speed depends on how evenly the router spreads the tokens, not only on how many parameters it has.

What exactly becomes the bottleneck?

For equal-cost experts, a simple compute-utilization bound is mean assigned tokens divided by maximum assigned tokens. The hot expert’s loads [5, 1, 1, 1] give a mean of 2 and maximum of 5, so only 40%mean over max8 tokens over 4 experts: a mean of 2the busiest expert has 52 ÷ 5 = 40% of the balanced paceA sharded step runs at mean load ÷ max load. of an ideal balanced compute schedule is useful at that straggler-limited pace. Real kernels add packing, launch and communication effects.

Capacity limits bound buffers but may drop or reroute tokens; that choice can affect learning. Balance losses and routing biases try to improve assignment. Node-aware routing can reduce costly cross-node destinations while constraining expert selection. Those are quality/performance trade-offs to validate, not purely cosmetic routing optimizations.

What carries forward?

Decision 5 is to add a new split only when the workload creates a new constraint. Neither change applies to our 8K dense run, so it keeps CP = 1 and no experts.

For the 128K branch, context parallelism is the split to test, next to recomputing activations instead of storing them. It takes GPUs away from the groups, just as a pipeline does: with TP 8 and CP 16 a group is 128 GPUs, so only 8 groups remain, each running 4 sequencesthe batch with CP 16a group is now TP 8 × CP 16 = 128 GPUs, so 1,024 ÷ 128 = 8 groupsthe batch: 4,194,304 tokens ÷ 131,072 = 32 sequences per step32 sequences ÷ 8 groups = 4 each, one after anotherEvery layout multiplies back to the GPU count: TP × CP × groups = 1,024. per step to keep the 4,194,304-token batch. For an MoE, expert parallelism arrives with the architecture, together with its own budgets.

We now have every split the run might need: tensor parallelism inside a node, with sequence parallelism; the state shared across the groups; a pipeline across nodes when the model outgrows a node; context parallelism when one sequence outgrows a GPU; and experts spread across GPUs when the model has them. Part 6 puts them together: one layout for the 8K run, checked against memory, the batch and the network before a single cluster hour is spent.

For alternative context/expert implementations and measured experiments, continue with the corresponding sections of the Ultrascale Playbook. The toy diagrams here isolate mechanisms; they are not throughput measurements.