Can we just give each GPU different examples?
Imagine the model already fits. Copy its parameters to four workers, give each a different batch, and let them calculate gradients. They disagree because their examples differ. We need one gradient for the combined batch before updating the shared model.
That is the starting idea of data parallelism. The figure below makes the update small enough to check by hand; the next section asks why every worker needs its own copy of the optimizer.
Does every worker need its own copy of every state?
A worker can run a full forward/backward computation on its local examples while owning only part of the persistent model state. This is the distinction behind ZeRO. Its stage number answers which state types have owners, not how many GPUs exist.
For our full 100B model, ZeRO-2 still leaves 200 GBwhy ZeRO-2 stops at 200 GBZeRO-2 shards gradients and optimizer states, 14 of the 16 bytes, but every GPU keeps all the weightsweights: 2 bytes × 100 × 10⁹ = 200 GB on every GPU, whatever d isthe sharded part: 14 × 10¹¹ bytes ÷ 1,024 = 1.4 GB, so 201.4 GB in allFloors: ZeRO-1 stops at 4P bytes (weights and gradients), ZeRO-2 at 2P, ZeRO-3 at nothing. of weights per GPU. More data-parallel workers cannot remove that floor. ZeRO-3 gets persistent state down to 1.56 GB1.56 GB per GPUZeRO-3 shards all 16 bytes1.6 × 10¹² bytes ÷ 1,024 GPUs = 1.5625 × 10⁹ bytes ≈ 1.56 GBDividing by 1,024 is dividing by about a thousand: 1.6 TB becomes about 1.6 GB. over 1,024 owners, but temporarily gathered layers and roughly 228 GB of unpartitioned local activations still stop the naive configuration.
This is progress. We have identified the next constraint rather than declaring victory at a small persistent-state number.
Derive the ZeRO memory floors
With P parameters and d data-parallel GPUs, our 16-byte state convention gives
\[\begin{array}{ll} \text{DDP:}&16P,\\ \text{ZeRO-1:}&4P+12P/d,\\ \text{ZeRO-2:}&2P+14P/d,\\ \text{ZeRO-3:}&16P/d. \end{array}\]These are persistent state counts, not peak memory formulas. In full sharding, the unit you gather matters: wrapping the whole model can require a whole-model temporary allocation; sensible layer/block wrapping bounds that temporary. Prefetching another unit can improve overlap while increasing the live memory set. PyTorch’s fully sharded data parallel (FSDP) lifecycle is documented in its fully_shard reference.
How do I keep all-reduce, reduce-scatter and all-gather apart?
Three collectives carry almost all data-parallel traffic, and their names blur together. Two questions name the right one every time.
What do the GPUs hold? If they hold different versions of the same numbers, like four gradients for the same weights computed from different examples, the versions must be added: the name says reduce. If they hold different pieces of one tensor, like four shards of the weights, nothing is added and the pieces are only collected: the name says gather.
What does each GPU need afterwards? If every GPU needs the whole result, the name says all. If each needs only the slice it owns, the name says scatter.
| the GPUs hold | each needs the whole | each needs its slice |
|---|---|---|
| versions, so add | all-reduce | reduce-scatter |
| pieces, so collect | all-gather | nothing to send |
The buffer sizes give the same answer from the other side. With d GPUs, an all-reduce returns as many numbers as it was given, a reduce-scatter returns d times fewer, and an all-gather d times more. PyTorch’s calls carry this in their signatures: all_reduce works in place, reduce_scatter_tensor writes into an output d times smaller than its input, and all_gather_into_tensor into one d times larger.
The cost follows too. A reduce-scatter or an all-gather sends about (d − 1)/d × M bytes per GPU(d − 1)/d × M, worked outthe tensor is M bytes, cut into d slices of M/d eachall-gather: each GPU holds 1 slice and must receive the other d − 1, passing on as many as it receives: (d − 1) × M/d = (d − 1)/d × Mreduce-scatter: each GPU hands over its own numbers for the d − 1 slices other GPUs own: the same (d − 1)/d × Min the figure, d = 4 and M = 4 numbers: 3/4 × 4 = 3 numbers, the three dots per GPUFor large d, (d − 1)/d is almost 1: each GPU sends about the whole tensor once., where M is the size of the whole tensor. An all-reduce sends twice that, because a ring all-reduce is a reduce-scatter followed by an all-gather: every slice is first summed at its owner, then the owners hand the sums around. Pick “versions” and “the whole result” in the figure and you can watch the two halves happen.
What crosses the wire, and when?
In training the first question answers itself. Gradients are versions: every GPU computed them from different examples, so they are reduced. Weights, once sharded, are pieces, so they are gathered. Reduce gradients, gather weights. From DDP to ZeRO-3, what changes is only where the update sits and how often the weights must be collected.
DDP all-reduces the gradients: for the 100B model, about 400 GBa 100B all-reduce2 × (1,023 ÷ 1,024) × 200 GB ≈ 400 GB sent per GPU400 GB ÷ 50 GB/s ≈ 8 s per stepAn all-reduce sends about twice its size; a reduce-scatter or an all-gather about once. per GPU per step.
ZeRO-1 and ZeRO-2 cut that all-reduce in two and put the update in the middle: reduce-scatter the gradients, let each GPU update the slice it owns, all-gather the new weights. The bytes are the same as DDP’s, and the optimizer each GPU keeps is d times smaller.
ZeRO-3 pays for one more pass. Each GPU still trains the whole model on its own examples, so it needs every weight of a layer while that layer runs, but it stores only its slice. So it borrows the missing slices just before each layer runs and lets go of them right after: at any moment it holds one borrowed layer, not the whole model.
That happens once on the way forward and once more on the way back, before the gradients are added up: about 600 GBZeRO-3’s traffic per stepone pass: (1,023 ÷ 1,024) × 2 bytes × 100 × 10⁹ ≈ 200 GBgather for forward + gather for backward + reduce-scatter: 3 × 200 GB ≈ 600 GB600 GB ÷ 50 GB/s ≈ 12 sDDP, ZeRO-1 and ZeRO-2 make two passes; ZeRO-3 makes three., half as much again.
Why gather the same layer twice? Under full sharding a layer is borrowed, not owned. Just before a layer runs forward, the GPUs all-gather its shards, and right after, each GPU frees them so the next layer can use the room. The backward pass needs the same weights again, to send the gradient down to the layer below through $W^\top$, so it gathers the layer a second time. Then it reduce-scatters the layer’s gradient, and each GPU keeps only its slice.
Letting go is a choice, and it has a price. A GPU could keep each layer after its forward pass instead: that saves the second borrow, but the kept layers pile up in memory, and keeping all of them means every GPU holds the whole model again. PyTorch’s version of ZeRO-3, called FSDP (fully sharded data parallel), lets you pick either.
One more thing multiplies the borrowing. A training step is often run as several smaller batches, called micro-batches, one after another, with their gradients added up before the single update (gradient accumulation). Each small batch borrows the weights again, so the borrowing happens once per small batch, not once per step.
Why gradient averaging is correct, and why scaling stops
For equally sized local batches with mean loss,
\[g=\frac1d\sum_{r=1}^d\nabla_\theta\ell_r(\theta).\]Linearity of differentiation makes this the gradient of the combined mean loss. Unequal token counts, padding masks and different loss normalization need corresponding weighting. Stochastic layers and reduction order can prevent bitwise parity even when the objective is matched.
A ring all-reduce of M bytes on each GPU sends approximately
\[2\frac{d-1}{d}M\]bytes per GPU, with 2(d − 1) sequential phases in the simple ring model. For the 100B model’s BF16 gradients, M = 200 GB, so plain DDP would send about 400 GB per GPU per step. The bytes per GPU approach a constant as d grows, but the latency of the extra phases does not. Meanwhile, at fixed global batch, more replicas mean fewer tokens and less compute per replica to hide communication behind. Starting the additions while the backward pass is still running helps only while there is compute left to hide them behind.
Gradient accumulation changes the batch equation:
\[B_{\mathrm{tokens}}=d\,b\,a\,s.\]Here b is sequences per local micro-batch, a is accumulated micro-batches, and s is tokens per sequence. Deferring DDP synchronization can avoid redundant reductions; sharded training has different live-gradient and parameter-fetch consequences that must be measured.
When would I choose this?
Start with DDP if the measured peak fits. Move to the least aggressive sharding that removes the actual memory wall, then benchmark. “Higher ZeRO stage” is not a universal ranking of speed.
For our 100B problem we keep full sharding: the state is down to 1.56 GB per GPU. But every GPU still builds a whole sequence’s activations, 228 GB, and nothing in this part touched them. Part 3 splits the computation inside each layer.
Mechanism and accounting: the ZeRO paper. Implementation behavior: PyTorch FSDP.