Each GPU: full model copy + different data batch
GPU 0
params (full)
grads (full)
optimizer (full)
batch 0
forward → backward
GPU 1
params (full)
grads (full)
optimizer (full)
batch 1
forward → backward
GPU 2
params (full)
grads (full)
optimizer (full)
batch 2
forward → backward
GPU 255
params (full)
grads (full)
optimizer (full)
batch 255
forward → backward
···
AllReduce: average all gradients
The ONLY communication in DDP — one per optimizer step
~50ms for 1B model gradients over Slingshot
optimizer.step()
optimizer.step()
optimizer.step()
optimizer.step()
Identical inputs → identical outputs → all copies stay synchronized
DDP = embarrassingly parallel + 1 sync point