Ring allreduce
Tree allreduce
G0
G1
G2
G3
G4
···
···
G255
Each GPU sends a
chunk to its neighbor,
accumulates from
the other side
Optimal bandwidth
O(N) latency: 510 steps
Root
Subtree L
Subtree R
G0–G63
G64–G127
G128–G191
G192–G255
G0
G1
···
Reduce up the tree (log₂N steps) → broadcast down (log₂N steps)
log₂(256) = 8 levels → ~16 total steps
vs ring's 510 steps — tree wins at scale
Latency (256 GPUs)
510 steps
Bandwidth efficiency
Optimal
Used at scale?
No — too slow