One node is not 4 GPUs — it's 8
1 node
Card 0
Die 0 · 64 GB
Die 1 · 64 GB
2 GPUs
Card 1
Die 2 · 64 GB
Die 3 · 64 GB
2 GPUs
Card 2
Die 4 · 64 GB
Die 5 · 64 GB
2 GPUs
Card 3
Die 6 · 64 GB
Die 7 · 64 GB
2 GPUs
PyTorch's view
4 cards × 2 dies = 8 "GPUs" per node
32 nodes × 8 GPUs = 256 world size
Getting this wrong → grad accum off by 2×, LR schedule silently broken