PyTorch DDP / FSDP
Application layer
Your code
RCCL (AMD collective comms)
Communication layer
Tree allreduce,
allgather,
reduce-scatter
libfabric (Slingshot CXI)
Transport layer
Network protocol
Hardware
4× Slingshot NICs
25 GB/s each = 100 GB/s
GPU-Direct RDMA
GPU → NIC, bypass CPU
CPU idle
Key tuning knobs at 256 GPU scale
NCCL_ALGO=Tree · NCCL_CROSS_NIC=1 · NCCL_NET_GDR_LEVEL=3 · NCCL_SOCKET_FAMILY=AF_INET