Wrong way
srun -n 256
Slurm launches 256 tasks
accelerate --num_processes 8
Each spawns 8 workers
256 × 8 = 2,048
processes launched
Port 29500 collision
2,040 fail immediately
Right way
srun --ntasks-per-node=1
Slurm launches 32 tasks
(1 per node)
accelerate --num_processes 256
Each spawns 8 local GPUs
32 × 8 = 256
processes — correct
Training starts
Slurm = inter-node, Accelerate = intra-node
They compose, not multiply — Slurm handles nodes, Accelerate handles GPUs