Splitting the batch

Data parallelism: the ring, the overlap, the ZeRO ladder, and the three walls it runs into