Micro-batch size
2 (attention)
4
8 (mamba)
16
World size (GPUs)
8 (1 node)
64 (8 nodes)
256 (32 nodes)
1024 (128 nodes)
Target global batch
2,048
4,096
8,192
Tokens per optimizer step
4.2M
Progress bar shows (micro-steps for 23,842 opt steps)
47,684