The problem
The fix
The epoch bug
GPU 0
GPU 1
GPU 2
···
GPU 255
Parallel filesystem
200 GB shared dataset
I/O contention → node timeouts → cascade crash
256 concurrent readers hammering random-access reads
Job crashed within minutes — not OOM, but filesystem overload
Shared filesystem
128 shards (pre-split)
Pre-copy to local NVMe before training
Node 0
local NVMe
shard 0
shard 1
shard 2
shard 3
Node 1
local NVMe
s4
s5
s6
s7
···
Node 31
local NVMe
s124
s125
s126
s127
Zero filesystem contention
Each node reads exclusively from local storage
128 shards ÷ 32 nodes = 4 shards per node
~762M tokens per shard × 4 = ~3B tokens per node
Node-sharded data breaks epoch calculation
Bug: local count only
len(dataset) on one node:
6,103,512 samples
÷ 4,096 global batch =
1,490 steps
32× too short!
Fix: multiply by nodes
len(dataset) × num_nodes:
195,312,384 samples
÷ 4,096 global batch =
47,683 steps
Correct!
Why this bug is invisible
Training completes "successfully" — loss decreases, checkpoints saved
Logs say "1 epoch finished" — but model only saw 3B of 100B tokens
The only clue: loss is suspiciously high at "end" of training
Fix: one line — global_samples = local_samples × num_nodes