Why isn’t Part 6’s screen proof?

Part 6 left us with a candidate, A, a rival, B, and a promise to test them properly. Every number behind them came from arithmetic on paper: bytes divided by a link’s speed, FLOPs divided by a GPU’s peak. That screen was the right tool for throwing out layouts that cannot work. From here on the series stops estimating and starts measuring, because the screen cannot see what decides between two layouts that can.

Start with what the screen does know. A splits every layer across 8 GPUs and runs 4 micro-batchesMicro-batchThe part of the batch one GPU pushes through a single forward and backward pass. A training step’s batch is too big to run at once, so each GPU runs several micro-batches one after another and adds up their gradients before the one update (gradient accumulation). In our 100B run a micro-batch is one 8,192-token sequence. per group; B splits across 4 and runs 2. Either way each GPU does a 1,024th of the step’s arithmetic, 6.21 seach GPU’s arithmetic per step6 × P × tokens per step = 6 × 10¹¹ × 4,194,304 ≈ 2.52 × 10¹⁸ FLOPs÷ 1,024 GPUs ≈ 2.46 × 10¹⁵ FLOPs eachat 40% of 989 × 10¹² FLOP/s: 2.46 × 10¹⁵ ÷ 3.96 × 10¹⁴ ≈ 6.21 sThe same for every layout: it is Part 1’s 6.2 s step. at 40% of peak. So the two layouts can only differ in time that is not arithmetic.

Most of that time is the tensor additions from Part 3. A spends 0.84 sA’s additions per stepone micro-batch at TP 8: 94.0 GB ÷ 450 GB/s ≈ 209 ms (Part 3)4 micro-batches a step: 4 × 209 ms ≈ 0.84 s a step on them, B only 0.36 sB’s additions per stepone micro-batch at TP 4: 1.5 messages a collective instead of 1.75, about 80.5 GB ÷ 450 GB/s ≈ 179 ms (Part 6)2 micro-batches a step: 2 × 179 ms ≈ 0.36 s. If every addition makes the GPU wait, A’s step takes 7.05 s and B’s 6.57 s, so B is about 7% fasterif every addition makes the GPU waitA: 6.21 + 0.84 = 7.05 sB: 6.21 + 0.36 = 6.57 s7.05 ÷ 6.57 ≈ 1.07: B about 7% faster, 0.48 s a stepSpeedup is the old time divided by the new.. If the additions run while the GPUs compute, both steps take 6.21 s and B gains nothing.

The screen cannot tell which of those two worlds we are in. It also cannot see whether B’s multiplies, twice as wide as A’s, run faster per FLOP.

So we write the prediction down before the run, in a form a measurement can break.

PredictionB’s step is shorter than A’s, by up to about 0.48 s, because it waits less on the tensor additions, and B still fits: its measured peak memory stays under 80 GB.
Proved wrong ifB is not faster at the same 4,194,304 tokens a step, or its peak memory does not fit. Then either A’s additions were already hidden, or B’s 63.2 GB costs more than the screen thought.

What has to stay the same?

A comparison means something only if both runs train the same model on the same work. B keeps the batch with 256 groupsB’s batch1,024 GPUs ÷ 4 per group = 256 groups256 × 1 sequence × 2 micro-batches × 8,192 = 4,194,304 tokens of 4 GPUs, each running 2 micro-batches: 256 × 1 × 2 × 8,192 = 4,194,304 tokens, the same as A’s 128 × 1 × 4 × 8,192. Let the tokens per step change and we are comparing two experiments, not two layouts.

Hold everything else fixed too: the same data in the same order, the loss averaged over the same tokens, the same optimizer and its settings, the same precision, the same learning-rate schedule. Then check that both runs are right before asking which is fast. Run a few dozen steps of each from the same starting weights and compare the loss curves.

The curves will not match to the last digit, because adding the same numbers in a different order rounds differently. They should track each other closely. A loss that drifts apart means one layout computes something different, and a faster wrong answer is a bug, not a speedup.

Why is time spent talking not time spent waiting?

We have met this twice already. In Part 2 the gradient additions started while the backward pass was still running; in Part 4 a pipeline stage sat idle until the one before it handed over. A measurement has to tell those two cases apart, because the same transfer can cost nothing or everything.

Communication that runs while the GPU computes is free; communication the GPU has to wait for adds straight onto the step. A step ends when its last necessary piece ends, so follow the chain of things that each wait for the one before, from the start of the step to its end. The longest such chain is the critical path, and only time on it makes the step longer.

That is why a timeline beats a total. A profiler trace shows every kernel and every transfer against the clock, so it tells us whether A’s additions sit on the critical path or beside it: exactly the question the screen could not answer. A trace that shows 500 ms of communication does not promise 500 ms of savings, only whatever part of it was on the critical path.

What does faster actually mean?

There are three clocks, and they disagree: how busy the GPU is, how long a step takes, and how many tokens a second the run trains. Only the third is the one we pay for.

A GPU can look busier by redoing work it threw away, and a step can look shorter by carrying fewer tokens. So we compare layouts by useful tokens per second, at the same tokens per step. The speedup is the old time divided by the new: if A takes 7.05 s and B 6.57 s for the same tokens, B is 1.07 times as fast.

When we add GPUs instead of changing the layout, the same idea gives the scaling efficiency: the speedup divided by how many times more GPUs we used. Twice the GPUs for 1.8 times the speedscaling efficiencyefficiency = speedup ÷ how many times more GPUs1.8 ÷ 2 = 0.9 = 90% is 90% efficient. Keep the total work fixed while adding GPUs (strong scaling), or grow it with them (weak scaling), but say which: they answer different questions.

What small experiment comes before the expensive one?

Do not start with 1,024 GPUs. Run A and B first on a few nodes, where an hour is cheap, and check three things: the loss curves track, the peak memory lands near the screen, and the trace shows the waits the prediction assumed.

A small run can mislead in one specific way. Communication that does not shrink as GPUs are added puts a floor under the step. With 1,600 ms of compute and 250 ms of talking, 16 workers are only 5.3 times16 workers, talking after computingone worker: 1,600 + 250 = 1,850 ms16 workers: 1,600 ÷ 16 + 250 = 350 ms1,850 ÷ 350 ≈ 5.3 times, an efficiency of 5.3 ÷ 16 ≈ 33%With perfect overlap the step is the longer of the two: at most 1,600 ÷ 250 = 6.4 times. as fast as one, not 16 times, and even perfect overlap stops at 6.4 times. Scale up in stages and watch for that floor before trusting a small result at full size.

The cell runs the same arithmetic. Change the 250 ms and watch the serialized time and the overlap bound move apart.

compute_ms = 1600.0
communication_ms = 250.0
for multiplier in [1, 2, 4, 8, 16]:
    local_compute = compute_ms / multiplier
    serialized = local_compute + communication_ms
    optimistic = max(local_compute, communication_ms)
    print(f"{multiplier:2d}x workers: serial={serialized:6.1f} ms, "
          f"overlap bound={optimistic:6.1f} ms")

These are example timings, not measurements. Replace them with numbers from your own trace before predicting a cluster result.

What must we record to reproduce the comparison?

A result nobody can rerun is an anecdote. For every trial keep the model and framework versions, the layout (tensor, pipeline, context and groups), how the state is shared and when it is borrowed, the micro-batch and the tokens per step, the precision, any recomputation, the compiler settings and the data pipeline.

Leave out the first steps, which compile code and warm up the data loader, then report the spread of step times and the peak memory over a stated window, not one lucky step. When a number surprises you, read the trace before turning another knob. Changing several settings until the average looks better teaches less than finding which wait changed and checking that in one more matched run.

Decision carried forward: screens shortlist the layouts; matched runs, correctness checks and the critical path in a trace choose between them.

Say B wins, at 6.57 s a step. Two trillion tokens is about 477,000 steps, so the run needs about 36 daysthe run at B’s pace2 × 10¹² tokens ÷ 4,194,304 per step ≈ 476,837 steps476,837 × 6.57 s ≈ 3.13 × 10⁶ s÷ 86,400 s per day ≈ 36 daysPart 1’s 34 days assumed no waiting at all. of wall time, and over a month GPUs fail. A fast step is not yet a month-long run. Part 8 asks what has to survive a failure: what to save, how often, and how to restart without changing the experiment.

The Ultrascale Playbook is a useful companion for real scaling experiments. Every timing in this part is either the screen’s arithmetic for A and B or a small example; no profiler data or GPU benchmark is claimed here.