What happens to a month-long run when a GPU fails?

Part 7 ended on a supposition: say B wins, at 6.57 s a step. Two trillion tokens is about 477,000 steps, so the run needs about 36 days of wall time. Over a month on 1,024 GPUs, something will break.

How often? Meta’s Llama 3 report counted 419 unexpected interruptions in 54 days on 16,384 GPUs, about one every three hoursLlama 3’s failure rate54 days × 24 h = 1,296 h1,296 h ÷ 419 interruptions ≈ 3.1 h between themAbout 78% were hardware, most of them GPUs and their memory., most of them hardware. If failures grow with the number of GPUs, our 1,024 would see about one every two daysscaled to our 1,024 GPUs16,384 ÷ 1,024 = 16 times fewer GPUs3.1 h × 16 ≈ 49.5 h ≈ 2 days between failuresAssumes failures grow with the number of GPUs, which is only roughly true.. We plan for one a day, because a bad driver or a flaky link stops a job just as well: about 36 failuresfailures over the runone failure a day × about 36 days ≈ 36 failures over the run.

And under our layout one failure stops everything. The GPU that dies held a slice of the weights, a piece of every multiply its tensor group does, and a share of every adding-up, so the whole job waits for it (the last figure on this page shows how). Everything held in GPU memory is gone.

What survives is what was written to storage: a checkpoint, a saved copy of the run’s state from which training can continue. The figure plays a short run that saves every 100 steps and fails at step 370.

The restart goes back to step 300, and the 70 steps after it are trained twice. A failure costs that replay, plus the time to notice it, restart and load. Two questions follow: what exactly must be saved, and how often.

What must a checkpoint hold to restart the same experiment?

A checkpoint has to let the run carry on as if nothing had happened: the same model, the same next update, the same next examples. The weights alone are not enough.

what is saved why the restart needs it under layout B
the weights the model itself each GPU’s slice
the master copy and Adam’s two moments the next update is computed from them each GPU’s slice
the step number it sets the learning rate in the schedule one number
the data position the next batch must be the one that would have come next one per group
the random state dropout and shuffling must carry on the same sequence one per GPU

The gradients are not saved: the next backward pass rebuilds them. So of the 16 bytes per parameter from Part 1, 14 are written.

Under layout B no GPU holds a whole copy of anything. Each holds its 1,024th of the state, so the checkpoint is 1,024 slices of about 1.37 GBone GPU’s slice of the checkpointsaved per parameter: weights 2 + master copy 4 + two moments 4 + 4 = 14 bytes; gradients are not saved14 bytes × 100 × 10⁹ = 1.4 TB in all1.4 TB ÷ 1,024 GPUs ≈ 1.37 GB per GPU, 1.4 TB in all, each written by the GPU that owns it, plus a small record of which slice is which.

One more trap. Saving takes time, and a crash can land in the middle of it. If two GPUs have written their slices of step 400 and two have not, the folder holds a model that never existed. A checkpoint counts only when every slice is written and a final marker says so.

SavingPause at a step boundary, let every GPU write its slice, write the notes (step, data position, random state), and write the marker last, once everything is safely on storage.
RestoringLoad the newest checkpoint that has its marker, give each GPU its slice, restore the schedule, the data position and the random state, then check that the next steps behave as an uninterrupted run would.

How often should we save?

Every save pauses training, and every failure throws away the work since the last save. Saving often wastes time in pauses; saving rarely wastes time in redone work. Somewhere between is the cheapest interval.

Put numbers on both. Say a save pauses the run for c = 2 minuteswhat a 2-minute save needs1.4 TB ÷ 120 s ≈ 11.7 GB/s written by the whole jobabout 11 MB/s from each of the 1,024 GPUsAn assumption to measure: storage, not the GPUs, usually sets it., and a failure comes once a day, M = 1,440 minutes. Saving every τ minutes spends c ÷ τ of the run in pauses. A failure lands, on average, half an interval after the last save, so each one costs τ ÷ 2 of redone work, once every M: a share of τ ÷ 2M.

The sum is lowest where the two costs are equal, at τ = √(2cM) (Young’s rule). With a 2-minute pause and one failure a day that is about 76 minutesthe best interval, worked outτ = √(2 × 2 min × 1,440 min) = √5,760 ≈ 75.9 minpauses 2 ÷ 75.9 = 2.6%, redone work 75.9 ÷ 2,880 = 2.6%2.6% + 2.6% ≈ 5.3% of the runAt the best interval the two costs are equal, so the total is twice either one.: 5.3% of the run goes to saves and redone work, about 1.9 daysover the whole run5.3% × about 36 days ≈ 1.9 days of the 36.

This is a planning rule, not a forecast. It leaves out the time to notice a failure and restart, failures during the restart, and failures that come in bursts. Saves that run in the background while training continues (asynchronous checkpointing) shrink c, but only to the slowdown they cause, which has to be measured.

Derive the best interval

Let c be the time a save stalls training, τ the training time between saves, and M the mean time between failures of the job. Over a long run, the share of time lost is about

\[W(\tau)\approx\frac{c}{\tau}+\frac{\tau}{2M}.\]

The first term is the pauses; the second assumes each failure costs half an interval on average, once every M. Setting the slope to zero,

\[\frac{dW}{d\tau}=-\frac{c}{\tau^2}+\frac{1}{2M}=0 \qquad\Rightarrow\qquad\tau^\star=\sqrt{2cM}.\]

At that interval c/τ and τ/2M are equal, so the total is twice either one: 2.6% + 2.6% for our numbers. Doubling the failure rate shortens the best interval by √2, not by half.

What does the layout change when we restart?

A failure stops all 1,024 GPUs, so the whole job restarts together. The question is onto what. The figure fails one GPU and brings the job back two ways.

The simple answer is a spare node. Swap it in for the broken one and the job comes back on 1,024 GPUs with the same layout: every GPU loads exactly the slice it wrote, the batch is unchanged, and the run carries on as the same experiment.

Without a spare, the job must run on 127 nodes, 1,016 GPUs. Now every assumption moves. With tensor groups of 4 there are 254 groupswithout a spare node127 nodes × 8 = 1,016 GPUs1,016 ÷ 4 per tensor group = 254 groups512 sequences a step ÷ 254 groups ≈ 2.02 each: not a whole number, and the 512 sequences of a step no longer divide evenly among them. Each GPU would also need a 1,016th of the state instead of a 1,024th, so the saved slices must be cut again for the new owners (resharding).

Libraries can re-cut the slices; they do not keep the experiment the same. Keeping the batch, the data order and the schedule intact is our job. So the plan keeps spare nodes, and the restart path is tested before the launch.

Why rehearse the failure before the launch?

A checkpoint that has never been loaded is a hope, not a recovery plan. Before the full run: train a short while, save, kill the job, restore, and compare the next steps with a run that was never interrupted. The loss should carry on along the same curve, within the small differences floating-point arithmetic allows.

Check the parts that fail quietly: the data position (does the restored run see the examples it should?), the learning-rate schedule, and the random state. A model that loads and then trains on the wrong data looks fine for days.

Then measure what a failure really costs: the save pause, the time to restart and load, and the steps redone. Together they give one number for comparing layouts: useful progress per hour that survives failures. A layout that is slightly slower per step but saves and restarts faster can finish the month first.

Decision carried forward: our 100B plan earns a long allocation only with a tested restart path: spare nodes, a checkpoint every hour or so, and a rehearsed failure whose loss curve matches the uninterrupted run.

That completes the plan: why one GPU cannot do it, how the state is shared, how each layer and the sequence are split, the layout and how to prove it faster, and how to keep it alive for a month. Part 9 hands you the cluster and asks you to defend each of those decisions, the way an interviewer or a reviewer would.

Implementation starting point: PyTorch Distributed Checkpoint. Approximation background: Optimal Checkpoint Interval with Availability as an Objective Function.