The Transformer · 7. Training

One sentence, one pass, one step

Ten tokens become nine next-word questions, answered in one pass. Each row costs −log p of the right word; the average is the loss, 0.669. The gradient at the scores is p − y, and one step on W_U raises every right word: “Breaking” 0.612 → 0.636, loss 0.647.

Open video ↗
Read the walkthrough

Shift by one. Inputs “Diary” … “Benjamin”, targets “of” … “.”: nine questions from one sentence.

One pass. The nine rows go up the block together; every row gives a probability for its true next word.

Surprise. L_by = −log P(Breaking) = −z_Breaking + log Σ e^z = −3.43 + 3.92 = 0.49.

The loss. The average of the nine surprises: 6.02 ÷ 9 = 0.669. That is cross-entropy.

The mask. Row i's target is input i + 1, one column to the right. The causal mask hides it; teacher forcing lets every row read the true earlier words, so nothing waits.

p − y. At “Breaking”: 0.61 − 1 = -0.39. Up by what it missed, down by what it took.

One weight. ∂L/∂W_U = Hᵀ(P − Y)/9. W_U[4, Breaking]: gradient -0.051, so 1.4 → 1.451.

The step. All nine right words rise; the loss falls from 0.669 to 0.647.

Batches. Short sequences are padded at the end. The pads sit in the masked future and are left out of the loss.