How does a transformer learn from one sentence?
In Part 1 the toy’s unembedding $W_U$ simply existed, and it happened to put 0.61 on “Breaking”. A real model has no such luck: its weights start as random numbers, and every one of them is found by training.
The data is just text. Nobody labels anything, because the next word of real text is the label. Give the toy “Diary of Jane is a song by Breaking Benjamin .” and it can quiz itself on every word: after “Diary”, what comes next? After “Diary of”, what next? All the way to the full stop.
The film follows one step of that quiz in numbers: shift the sentence by one, run one pass, grade every row, and nudge $W_U$ downhill. One step lowers the loss from 0.669 to 0.647.
The 30-second version
Training shifts the sequence by one: tokens 1 to n − 1 are the inputs and tokens 2 to n the targets. One forward pass with the causal mask gives a row of logits per position, and each row's cross-entropy against its target is averaged into the loss. Two things make that one pass legitimate. The mask hides every row's answer, which sits one column to its right, so row i is computed from tokens 1 to i only. And the inputs are the true tokens, not the model's guesses (teacher forcing), so no row waits for another. There is no recurrent state, so the whole sequence is a few large matrix multiplications per layer. Generation has no true next tokens to read, so it must feed its own output back in, one pass per word.What exactly is the loss?
Shift the sentence by one. The ten tokens give nine inputs, “Diary” to “Benjamin”, and nine targets, “of” to the full stop. Under each input sits the word that follows it, so one sentence is nine questions. A real training sequence of 4,096 tokens is 4,095 of them.
The nine inputs go up the block together, exactly as the seven words did in Part 1, and every row ends in a probability for each of the eleven words. Each row is graded on one number: the probability $p_i$ it gave its true next word. The cost is the surprise, $-\log p_i$, and the loss is the average:
\[L = -\frac{1}{n}\sum_{i=1}^{n} \log P(w_{i+1} \mid w_1, \dots, w_i)\]Read it as: how surprised was the model by each true next word, given only the words before it, on average. That is cross-entropy. A sure right answer costs 0, a timid one costs more, and a zero would cost infinity.
Open one row up. The row for “by” must predict “Breaking”, and its probability is a softmax of the row’s scores $z$:
\[L_{\text{by}} = -\log \frac{e^{z_{\text{Breaking}}}}{\sum_j e^{z_j}} = -z_{\text{Breaking}} + \log \sum_j e^{z_j} = -3.43 + 3.92 = 0.49\]The right word’s score against a soft maximum of all eleven. Raise the first or lower the others, and the cost falls. Averaged over the nine rows, the toy’s loss is 0.6690.669, worked outsurprises: 0.607, 0.464, 0.962, 0.929, 0.713, 0.293, 0.491, 0.934, 0.631sum: 6.0236.023 ÷ 9 = 0.669The costliest rows are the least sure: “is” after “Jane” got only 0.38..
Every row counts here, the question’s words as much as the answer’s: in pretraining, every word of the text is a lesson. When a model is fine-tuned to chat, the loss on the prompt’s rows is often switched off, so it learns to answer rather than to ask.
Why is it fair to grade every row in one pass?
Look at where the answers are. Row $i$’s target is input $i + 1$: the next word of the same sentence, one column to the right. Without the causal mask, attention could read it. A head that looks one column ahead copies every target into its own row, the training loss heads for zero, and the model learns nothing, because at generation time that column does not exist yet.
The mask adds $-\infty$ above the diagonal before the softmax, so row $i$ reads inputs 1 to $i$ and nothing else:
\[M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}\]With the shortcut gone, the only way down is to predict the next word from the past, which is the skill you wanted.
The mask also makes the pass parallel. An RNN’s guess at word 7 needs its state after word 6, which needs word 5: nine steps in a row. A transformer keeps no state, and every row reads the true earlier words rather than the model’s guesses (teacher forcing). No row waits for another, so a 4,096-token sequence is a handful of large matrix multiplications per layer instead of 4,096 small steps. That is what let transformers use a GPU whole.
How does one step change the weights?
Differentiate one row’s surprise with respect to its scores and almost everything cancels:
\[\frac{\partial L_i}{\partial z_i} = p_i - y_i\]Here $y$ is 1 at the right word and 0 elsewhere. On the row for “by”, “Breaking” gets 0.61 − 1 = −0.39, negative, so raise it; “Jane” gets +0.25, so lower it. In words: push the right word up by what it missed, and each wrong word down by what it took. The entries add to zero.
The scores come from a linear layer, $z = hW_U + b_U$, with the rows as inputs. So the rule from The Gradient applies: inputs transposed times $\delta$, with $\delta = p - y$ divided by 9 for the average. Stacked over the nine rows:
\[\frac{\partial L}{\partial W_U} = \frac{1}{9}\,H^\top (P - Y), \qquad W_U \leftarrow W_U - \eta\,\frac{\partial L}{\partial W_U}\]$H$ is $9 \times 4$ and $P - Y$ is $9 \times 11$, so the gradient is $4 \times 11$, the shape of $W_U$. Take the weight from feature 4, “asks for a creator”, to “Breaking”. Its gradient is −0.051−0.051, worked outfeature 4 at each row times p − y at Breakingrow “by”: 1.68 × (−0.388) = −0.650row “of”: 1.15 × 0.181 = +0.209; the other seven add −0.022sum −0.463, and −0.463 ÷ 9 = −0.051Only row “by” has Breaking as its target, and it wins the tug of war., so one step with $\eta = 1$ moves it from 1.4 to 1.451.
All 55 numbers of $W_U$ and its bias move at once. Run the sentence again and every one of the nine right words is more likely than before: “Breaking” after “by” goes from 0.612 to 0.636, and the loss from 0.669 to 0.6470.647, worked outsurprises after the step: 0.592, 0.456, 0.948, 0.907, 0.688, 0.272, 0.453, 0.902, 0.608sum 5.827, and 5.827 ÷ 9 = 0.647Every row improved, which one step does not always manage..
The toy moves only $W_U$ because its gradient fits on one line. Backpropagation carries the same $p - y$ further down: into the FFN, through attention to the queries, keys and values, and into the embedding rows. One step moves every weight, and trillions of tokens of such steps is pretraining.
A batch is many sequences at once, and a GPU wants a rectangle, so short ones are padded at the end. The pads sit to the right of every real token, in the part the causal mask already hides, so no real row reads them. Their rows are left out of the loss: PyTorch’s cross-entropy skips them with ignore_index. Many pipelines skip padding altogether and pack documents end to end.
Rapid fire: can you do these from memory?
- Given a sequence of n tokens, what are the inputs and the targets, and how many predictions does one pass train?
- Write the cross-entropy loss for a sequence, and say in words what one term measures.
- What would a model learn if you trained it without the causal mask, and why would it fail at generation?
- What is the gradient of cross-entropy with respect to the logits, and what is the gradient of the unembedding for a batch of rows?
- Pads at the end of a training sequence: why does no real row read them, and how are they kept out of the loss?