Who should “by” listen to?

Part 1 ran “Diary of Jane is a song by” through one block and kept attention shut, like a box. The row for “by” went in and came out carrying “a work, and a creator is due”. This part opens the box.

The question attention answers is small. “by” has six words behind it. To guess “Breaking” it needs the thing that was created, “song”, and a little of the title, “Diary”. The other four words are noise. How does it pick, with nothing but numbers?

Every word gets three roles: a query (what it looks for), a key (what it offers) and a value (what it hands over). A query that lines up with a key scores high, a softmax turns the scores into weights, and the weights mix the values. That is one line of math, with a square root in it that interviews love.

It is also the line a coding agent writes for you in a second, tests green. The square root, the mask and the heads are exactly what you need on your fingertips to check that line, so the first film builds it from the toy’s numbers: one query by hand, all 49 scores, why the square root is there, and the weighted sum.

What does attention compute, in symbols?

Rows are tokens. The seven words are a $7 \times 4$ matrix $X$, and three learned matrices give every row its three roles:

\[Q = XW_Q, \qquad K = XW_K, \qquad V = XW_V\]

Read it as: the same row, multiplied three ways. In the toy, $W_Q$ moves “asks for a creator” into slot 1, so the query of “by” is $[2, 0, 0, 0]$: it is looking for a work. “song” offers exactly that key, $[2, 0, 0, 0]$.

The scores are every query against every key:

\[S = \frac{QK^\top}{\sqrt{d_k}}, \qquad S_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}\]

Row $i$, column $j$: how well what word $i$ looks for matches what word $j$ offers, divided by the square root of the key width. The dot product is a match: large when the key points where the query points, zero at right angles. For “by” the raw scores are $[2, 0, 0, 0, 0, 4, 0]$, and with $d_k = 4$ they are halved.

Each row becomes weights, and the weights average the values:

\[\mathrm{Attention}(Q, K, V) = \mathrm{softmax}(S)\,V, \qquad o_i = \sum_j w_{ij}\, v_j\]

The softmax works row by row: $e$ to each score over the row’s total, so every weight is positive and every row adds to 1. The output of “by” is [1.16, 0.31, 0.26, 1.16]The output of “by”, worked outweights: song 0.489, Diary 0.180, each other word 0.0660.180 × [1, 1, 0, 1] + 0.489 × [2, 0, 0, 2]+ 0.066 × (the other five values, added: [0, 2, 4, 0])= [0.18 + 0.98, 0.18 + 0.13, 0.26, 0.18 + 0.98] = [1.16, 0.31, 0.26, 1.16]A weighted average: mostly the value of “song”., mostly the value of “song”. Out comes $7 \times 4$ again, one row per word.

Why divide by the square root of the key width?

Ask a room why the scores are divided by $\sqrt{d_k}$ and most people say “to keep the variance at one”. That is half an answer. The other half is why the variance matters.

Half one. If $q$ and $k$ have $d_k$ independent entries of mean 0 and variance 1, each product $q_i k_i$ has variance 1, and $d_k$ of them add up:

\[\mathrm{Var}(q \cdot k) = d_k, \qquad \mathrm{Var}\!\left(\frac{q \cdot k}{\sqrt{d_k}}\right) = \frac{d_k}{d_k} = 1\]

Read it as: scores grow like $\sqrt{d_k}$, and dividing by $\sqrt{d_k}$ cancels that at every width. In the film, 4,000 random pairs have spreads of 2.0, 7.8 and 22.6 at widths 4, 64 and 512.

Half two. Big scores push the softmax toward one-hot, and the softmax’s slope is

\[\frac{\partial w_i}{\partial s_j} = w_i\,(\delta_{ij} - w_j), \qquad J = \mathrm{diag}(w) - ww^\top\]

which is zero everywhere once $w$ is one-hot. Read it as: no slope, no gradient into $W_Q$ and $W_K$, and the head stops learning where to look. Score the row for “by” the way a head 512 wide would and the slope for “song” is 1.5 × 10⁻¹⁰The frozen slope, worked outscores grow like √d_k: × √(512 ÷ 4) = × 11.3[2, 0, 0, 0, 0, 4, 0] becomes [22.6, 0, 0, 0, 0, 45.3, 0]w(song) = 1 − 1.5 × 10⁻¹⁰, almost exactly 1slope w(1 − w) = 1.5 × 10⁻¹⁰At the toy’s own scale the slope is 0.25, about the largest a softmax weight can have.. With the toy’s own $\frac{1}{2}$ it is 0.25.

The 2017 paper makes both points. Unscaled dot products did worse than additive attention for large $d_k$, the authors suspected the softmax’s tiny gradients, and a footnote gives the variance argument.

They will ask Why is dot-product attention scaled by one over the square root of the key width, and what goes wrong without it?
The 30-second version With independent entries of mean zero and variance one, a dot product of width $d_k$ has variance $d_k$, so scores grow like $\sqrt{d_k}$. Large scores make the softmax nearly one-hot, and its Jacobian $\mathrm{diag}(w) - ww^\top$ goes to zero, so almost no gradient reaches $W_Q$ and $W_K$: the attention pattern freezes. Dividing by $\sqrt{d_k}$ brings the variance back to one at every width, so the softmax starts soft and stays learnable. It is a promise about initialisation only: training can still grow the logits, which is why large models add QK-norm (Part 9 covers it).

How do the mask and the extra heads work?

The second film adds the two pieces a decoder needs: a mask, so no word reads ahead, and more than one head, so a word can look in more than one way.

In training the whole sentence goes in at once, and each word’s target is the next word, which sits in the same input. A word that could read ahead would copy its answer. The causal mask hides it:

\[M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}, \qquad A = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)\]

Read it as: add minus infinity to every score above the diagonal, then take the softmax. Since $e^{-\infty} = 0$, the future gets exactly zero weight and drops out of the row’s total. For “Jane” that leaves 0.35, 0.08 and 0.57Jane’s weights, worked outscaled scores it may read: Diary 1.5, of 0, Jane 2e¹·⁵ = 4.48, e⁰ = 1, e² = 7.39, and e^(−∞) = 0 for the four after ittotal: 4.48 + 1 + 7.39 = 12.874.48 ÷ 12.87 = 0.35, 1 ÷ 12.87 = 0.08, 7.39 ÷ 12.87 = 0.57Three weights that add to 1, and four exact zeros. and four zeros. Masking after the softmax would be wrong: the rows would add to 0.18, 0.28 and so on instead of 1.

Multi-head attention cuts the columns into $h$ heads, runs the same attention in each, and mixes the results:

\[\mathrm{MultiHead}(X) = \mathrm{Concat}(\mathrm{head}_1, \ldots, \mathrm{head}_h)\,W_O, \qquad \mathrm{head}_i = \mathrm{softmax}\!\left(\frac{Q_iK_i^\top}{\sqrt{d_k}} + M\right)V_i\]

Read it as: each head gets its own slice of $Q$, $K$ and $V$ and its own weights; the outputs sit side by side and $W_O$ blends them. One head returns one average; $h$ heads return $h$ of them. In the toy, head 1 puts 0.65 on “song” while head 2 spreads 0.14 over every word. The 2017 base model used 8 heads, each 64 wide.

Rapid fire: can you do these from memory?

  1. For n tokens and key width dk, what are the shapes of Q, K, V and QK⊤, and what does row i of QK⊤ hold?
  2. Derive the variance of q · k. Which assumption did you need, and does it still hold after training?
  3. Write the softmax Jacobian and show that it vanishes for a one-hot distribution.
  4. Why is the mask added before the softmax rather than applied after it?
  5. Why use h heads of width d / h instead of one head of width d, and what does WO do?