Why does a transformer need position at all?

Here is a test I like to run on the toy from Part 1. Shuffle “Diary of Jane is a song by” into “song by Diary a of is Jane” and look at the row for “by”.

With the mask off, so that every word sees every word, nothing changes. “by” still puts 0.489 on “song” and 0.180 on “Diary”, and its output is the same four numbers to the last digit. Put the causal mask back, keep “by” last, and word salad gets “Breaking” at 0.61, exactly like the real sentence.

The toy has no position signal on purpose. This part is about what it costs, and the two ways people have paid for it: the 2017 clocks that are added to each row, and RoPE, which turns the query and the key instead.

They will ask Why RoPE instead of adding a position vector to the input?
The 30-second version Attention on its own is a set operation: shuffle the rows in and the same rows come out, shuffled. Adding a position vector fixes that, but it writes position into the same numbers as meaning, once, at the bottom, so every score mixes word with place and depends on where the sentence starts. RoPE rotates each pair of query and key dimensions by an angle proportional to the position. Rotations compose, so $q_m \cdot k_n = q^\top R_{n-m} k$: the score sees only the offset, the lengths that carry the content are untouched, there are no parameters, it acts in every layer, and a cached key never needs updating. Its frequencies are also the knob for longer contexts.

Why can attention not tell the order? Because it is a weighted sum, and a sum does not care in which order its terms arrive. Permute the rows of $X$ with a permutation matrix $P$ and the queries, keys and values are permuted with it:

\[\mathrm{softmax}\big(PQ\,(PK)^\top\big)\,PV = P\,\mathrm{softmax}\big(QK^\top\big)\,P^\top P\,V = P\,\mathrm{Attn}(X)\]

In words: shuffle in, shuffle out. The softmax works row by row, so it does not care where a row sits, and $P^\top P = I$. The FFN and the norm also work one row at a time, so the whole block is blind to order.

The causal mask only half rescues it. It tells each word which words came before it, as a set, not in what order. In a deep stack that asymmetry leaks a little order back in (Haviv et al., 2022 trained models with no position at all and came close), but the models you can download still put position in on purpose.

How did the 2017 transformer add position?

It added a vector of sines and cosines to every row:

\[PE(pos, 2i) = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE(pos, 2i+1) = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)\]

Read it as clocks. Slots $2i$ and $2i+1$ form a pair, and the pair is a clock hand that turns $1/10000^{2i/d}$ radians per word. With $d = 4$ the toy has two hands: one turns 1 radian per word, the other 0.01. Fast hands tell neighbours apart; slow hands tell near from far. “by” sits at position 6, so it gets $[-0.28, 0.96, 0.06, 1.00]$ added to its row.

That fixes the shuffle, since “by” at position 1 gets a different vector. But the clock lands in the slots that carry meaning: slot 2 means “a name”, and “by” is now 0.96 of a name. Every score mixes the two. Write $W = W_Q W_K^\top$; a score is a row times $W$ times a row, and with the clocks added it splits in four:

\[(x_m + p_m)\,W\,(x_n + p_n)^\top = x_m W x_n^\top + x_m W p_n^\top + p_m W x_n^\top + p_m W p_n^\top\]

Word with word, word with place, place with word, place with place. For “by” at 6 and “song” at 5 that is 3.393.39, worked outword · word: 4 (by asks for a work, song is one)word · place: −1.918, place · word: 1.996, place · place: −0.6854 − 1.918 + 1.996 − 0.685 = 3.39Three words later the same pair scores 4 + 1.979 + 1.992 + 1.118 = 9.09.. Put three words in front of the sentence and the same two words, still one apart, score 9.09. The score depends on where the sentence starts.

How does RoPE turn position into an angle?

Rotary position embedding (Su et al., 2021) keeps the clock speeds and drops the addition. It cuts the query and the key into pairs and turns each pair by the word’s position $m$ times the pair’s speed $\theta$:

\[\begin{bmatrix} q_1^\prime \\ q_2^\prime \end{bmatrix} = \begin{bmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{bmatrix} \begin{bmatrix} q_1 \\ q_2 \end{bmatrix}\]

The query of “by” starts as the pair $(2, 0)$. At position 6 it turns 6 radians, to $(1.92, -0.56)$. The key of “song” is also $(2, 0)$; at position 5 it turns to $(0.57, -1.92)$. The two arrows end one radian apart, and their dot product is 2.162.16, worked outq at 6: (1.920, −0.559); k at 5: (0.567, −1.918)1.920 × 0.567 + 0.559 × 1.918 = 1.089 + 1.072 = 2.16same thing: 2 × 2 × cos 1 = 4 × 0.540 = 2.16A dot product is length × length × the cosine of the angle between.. Turning keeps every length, so what the words are survives intact; only the angle carries place.

Now the reason it works. Write $R_m$ for the turn by position $m$:

\[q_m \cdot k_n = (R_m q)^\top (R_n k) = q^\top R_m^\top R_n\, k = q^\top R_{n-m}\, k\]

Turning back by $m$ and then forward by $n$ is one turn by $n - m$. The positions only survive as the distance between the two words. Put three words in front of the sentence and “by” and “song” still score 2.16; shuffle it and they still score 2.16, because “song” is still right before “by”. “Diary” moved relative to “by”, so its score goes from 1.36 to −0.60. Attention can finally tell the shuffle apart.

RoPE has no parameters. It turns $q$ and $k$ inside every attention layer and never touches $v$, and each key is turned once, by its own position, so a cached key never changes. Earlier schemes such as ALiBi also put the offset into the score; RoPE is the one that won, and Llama, Gemma and DeepSeek-V3 all use it.

What breaks when the context gets longer?

The slow hands. Say the toy trained on 7 positions: its slow hand never turned past $6 \times 0.01 = 0.06$ radians. Ask for 28 positions and it reaches 0.27, angles the weights have never seen. The fast hand has no such problem, since it wraps round and has seen every angle.

There are two well-known fixes. Position interpolation (Chen et al., 2023) divides every angle by the stretch factor, here 4, so position 27 turns like position 6.75; the price is that neighbours crowd together on the fast hands. Raising the base slows only the slow hands: from 10,000 to 160,000 the toy’s slow hand drops from 0.01 to 0.0025 radians per word, and its fast hand stays at 1. NTK-aware scaling and YaRN refine that idea.

Llama 3.1 is a good example: its config sets a base of 500,000, keeps its fast pairs, and slows its slow pairs 8 times to stretch 8,192 positions to 131,072.

Rapid fire: can you do these from memory?

  1. Prove in one line that attention with no mask is permutation equivariant. What does the causal mask change?
  2. Write the sinusoidal encoding. What does each pair of slots do, and why use several speeds?
  3. Expand (xm + pm) W (xn + pn)⊤. Which terms depend on where the words sit?
  4. Derive qm · kn = q⊤ Rn−m k. Why does RoPE turn q and k but not v?
  5. Which RoPE pairs break past the training length, and what do position interpolation and a bigger base each do about it?