The Transformer · 5. Position and RoPE

A worked example, every number computed

Shuffle “Diary of Jane is a song by” and attention gives “by” the same weights and the same output: it reads a bag of words. The 2017 fix adds a vector of sine and cosine clocks to each row; RoPE turns each pair of numbers in the query and the key instead, so a score depends only on how far apart two words are.

Open video ↗
Read the walkthrough

Shuffle. With no mask, “by” puts 0.489 on “song” and 0.180 on “Diary” in either order, and its output is [1.16, 0.31, 0.26, 1.16] both times. With the mask on and “by” kept last, word salad still gets “Breaking” at 0.61.

A set, not a list. Permute the rows of X by P and attention returns P times the old output: softmax(PQKᵀPᵀ)PV = P softmax(QKᵀ)V. The FFN and the norm work row by row, so the block is blind to order.

Clocks. PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(the same). With d = 4 the first pair turns 1 radian per word and the second 0.01. Position 6 is [-0.28, 0.96, 0.06, 1.00].

Adding. x + PE puts the clock into the meaning's slots: “by” becomes 0.96 of a name. A score (x_m + p_m) W (x_n + p_n)ᵀ splits into word·word, word·place, place·word and place·place: 3.39 for “by” and “song”, 9.09 for the same pair three words later.

Rotating. RoPE turns each pair (q₁, q₂) by mθ: [cos mθ, −sin mθ; sin mθ, cos mθ]. “by”'s query (2, 0) at m = 6 and “song”'s key (2, 0) at n = 5 end 1 radian apart: 2 · 2 · cos 1 = 2.16.

Only the distance. q_m · k_n = qᵀ R_mᵀ R_n k = qᵀ R_(n−m) k. Three words later, or in the shuffle, “by” and “song” still score 2.16; “Diary” moves from 1.36 to -0.60.

Longer contexts. Trained on 7 positions, the slow hand has seen 0 to 0.06 rad; 28 positions need 0.27. Dividing every angle by 4 (position interpolation) fits but crowds the fast hand; raising the base 16 times slows only the slow hand. Llama 3.1 uses base 500,000.