The Transformer · 2a. Who should “by” listen to?
Queries, keys, values and the square root, every number computed“by” asks for a work. Its query meets every key in a 7 × 7 score matrix, the scores are divided by √d_k = 2 so a wide head does not freeze its softmax, and the weights (0.49 on “song”, 0.18 on “Diary”) mix the values into [1.16, 0.31, 0.26, 1.16].
Open video ↗Read the walkthrough
Three roles. Each row of X is multiplied by W_Q, W_K and W_V. For “by”, q = [2, 0, 0, 0]: it looks for a work. “song” offers k = [2, 0, 0, 0] and hands over v = [2, 0, 0, 2].
A score. q · k = Σ qᵢkᵢ, length times the key's shadow on the query: 4 for “song”, 2 for “Diary”, 0 for “Jane”. All 49 scores at once are S = QKᵀ; the row for “by” is [2, 0, 0, 0, 0, 4, 0].
Why √d_k. A dot product of d_k random numbers of variance 1 has variance d_k: measured spreads 2.0, 7.8 and 22.6 for widths 4, 64 and 512. Big scores make the softmax one-hot, and the slope w(1 − w) falls to 3.4 × 10⁻⁴ at d_k = 64 and 1.5 × 10⁻¹⁰ at 512. Dividing by √d_k brings the variance back to 1.
The toy's ½. d_k = 4, so the scores are halved: “song” goes from 0.82 to 0.49, and its slope from 0.15 to 0.25.
Weights and values. A softmax on every row; then o = Σ wⱼvⱼ. For “by”: [1.16, 0.31, 0.26, 1.16]. Attention(Q, K, V) = softmax(QKᵀ/√d_k)V.