The Transformer · 2b. No peeking, and more than one view
The causal mask and multi-head attention, every number computedIn training every answer sits in the input, so a mask M adds −∞ above the diagonal before the softmax: the future gets exactly 0 and every row still adds to 1. Masking after the softmax leaves row sums like 0.18. Two heads of width 2 look two ways at once: head 1 puts 0.65 on “song”, head 2 spreads 0.14 on every word.
Open video ↗Read the walkthrough
No peeking. The target of each word is the next word, which is also in the input. A word that could read ahead would copy its answer and learn nothing.
The mask. M has 0 on and below the diagonal and −∞ above. softmax(QKᵀ/√d_k + M): e^(−∞) = 0, so the future gets weight 0. For “Jane”: e^1.5, e^0, e^2 over 12.87 gives 0.35, 0.08, 0.57.
Not after. A softmax over all seven columns and then zeroing the future leaves rows that add to 0.18, 0.28, 0.69, 0.57, 0.71, 0.86, 1.00.
Two heads. Columns 1 to 2 and 3 to 4 of Q, K and V make two heads with d_k = 2. Head 1 puts 0.65 on “song” and 0.16 on “Diary”; head 2 has no query and puts 0.14 on every word. Concat(head₁, head₂) W_O = [1.46, 0.23, 0.57, 0.43] for “by” (W_O is the identity in the toy).