The Transformer · 6. The whole stack
Every piece in order, two blocks deepThe whole forward pass, each piece with the part that explained it. Two blocks stacked: “Breaking” rises from 0.61 to 0.69. The toy holds 291 weights, half of them in the FFN; GPT-2 small holds 124,439,808, with the FFN the biggest piece and a third in the vocabulary.
Open video ↗Read the walkthrough
Words to rows. x_i = E[w_i] + PE_i. The toy adds no position, so “by” is E[by] = [0, 0, 1, 2].
One block. H₁ = LN(X + Attn(X)), H₂ = LN(H₁ + FFN(H₁)). The row for “by” goes [0, 0, 1, 2] → [−0.30, −1.12, −0.20, 1.62] → [−0.90, −0.60, −0.18, 1.68]. Shape in, shape out: 7 × 4.
Stack it. H⁽ℓ⁾ = Block_ℓ(H⁽ℓ⁻¹⁾). The toy's block 2 reuses block 1's weights; a real model gives each block its own. In block 2, “by” still puts 0.56 on “song”. P(Breaking) goes 0.61 → 0.69.
The top. Pre-norm models put one last LN before W_U; the post-norm toy needs none. Then softmax(h W_U + b_U), and the loop: Breaking 0.69, Benjamin 0.34, “.” 0.48.
The toy's weights. Embedding 44, attention 48, FFN 144, unembedding 55: 291 in all.
GPT-2 small. Token embedding 38,597,376, positions 786,432, attention 28,348,416, FFN 56,669,184, norms 38,400, unembedding tied to the embedding. Total 124,439,808.
Five lines. H⁽⁰⁾ = E[w] + PE; A⁽ℓ⁾ = LN(H⁽ℓ⁻¹⁾ + Attn(H⁽ℓ⁻¹⁾)); H⁽ℓ⁾ = LN(A⁽ℓ⁾ + FFN(A⁽ℓ⁾)) for ℓ = 1 … N; Z = H⁽ᴺ⁾ W_U + b_U; P = softmax(Z).