The Transformer · 9. What changed since 2017

One box at a time, each with its reason

The toy's 2017 block beside a 2026 block, changed one box at a time, each change with its reason and its numbers: RMSNorm before each sublayer, RoPE, SwiGLU, grouped-query attention, experts, FlashAttention and contexts of 128k tokens.

Open video ↗
Read the walkthrough

The norm. 2017: x ← LN(x + F(x)). 2026: x ← x + F(RMSNorm(x)), so the stream is x₀ plus every sublayer's output. RMSNorm divides by the root mean square: “by”'s row [1.16, 0.31, 1.26, 3.16] has RMS 1.80 and becomes [0.64, 0.17, 0.70, 1.75].

Position. The sinusoids added at the bottom are gone; RoPE turns q and k in every layer, so q_m · k_n = qᵀ R_(n−m) k.

The gate. FFN(h) = (SiLU(hW₁) ⊙ hW₃) W₂. Three matrices instead of two, so the hidden width drops from 4d to about 8d/3: 128 weights in the toy either way, d_ff = 10.67.

Fewer key and value heads. Llama 3.1 70B caches 2 × 80 × 8 × 128 × 2 = 327,680 bytes (320 KiB) per token; with 64 key-value heads it would be 2.5 MiB. At 128k tokens: 43 GB against 344 GB.

Experts. A router picks a few experts per token. DeepSeek-V3: 256 routed experts and 1 shared, 8 routed per token, 37B of 671B parameters used per token.

FlashAttention. “by”'s scores in two blocks: block A gives m = 1, ℓ = 2.10; block B raises m to 2, rescales ℓ and adds its terms: ℓ = 2.04. The weight on “song” is 0.489, exactly as before.

Longer contexts. Gemma 3 has 5 local layers, each seeing the last 1,024 tokens, per global layer; at 128k a local layer scores about 1/64 of the causal triangle. Qwen3-Next and Kimi Linear replace 3 of every 4 attention layers with linear-time layers.