What changed between the 2017 block and a 2026 model?

Ask someone to draw the block inside Llama 3 and they will usually draw the 2017 paper. They get the skeleton right: two sublayers on a residual stream, attention and then a feed-forward network. Then almost every box inside it is wrong.

The toy that finishes “Diary of Jane is a song by” with “Breaking Benjamin” is a 2017 block: sinusoids added at the bottom, attention, add then LayerNorm, a ReLU FFN, add then LayerNorm. The film puts it beside a 2026 block and changes one box at a time.

Every change gets its reason, because the reason is what the interviewer is after. The diagram is famously unchanged since 2017; the boxes are not.

They will ask What changed in the transformer block between 2017 and a model like Llama 3 or DeepSeek-V3?
The 30-second version The skeleton is the same, a residual stream with attention and an FFN, now decoder-only. The norm moved from after each add to before each sublayer, so the stream stays a clean sum and deep stacks train stably, and LayerNorm became RMSNorm. Position moved from sinusoids added at the input to RoPE, a rotation of queries and keys in every layer, so scores see relative distance. The ReLU FFN became SwiGLU, and in the largest models a mixture of experts. Grouped-query attention, or latent attention, shrinks the KV cache that limits serving. FlashAttention computes exactly the same attention in blocks, without writing the score matrix to memory. Biases and dropout are gone, and contexts went from hundreds of tokens to 128k.

What are the changes, one line each?

What 2017 2026 Why
norm placement after each add before each sublayer the stream stays a clean sum, so gradients reach the first layers
norm type LayerNorm RMSNorm skip the mean: one pass over the row, trains as well
position sinusoids added at the bottom RoPE in every attention layer the score sees the distance; no parameters
FFN ReLU, 4d wide SwiGLU, about 8d/3 wide a gate trains better at the same size
keys and values one head per query head one head per group of query heads the KV cache shrinks by the group size
FFN at scale one dense FFN experts, a router picks a few more parameters, same work per token
attention kernel scores written to memory FlashAttention, in blocks the same math, far less memory traffic
context 512 tokens (GPT-1, BERT) 128k (Llama 3.1, Gemma 3) local windows, bigger RoPE bases

Two changes sit outside the box diagram. The encoder is gone: GPT kept only the decoder, and the prompt does the encoder’s job. And some things were simply deleted. Biases went because large models train more stably without them (PaLM’s stated reason), and dropout went from pretraining because a single pass over trillions of tokens rarely shows the model the same text twice, so underfitting is the risk.

One newer change guards the softmax: QK-norm normalises each head’s queries and keys before the dot product, so the logits cannot grow without bound over a long run. Gemma 3 replaced Gemma 2’s logit soft-capping with it.

Which numbers come with these changes?

The norm. Pre-norm writes the update as $x \leftarrow x + F(\mathrm{RMSNorm}(x))$, so after $L$ sublayers the stream is

\[x_L = x_0 + F_1 + F_2 + \dots + F_L\]

the input plus every sublayer’s output, with nothing in the way of the gradient. RMSNorm divides by the root mean square and skips the mean: “by”’s row after the first add, $[1.16, 0.31, 1.26, 3.16]$, has RMS 1.80 and becomes $[0.64, 0.17, 0.70, 1.75]$.

The gate. SwiGLU has three matrices where ReLU had two:

\[\mathrm{FFN}(h) = \big(\mathrm{SiLU}(hW_1) \odot hW_3\big)\,W_2, \qquad \mathrm{SiLU}(u) = u\,\sigma(u)\]

To keep the parameter count, $2 \cdot d \cdot 4d = 3 \cdot d \cdot d_{ff}$, so the hidden width drops to $8d/3$. Shazeer (2020) found gated units train to a lower loss at the same size.

The cache. Generating a token reads every cached key and value, so the cache size sets the speed and the batch. Llama 3.1 70B has 64 query heads and 8 key-value heads, and caches 320 KiB per token320 KiB, worked out2 (key and value) × 80 layers × 8 KV heads × 128 dims × 2 bytes= 327,680 bytes = 320 KiB per tokenwith 64 KV heads: 2,621,440 bytes = 2.5 MiBat 131,072 tokens: 43 GB against 344 GBGrouping 8 query heads per KV head makes the cache 8 times smaller.. With one key-value head per query head it would be 2.5 MiB. DeepSeek-V3 goes further with multi-head latent attention: per layer it caches one 512-number latent plus a 64-number key for RoPE, about 70 KB per token across its 61 layers.

The experts. DeepSeek-V3 has 256 routed experts and 1 shared expert per MoE layer, routes each token to 8, and uses 37B of its 671B parameters per token: 5.5%.

The kernel. FlashAttention (Dao et al., 2022) reads the keys in blocks and keeps a running max $m$ and a running sum $\ell$. When a block with a larger max $m^\prime$ arrives:

\[m_{\text{new}} = \max(m, m^\prime), \qquad \ell \leftarrow \ell\, e^{m - m_{\text{new}}} + \sum_{j \in \text{block}} e^{s_j - m_{\text{new}}}\]

On “by”’s scores in two blocks this gives $\ell = 2.04$ with $m = 2$, and a weight of 0.489 on “song”: exactly the number from Part 1, because nothing is approximate.

The context. Gemma 3 interleaves 5 local layers, each seeing only the last 1,024 tokens, with every global layer; at 128k a local layer scores about 1/64 of the causal triangle and caches 1,024 tokens instead of 128k. Its global layers raise the RoPE base from 10,000 to 1,000,000. Some 2025 models go further: Qwen3-Next and Kimi Linear replace three of every four attention layers with linear-time layers.

Can you write the 2026 block in code?

Here is the block with every change in place except the experts: RMSNorm before each sublayer, RoPE, grouped-query attention through a fused kernel, SwiGLU, and no biases.

import torch, torch.nn as nn, torch.nn.functional as F

class RMSNorm(nn.Module):
    def __init__(self, d, eps=1e-6):
        super().__init__()
        self.g, self.eps = nn.Parameter(torch.ones(d)), eps
    def forward(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.g

def rope(x, base=500_000):                    # x: (B, h, T, dk); turn each pair
    T, dk = x.shape[-2], x.shape[-1]
    inv = base ** (-torch.arange(0, dk, 2, device=x.device) / dk)  # theta_i
    ang = torch.arange(T, device=x.device)[:, None] * inv          # m * theta_i
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., 0::2], x[..., 1::2]
    return torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1).flatten(-2)

class Block(nn.Module):
    def __init__(self, d, n_heads, n_kv_heads, d_ff):
        super().__init__()
        self.h, self.kv, self.dk = n_heads, n_kv_heads, d // n_heads
        self.norm1, self.norm2 = RMSNorm(d), RMSNorm(d)
        self.wq = nn.Linear(d, n_heads * self.dk, bias=False)
        self.wk = nn.Linear(d, n_kv_heads * self.dk, bias=False)   # fewer KV heads
        self.wv = nn.Linear(d, n_kv_heads * self.dk, bias=False)
        self.wo = nn.Linear(d, d, bias=False)
        self.w1 = nn.Linear(d, d_ff, bias=False)                   # SwiGLU: three
        self.w3 = nn.Linear(d, d_ff, bias=False)
        self.w2 = nn.Linear(d_ff, d, bias=False)

    def attention(self, x):
        B, T, d = x.shape
        q = self.wq(x).view(B, T, self.h, self.dk).transpose(1, 2)
        k = self.wk(x).view(B, T, self.kv, self.dk).transpose(1, 2)
        v = self.wv(x).view(B, T, self.kv, self.dk).transpose(1, 2)
        q, k = rope(q), rope(k)                                    # position here
        k = k.repeat_interleave(self.h // self.kv, dim=1)          # share KV heads
        v = v.repeat_interleave(self.h // self.kv, dim=1)
        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)  # fused kernel
        return self.wo(y.transpose(1, 2).reshape(B, T, d))

    def forward(self, x):
        x = x + self.attention(self.norm1(x))                     # pre-norm
        h = self.norm2(x)
        return x + self.w2(F.silu(self.w1(h)) * self.w3(h))       # SwiGLU

Llama 3.1 8B is this block with $d = 4096$, 32 query heads, 8 key-value heads and $d_{ff} = 14336$, stacked 32 times.

Agents write most blocks like this now, and they get them right most of the time. The reasons in the table above are what let me read what one wrote and know which line is wrong when it is not.

Rapid fire: can you do these from memory?

  1. Write post-norm and pre-norm as one line each. Why does pre-norm keep deep stacks stable?
  2. What does RMSNorm drop from LayerNorm, and why is that safe?
  3. Why does SwiGLU use a hidden width of about 8d/3 instead of 4d?
  4. Work out the KV cache per token from layers, KV heads, head dim and bytes. What does grouping 8 query heads per KV head save?
  5. Derive the online softmax update when a block with a larger max arrives. Why is FlashAttention exact?