How do the pieces become a whole model?

Part 1 pushed “Diary of Jane is a song by” through one block and got “Breaking” at 0.61. The parts since took that block apart: attention, the FFN, the residual and the norm, position. Each piece made sense on its own bench.

This part puts them back together. Words become rows, the rows climb two stacked blocks, then a final norm where the model has one, the unembedding and a softmax, and out comes “Breaking Benjamin”. Then I count what each piece costs: the toy’s 291 weights, and GPT-2 small’s 124 million.

That second half is the part interviews probe. “Walk me through the architecture” is really three questions: every step from token ids to the next token, the shape at each step, and roughly where the parameters sit.

They will ask Roughly how many parameters does one transformer block hold, and where are most of them?
The 30-second version For width d, attention has four d × d matrices (queries, keys, values, output): 4d². The FFN has two matrices between d and 4d: 8d². So a block is about 12d², two thirds of it in the FFN; biases and norms are a rounding error. Outside the blocks sit the embedding, V × d, and the unembedding, which is often the same matrix (weight tying). GPT-2 small: 12 blocks of about 7.1 million plus 38.6 million of vocabulary, 124 million in all. Grouped-query attention shrinks the key and value matrices, and a SwiGLU FFN uses three matrices about 8d/3 wide to stay near 8d², so modern blocks land close to the same rule.

What does the whole chain say, in symbols?

Five lines and a loop:

\[H^{(0)} = E[w] + PE\] \[A^{(\ell)} = \mathrm{LN}\big(H^{(\ell-1)} + \mathrm{Attn}(H^{(\ell-1)})\big), \qquad H^{(\ell)} = \mathrm{LN}\big(A^{(\ell)} + \mathrm{FFN}(A^{(\ell)})\big), \qquad \ell = 1, \dots, N\] \[Z = H^{(N)}\,W_U + b_U, \qquad P = \mathrm{softmax}(Z)\]

The first line looks each word up in the embedding table and adds its position. For “by” that is $E[\text{by}] = [0, 0, 1, 2]$ plus nothing, because the toy has no position signal; Part 5 shows what that costs. A RoPE model adds nothing here either: it rotates the queries and keys inside attention instead.

The middle line is one block: attention, add and normalise, then the FFN, add and normalise. It runs $N$ times, 12 in GPT-2 small and 80 in Llama 3 70B. Every block takes an $n \times d$ matrix and hands back one of the same shape, and that is the whole reason blocks stack.

A real model gives every block its own weights. My toy has one block’s worth, so its second block reuses them, the way ALBERT shares weights across layers. The answer survives the second pass, and sharpens: “by” still puts 0.56 on “song” in block 2, and “Breaking” rises from 0.61 to 0.690.69, worked outby after block 1: [−0.90, −0.60, −0.18, 1.68]by after block 2: [−0.59, −0.48, −0.65, 1.73]times W_U plus b_U, then softmax: Breaking 0.692, Jane 0.135The same W_U as Part 1: only the row under it changed..

The last line is Part 1’s: the top row times $W_U$, one score per word, then a softmax. In a pre-norm model, which is GPT-2 and almost everything since, each block normalises the input of its sublayers rather than their sum, so one more LN sits on $H^{(N)}$ before $W_U$. The toy is post-norm, the 2017 layout, and its rows already leave every block normalised.

Generation is this chain in a loop. Two blocks deep, the toy says “Breaking” (0.69), “Benjamin” (0.34), then the full stop (0.48). Part 8 takes the loop apart.

Where do the parameters live?

Start with the toy. The embedding is 11 words by 4 numbers, 44. Attention’s three 4 × 4 matrices are 48 (its output matrix is the identity and its norms are fixed, so neither counts). The FFN is 4 × 16 + 16 + 16 × 4 = 144, and the unembedding 4 × 11 + 11 = 55. That is 291 numbers, and half of them are the FFN.

GPT-2 small, counted from the tensor shapes in its published checkpoint:

piece shape parameters share
token embedding 50,257 × 768 38,597,376 31.0%
positions 1,024 × 768 786,432 0.6%
attention, 12 blocks 12 × 2,362,368 28,348,416 22.8%
FFN, 12 blocks 12 × 4,722,432 56,669,184 45.5%
25 LayerNorms 25 × 2 × 768 38,400 0.03%
unembedding the token embedding, reused 0 0%
total   124,439,808  

The FFN is the biggest piece, as in the toy. Inside a block the rule of thumb is 12d²12d², worked outattention: Q, K, V and output, 4 × 768² + 4 × 768 = 2,362,368FFN: 768 → 3,072 → 768, 8 × 768² + 5 × 768 = 4,722,432rule: 12 × 768² = 7,077,888exact, with two norms: 7,087,872The biases and norms are the 0.1% the rule leaves out.: four $d \times d$ matrices for attention and two $d \times 4d$ for the FFN.

The surprise is the vocabulary: at this size, a third of the model is one lookup table. The unembedding costs nothing extra because GPT-2 ties it to the embedding, so the same 50,257 rows read words in and score them on the way out.

That share falls fast as models grow, because the blocks grow with $d^2$ times the depth and the vocabulary only with $d$. Llama 3 70B keeps separate embedding and unembedding tables, and they still come to about 3%3%, worked outvocabulary 128,256, width 8,192, two tables: 2 × 128,256 × 8,192 = 2.10 billion80 blocks of 855.7 million, plus the tables: 70.55 billion2.10 ÷ 70.55 = 3.0%Counted from its published config: 80 layers, FFN width 28,672, 8 key-value heads. of the model.

Rapid fire: can you do these from memory?

  1. Write the whole forward pass of a decoder-only transformer in five lines, with the shape after each.
  2. Why can blocks be stacked at all? What would break if a block changed the width of its rows?
  3. Where does a pre-norm model put its final norm, and why does a post-norm model not need one?
  4. About how many parameters does one block of width d hold, and which sublayer holds most of them?
  5. What is weight tying, and how many parameters does it save GPT-2 small?