The Transformer · 3. The FFN: each word thinks on its own

A worked example, every number computed

Attention mixes rows; the FFN works on one row at a time with the same weights. The row for “by”, [−0.303, −1.116, −0.200, 1.619], is widened to 16 numbers, ReLU keeps 4, and they write [0, 1.038, 0.432, 0] back: a name comes next.

Open video ↗
Read the walkthrough

One row at a time. Attention is the only step where words meet. The FFN takes each row alone, with the same W₁ and W₂ for every row.

Widen. h W₁ + b₁: each of the 16 columns of W₁ is a pattern, every bias is −1. Unit 4 is 1.619 − 1 = 0.619.

ReLU. max(0, u) silences the negatives. 4 of 16 units fire for “by”: +asks 0.619, −name 0.116, work+asks 0.316, small+asks 0.419.

Write back. Each live unit adds its activation times its row of W₂: [0, 1.038, 0.432, 0]. “a name” goes from −1.116 to −0.078 once it is added to the row.

Every row. “of” and “by” fire the same two name-writing units; after “of” comes Jane, after “by” comes Breaking.

Keys and values. FFN(x) = Σᵢ ReLU(x · kᵢ + bᵢ) vᵢ, with kᵢ the i-th column of W₁ and vᵢ the i-th row of W₂.

Why 4× wide. Attention has 4d² weights, the FFN 8d²: in the 2017 base model 1,048,576 against 2,097,152 per block, two thirds in the FFN. SwiGLU uses three matrices and about 8d/3 hidden units to keep the count.