What does the feed-forward network do in a transformer?
For a long time I treated the FFN as the part you mention for completeness. Attention was the idea; the FFN was two linear layers with a ReLU between them. Then I counted the parameters, and it was two thirds of every block.
Our toy makes the point in miniature. After attention, the row for “by” in “Diary of Jane is a song by” knows that a work was named and a creator is due. Nothing in attention says what kind of word comes next.
The FFN is where that happens. It takes each row on its own, lights up a handful of detectors, and for “by” it writes one thing back: a name comes next. The film follows that row through all sixteen hidden units.
The 30-second version
Attention moves information between positions: each output row is a weighted sum of value rows, so once the weights are fixed it is linear in the values. The FFN is the block's nonlinearity on each token's own representation: widen, ReLU, project back, applied to every row separately with the same weights. Attention mixes along the sequence with weights computed from the data; the FFN mixes along the features with weights learned once. It also holds most of the parameters, 8d² against attention's 4d², and reads naturally as a key-value memory.What does the FFN compute, in symbols?
Rows are tokens. For one row $x$ of width $d$:
\[\mathrm{FFN}(x) = \mathrm{ReLU}(xW_1 + b_1)\,W_2 + b_2, \qquad W_1: d \times 4d, \quad W_2: 4d \times d\]In words: widen the row to $4d$ numbers, keep the positive ones, and write a row of width $d$ back. The same $W_1$ and $W_2$ act on every row of the matrix, one row at a time. That is what “position-wise” means.
In the toy, $d = 4$ and the hidden width is 16. The row for “by” arrives as $h = [-0.303, -1.116, -0.200, 1.619]$. Each column of $W_1$ is a pattern: column 4 is $[0, 0, 0, 1]$, “asks for a creator”, so unit 4 computes 0.619unit 4, worked outcolumn 4 of W₁: [0, 0, 0, 1]h · column = −0.303 × 0 − 1.116 × 0 − 0.200 × 0 + 1.619 × 1 = 1.619plus its bias: 1.619 − 1 = 0.619The bias of −1 sets how strongly a pattern must be present before the unit fires..
ReLU sets the twelve negative units to 0. Four fire: “asks for a creator”, “not a name”, “a work and asks”, “a small word that asks”. Each adds its activation times its row of $W_2$, which gives [0, 1.038, 0.432, 0]FFN(h), worked outunits 4 and 14 write “a name”: 0.619 + 0.419 = 1.038units 6 and 11 write “a small word”: 0.116 + 0.316 = 0.432the other twelve units are 0, and the toy’s b₂ is 0Row times matrix: activation i scales row i of W₂, and the rows add up.. Added back to the row, the “a name” feature rises from $-1.116$ to $-0.078$.
The same weights run on “of”. It is also a small word that asks for a creator, so the same two name-writing units fire and it gets $[0, 1.217, 0, 0]$. After “of” comes “Jane”; after “by” comes “Breaking”. One rule, learned once, used at every position.
Why the ReLU? Without it, $xW_1W_2$ is a single $d \times d$ matrix and the FFN collapses into one linear map. The switch is the part that thinks.
Why is the FFN a key-value memory?
Write the same formula one hidden unit at a time:
\[\mathrm{FFN}(x) = \sum_{i=1}^{d_{ff}} \mathrm{ReLU}(x \cdot k_i + b_i)\, v_i, \qquad k_i = \text{column } i \text{ of } W_1, \quad v_i = \text{row } i \text{ of } W_2\]Each $k_i$ is a key: $x \cdot k_i$ asks how well the row matches pattern $i$. Each $v_i$ is a value, added to the row in proportion to the match. In the toy, unit 4’s key is “asks for a creator?” and its value is “write: a name”.
It reads like attention: a match score, then a weighted value. The differences are the point. These keys and values are learned once and stored in the weights, not computed from the sentence, and ReLU replaces the softmax, so any number of slots can fire at once, or none.
Geva et al. (2021) read trained language models this way. Keys in lower layers match shallow patterns, such as particular n-grams, and keys in upper layers match meaning. The values push the output toward words that plausibly follow.
ROME (Meng et al., 2022) went further: it located facts like “the Eiffel Tower is in Paris” in mid-layer FFN weights and edited them by rewriting one value. When people say a model knows something, this is mostly where they mean.
Why is the hidden width four times the model width?
Count the weights. Attention has four $d \times d$ matrices, $W_Q$, $W_K$, $W_V$ and $W_O$: $4d^2$. The FFN has two matrices of $d \times 4d$: $8d^2$. In the 2017 base model, with $d = 512$, that is two thirds of every blocktwo thirds, worked outattention: 4 × 512 × 512 = 1,048,576FFN: 2 × 512 × 2,048 = 2,097,1522,097,152 ÷ (2,097,152 + 1,048,576) = 2/3Rule of thumb: 8d² against 4d², whatever d is..
The 30-second version
It is the 2017 paper's choice, 512 wide and 2,048 hidden, and it stuck because nothing nearby is clearly better: scaling studies found the loss changes only a few percent across a wide range of ratios at a fixed parameter count. In the memory view the hidden width is the number of key-value slots per layer, so a wide FFN stores more patterns. It costs 8d² weights against attention's 4d², two thirds of the block. Gated FFNs such as SwiGLU use three matrices, so they shrink the hidden width to about 8d/3 to keep the same count.Kaplan et al. (2020) varied the ratio over more than an order of magnitude and saw the loss move by a few percent. Four is a convention with a wide basin around it.
Modern models gate the FFN. SwiGLU computes $\big(\mathrm{Swish}(xW_1) \odot xW_3\big)W_2$: one branch decides which slots fire, the other carries a value, multiplied entry by entry. With three matrices the hidden width drops to about $8d/3$ to keep $8d^2$ weights. Llama 3 8B has $d = 4096$ and a hidden width of 14,336. Part 9 opens the gated FFN up.
Mixture of Experts goes the other way: many FFNs per layer, a few run per token. It is the FFN that gets multiplied, not attention, because the FFN holds the weights and works on each token alone. The Mixture of Experts series takes that apart.
Rapid fire: can you do these from memory?
- Write the FFN for one row of width d, with the shapes of W1 and W2.
- Which sublayer mixes information across tokens, and which works on each token alone with shared weights?
- What would the FFN reduce to without the ReLU?
- Rewrite ReLU(xW1 + b1)W2 as a sum over hidden units, and say which matrix holds the keys and which the values.
- How many weights does one block's FFN hold against its attention, in terms of d, and how does SwiGLU keep that count?