Why does batch norm break in a language model?

A deep network’s numbers drift: each layer stretches or shrinks what it hands on, and eighty layers turn a small habit into a huge one. The fix from Part 1 is to re-measure at every layer. Standardise a group of numbers (subtract their mean, divide by their standard deviation), then let the layer scale them by a learned γ and shift them by a learned β.

Part 2 picked the group: one feature, averaged down the batch. A batch is the stack of examples a network processes in one go, and a feature is one of the numbers that describes each example. That is batch norm, and it made image networks train much faster.

Try it on a language model and it breaks in three places.

The puzzle A chatbot server batches requests from different users. Under batch norm, could your answer change because of someone else's prompt? Under layer norm?
The 30-second version Under batch norm computed on the batch, yes: every feature of your tokens is standardised with a mean and spread pooled over everyone's tokens, so a stranger's prompt moves your numbers. Batch norm escapes that at serving time only by switching to fixed running averages from training, which then describe other text, not yours. Under layer norm, no: each token is standardised with the mean and spread of its own features, so nothing else in the batch reaches it.

The first break is length. Sentences differ in length, and a batch must be a rectangle, so the short ones are padded to the longest with dummy tokens. In a language model each token carries its own row of features, so batch norm would give each feature one mean over every token in the batch, pads included.

Take three sentences: the cat sat on (4 tokens), good morning (2) and how are you (3), padded to 4 with 3 pads whose activations are 0. Feature 1 of the nine real tokens is 8, 4, 7, 5 / 9, 3 / 6, 8, 4.

Over the real tokens alone, feature 1 has mean 66, worked out8 + 4 + 7 + 5 + 9 + 3 + 6 + 8 + 4 = 5454 ÷ 9 = 6 and spread 2a spread of 2, worked outdeviations from 6: 2, −2, 1, −1, 3, −3, 0, 2, −2squares: 4 + 4 + 1 + 1 + 9 + 9 + 0 + 4 + 4 = 36variance 36 ÷ 9 = 4, spread √4 = 2. Count the pads in and the same feature has twelve numbers: mean 4.54.5, worked outthe pads add 0 to the sum: still 5454 ÷ 12 = 4.53 pads in 12 cells: the mean drops by a quarter. and spread 3.13.1, worked outsum of squares: 64 + 16 + 49 + 25 + 81 + 9 + 36 + 64 + 16 = 360, the pads add 0mean square 360 ÷ 12 = 30variance 30 − 4.5² = 30 − 20.25 = 9.75spread √9.75 ≈ 3.12.

The token how sits at 6, exactly on the real mean, so it should standardise to 0. With the pads counted it comes out 0.480.48, worked out(6 − 4.5) ÷ 3.12= 1.5 ÷ 3.12 ≈ 0.48. Your sentence’s numbers now depend on how long the other sentences in the batch were.

You can mask the pads out of the sums, and that fixes this break. It leaves the other two.

The second break is generation. A model writing text produces one token at a time, and with a KV cache (the stored keys and values of earlier tokens) only the newest token goes through the layers. For one user that is a batch of one: each feature holds a single number, which is its own mean, with a spread of 0. Standardised, it is 0 whatever the token said.

So batch norm has to fall back on running averages: a mean and spread for each feature, averaged over the batches seen in training and frozen for use afterwards. They describe the training text on average, not the sentence in front of the model.

The third break is the server. Serving systems batch many users’ requests together to keep the GPU busy. If batch norm used the batch’s own statistics there, your answer would shift with whatever strangers happened to type in the same millisecond. That is the puzzle, and it is why the column cannot be the group for text.

What does layer norm average over?

The row. Layer norm (Ba, Kiros and Hinton, 2016) standardises each example across its own features. Here is the series’ grid again: a batch of 4 examples, each with 6 features.

f1f2f3f4f5f6
example 1310−2601
example 2964267
example 3111012609
example 417662615

Batch norm read this table down each column. Layer norm reads it along each row, and never looks at another row.

Example 1 is 3, 10, −2, 6, 0, 1. Its mean is 3a mean of 3, worked out3 + 10 − 2 + 6 + 0 + 1 = 1818 ÷ 6 = 3, its variance 16a variance of 16, worked outdeviations from 3: 0, 7, −5, 3, −3, −2squares: 0 + 49 + 25 + 9 + 9 + 4 = 9696 ÷ 6 = 16 and its standard deviation 4a spread of 4, worked out√16 = 4.

Subtract 3 and divide by 4, and the row becomes 0, 1.75, −1.25, 0.75, −0.75, −0.5row 1 under layer norm(3 − 3) ÷ 4 = 0, (10 − 3) ÷ 4 = 1.75(−2 − 3) ÷ 4 = −1.25, (6 − 3) ÷ 4 = 0.75(0 − 3) ÷ 4 = −0.75, (1 − 3) ÷ 4 = −0.5The result always has mean 0 and spread 1.. Every other row gets the same treatment with its own mean and spread.

Written out for one example with $d$ features $x_1, \dots, x_d$:

\[\mu = \frac{1}{d}\sum_{j=1}^{d} x_j, \qquad \sigma^2 = \frac{1}{d}\sum_{j=1}^{d} (x_j - \mu)^2, \qquad y_j = \gamma_j \,\frac{x_j - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta_j\]

In words: average this example’s own features, measure how far they spread around that average, standardise each one, then scale feature $j$ by its own γ and shift it by its own β. The ε is a tiny number, such as $10^{-5}$, that keeps the division safe when the spread is 0.

The statistics belong to the example; the parameters belong to the feature. There is one γ and one β per feature, learned in training and shared by every example in the batch (and, in a transformer, by every token). The paper calls them each unit’s adaptive bias and gain.

Why does layer norm behave the same in training and in generation?

Because it never asks the batch anything. Rewrite rows 2 to 4 of the grid with any numbers you like: row 1 still has mean 3 and spread 4, and its output does not move.

Batch norm’s output for row 1 moves every time. Swap the other three rows for different sentences and its 10, which batch norm turned into 1, becomes −1.34−1.34, worked outfeature 2 in the new batch: 10, 14, 12, 16mean 52 ÷ 4 = 13, deviations −3, 1, −1, 3variance (9 + 1 + 1 + 9) ÷ 4 = 5, spread √5 ≈ 2.24(10 − 13) ÷ 2.24 ≈ −1.34. In a batch of two, every feature of row 1 comes out −1 or +1two numbers, always ±1two numbers a and b: mean (a + b) ÷ 2each sits half the gap between them from the mean, so the spread is that half-gapstandardised: one is −1, the other +1, and alone it comes out 0.

Everything that made batch norm awkward goes with it. There are no batch statistics, so there are no running averages to keep, and no separate training and evaluation modes. A batch of one is fine. One user’s prompt cannot touch another’s numbers.

Ba, Kiros and Hinton had recurrent networks in mind, where the batch statistics would change at every time step. The same property is what a language model needs, and the Transformer adopted layer norm from the start.

Strictly, a GPU kernel may add up a row’s numbers in a different order when the batch size changes, and floating-point sums depend on their order, so the last bits can still wobble. Thinking Machines described how to make RMSNorm and the other kernels batch-invariant in 2025. That is rounding, not a shared statistic.

What does layer norm do to a vector, geometrically?

Think of an example’s $d$ features as the coordinates of one arrow in $d$ dimensions. Layer norm does two things to that arrow, and each has a picture.

Centring subtracts the same amount, the mean, from every coordinate. That moves the arrow along the all-equal direction $(1, 1, \dots, 1)$ until its coordinates add up to 0. The vectors whose coordinates add up to 0 form a flat plane through the origin (a hyperplane, in $d$ dimensions), and the all-equal direction is perpendicular to it, so centring is a projection onto that plane.

Dividing by the spread then fixes the length. The centred coordinates have squares that average $\sigma^2$, so they add up to $d\sigma^2$. After dividing by $\sigma$ they add up to $d$:

\[\sum_{j=1}^{d} \left(\frac{x_j - \mu}{\sigma}\right)^2 = \frac{d\,\sigma^2}{\sigma^2} = d\]

So every layer-norm output, before γ and β, has mean 0 and length $\sqrt d$: it sits on a sphere of radius $\sqrt d$ inside the plane. Row 1 checks out: $0^2 + 1.75^2 + 1.25^2 + 0.75^2 + 0.75^2 + 0.5^2$ is 66, worked out0 + 3.0625 + 1.5625 + 0.5625 + 0.5625 + 0.25= 6, the number of features, a length of √6√6, worked outlength = √(sum of squares) = √6 ≈ 2.45d features: length √d, always..

Three dimensions are enough to see it. Take $x = (4, 1, 1)$. Its mean is 2, so centring gives (2, −1, −1)(2, −1, −1), worked outmean (4 + 1 + 1) ÷ 3 = 2(4 − 2, 1 − 2, 1 − 2) = (2, −1, −1)check: 2 − 1 − 1 = 0, on the plane, with spread √2√2, worked outsquares 4 + 1 + 1 = 6variance 6 ÷ 3 = 2, spread √2 ≈ 1.41.

Dividing gives (1.41, −0.71, −0.71)(1.41, −0.71, −0.71), worked out2 ÷ 1.414 ≈ 1.41−1 ÷ 1.414 ≈ −0.71, twice, whose length is √3√3, worked outsquares: 2 + 0.5 + 0.5 = 3length √3 ≈ 1.73: a point on the circle of radius √3 in the plane where the entries add up to 0.

A shorter arrow with a different mean, $(0, 2, 1)$, lands on the same circle at (−1.22, 1.22, 0)(−1.22, 1.22, 0), worked outmean 3 ÷ 3 = 1, centred (−1, 1, 0)variance (1 + 1 + 0) ÷ 3 = 2/3, spread ≈ 0.82(−1, 1, 0) ÷ 0.82 ≈ (−1.22, 1.22, 0). And $(5, -1, -1)$, longer and with mean 1, lands on exactly the same point as (4, 1, 1)the same point, worked outmean 3 ÷ 3 = 1, centred (4, −2, −2), twice (2, −1, −1)variance (16 + 4 + 4) ÷ 3 = 8, spread √8 ≈ 2.83, twice √2twice over twice: (1.41, −0.71, −0.71) again.

That is what layer norm means as a shape: it throws away an example’s overall size and its average, and keeps only its direction within the plane. γ and β then stretch each axis and shift the result, so the final output can sit anywhere; the norm only fixes where it starts.

What does RMSNorm leave out, and why does nobody miss it?

The mean. RMSNorm (Zhang and Sennrich, 2019) skips the centring and divides by one number, the root mean square: square every feature, average the squares, take the root.

\[\mathrm{RMS}(x) = \sqrt{\frac{1}{d}\sum_{j=1}^{d} x_j^2 + \epsilon}, \qquad y_j = \gamma_j\,\frac{x_j}{\mathrm{RMS}(x)}\]

In words: measure the typical size of the features from 0 rather than from their mean, and divide by it. There is a γ per feature and no β, since nothing was subtracted that would need putting back.

Row 1 again. Its squares are 9, 100, 4, 36, 0, 1, which average 2525, worked out9 + 100 + 4 + 36 + 0 + 1 = 150150 ÷ 6 = 25, so the RMS is 5an RMS of 5, worked out√25 = 5. RMSNorm turns the row into 0.6, 2, −0.4, 1.2, 0, 0.2row 1 under RMSNorm3 ÷ 5 = 0.6, 10 ÷ 5 = 2, −2 ÷ 5 = −0.46 ÷ 5 = 1.2, 0 ÷ 5 = 0, 1 ÷ 5 = 0.2Its mean is still 3 ÷ 5 = 0.6: nothing re-centres it., where layer norm gave 0, 1.75, −1.25, 0.75, −0.75, −0.5.

Look at the three numbers: mean 3, spread 4, RMS 5. That is no accident. Write each feature as its deviation plus the mean and expand the square:

\[\frac{1}{d}\sum_{j} x_j^2 = \frac{1}{d}\sum_{j} \big((x_j - \mu) + \mu\big)^2 = \underbrace{\frac{1}{d}\sum_{j} (x_j - \mu)^2}_{\sigma^2} + \frac{2\mu}{d}\underbrace{\sum_{j} (x_j - \mu)}_{0} + \mu^2\]

The deviations from a mean always add up to 0, so the middle term vanishes and $\mathrm{RMS}^2 = \mu^2 + \sigma^2$. The mean and the spread are the legs of a right triangle and the RMS is its hypotenuse, and row 1 is the 3-4-5 triangle3-4-5, worked outmean² + spread² = 3² + 4² = 9 + 16 = 25RMS² = 5² = 25.

So the two norms agree exactly when the mean is 0, and nearly agree when it is small. RMSNorm’s output also has length $\sqrt d$: row 1’s comes to √6√6 again, worked out0.36 + 4 + 0.16 + 1.44 + 0 + 0.04 = 6in general: d · RMS² ÷ RMS² = d, just like layer norm’s. In the picture, RMSNorm skips the drop onto the plane and puts every arrow straight onto a sphere of radius $\sqrt d$ around the origin.

Why does nobody miss the mean? Zhang and Sennrich argued that what helps training is the re-scaling: the output does not change when the inputs are scaled up or down. They hypothesised that re-centring is dispensable, and across their experiments RMSNorm matched layer norm while cutting running time by 7% to 64%, depending on the model. One statistic instead of two, and no subtraction.

The trade stuck. Nearly every open large language model of the last few years, Llama, Qwen and DeepSeek among them, normalises with RMSNorm; Part 5 goes through who uses what in 2026.

How does a transformer apply it, token by token?

Inside a transformer the activations, the numbers flowing between layers, have shape (batch, tokens, $d$). The norm treats every token as its own example: one mean and one spread per token, over that token’s $d$ features. A batch of 8 sentences of 5 tokens gets 40 means40, worked outone per token: 8 sentences × 5 tokens = 40, and no token’s statistics touch another’s.

γ and β are the other way round: one of each per feature, shared by every token of every sentence. With $d = 4{,}096$ a layer norm holds 8,192 parameters8,192, worked outγ: 4,096, one per featureβ: 4,0964,096 + 4,096 = 8,192; RMSNorm keeps only the 4,096 γs, a rounding error next to the layers around it.

Each transformer block normalises twice. In the modern pre-norm layout, the residual stream (the running sum that every block adds its output to) is normalised just before attention reads it, and again just before the MLP reads it. One more norm sits at the end of the whole stack, before the output layer.

An 80-layer model therefore has 161 norm layers161, worked out2 per block × 80 blocks = 160plus the final norm: 161. The original Transformer put them elsewhere, after each residual addition (“Add & Norm”), and where exactly the norm should sit is the first question of Part 5.

Layer norm is not the end of the story. In an image the features are channels of different kinds (one detects edges, another a colour), so one mean across all of them is crude, and detection or diffusion runs with one or two images per GPU cannot use the column either. Part 4 splits the difference.

Rapid fire: can you do these from memory?

  1. Name three reasons batch norm breaks in a language model.
  2. Layer norm a row of six numbers with mean 3 and spread 4. What does the 10 become, and what are the mean and spread of the output?
  3. Write the layer-norm formula. Which quantities belong to the token, and which to the feature?
  4. Every layer-norm output of d features, before γ and β, lies on what shape, and what is its radius?
  5. Show that RMS² = mean² + spread². When do RMSNorm and layer norm agree exactly?
  6. Why does RMSNorm have a γ but no β?
  7. Activations of shape (8, 5, 4,096) go through a layer norm. How many means are computed, and how many γ values are used?
  8. How many norm layers does an 80-block pre-norm transformer have, and where do they sit?

Where can you go deeper?

  • Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, Layer Normalization (arXiv, 2016), arXiv:1607.06450. The row instead of the column, the same computation in training and at test time, and the adaptive bias and gain, with recurrent networks as the motivating case.
  • Biao Zhang and Rico Sennrich, Root Mean Square Layer Normalization (NeurIPS 2019), arXiv:1910.07467. The hypothesis that re-centring is dispensable, RMSNorm and its partial variant, and the 7% to 64% speed-ups.
  • Ashish Vaswani and others, Attention Is All You Need (NeurIPS 2017), arXiv:1706.03762. Section 3.1: each sub-layer’s output is LayerNorm(x + Sublayer(x)), the “Add & Norm” boxes of the diagram.
  • Ruibin Xiong and others, On Layer Normalization in the Transformer Architecture (ICML 2020), arXiv:2002.04745. Why moving the norm inside the residual branch (pre-norm) tames the gradients at initialisation and lets training drop the learning-rate warm-up. More in Part 5.