Why do a deep network’s numbers drift?
A layer’s output is a list of numbers, one per unit of the layer, often thousands of them. Keep it to four. For one example, a layer might hand on −1.4, −0.2, 0.2 and 1.4.
Those numbers scatter around their average, 0. How widely they scatter is the output’s spread: the typical distance of a number from the average. Measured the standard way, the standard deviation that section 1.2 builds, it is exactly 1the spread of −1.4, −0.2, 0.2, 1.4average: (−1.4 − 0.2 + 0.2 + 1.4) ÷ 4 = 0squared distances from it: 1.96, 0.04, 0.04, 1.96their average: 4 ÷ 4 = 1, and √1 = 1This is the standard deviation; section 1.2 builds it step by step. here.
Now let the next layer hand on the same pattern 10% wider: −1.54, −0.22, 0.22 and 1.54. Same shape, spread 1.1. Follow that one number, the spread, up an 80-layer network in which every layer does this. Ten percent sounds harmless.
It is not, because the widenings multiply. After two layers the spread is 1.1 × 1.1 = 1.21 times the input’s. After 80 it is $1.1^{80}$, about 2,048 times1.1⁸⁰, worked outeach layer multiplies the spread by 1.1, so 80 layers multiply it by 1.1⁸⁰seven layers nearly double it: 1.1⁷ = 1.95, and 7.27 layers double it exactly80 ÷ 7.27 ≈ 11 doublings, and 2¹¹ = 2,048The rule of 72: at x% per layer, the spread doubles about every 72 ÷ x layers. as wide. Turn the 10% the other way, 0.9 per layer, and the top layer gets 0.00020.9⁸⁰, worked outeach layer multiplies the spread by 0.90.9⁸⁰ = 0.0002181 ÷ 0.000218 ≈ 4,600: the top layer sees about 1/4,600 of the input’s spread of the input’s spread: the signal has all but vanished.
The 30-second version
About 2,048 times. The widenings multiply rather than add: 1.1⁸⁰ ≈ 2,048, where adding 80 steps of 10% would give only 9. The fix is to re-measure at every layer: subtract the mean of the layer's outputs and divide by their standard deviation, so every layer hands on numbers with a spread of 1, then let the layer rescale them with two learned numbers, γ and β. That is normalisation. Which numbers share one mean is what tells batch norm, layer norm and group norm apart.Call the factor by which a layer multiplies the spread its gain. What sets it is mostly the size of the layer’s weights. A careful initialisation draws them with a spread of $1/\sqrt{n}$ for a layer with $n$ inputs, which sets every gain to about 1 on the first step (Probability, Part 5 works that out).
Then training starts, and every update moves the weights. Nothing holds the gains at 1. A gain of 1.05, which nobody would notice in a single layer, is already about 50 times1.05⁸⁰, worked out1.05⁸⁰ = 49.6by the rule of 72: 5% doubles every 72 ÷ 5 ≈ 14 layers80 ÷ 14 ≈ 5.7 doublings: between 2⁵ = 32 and 2⁶ = 64 after 80 layers.
Real gains wander from layer to layer. The figure’s last phase is an illustrative run whose gains stay between 0.96 and 1.06: no single layer looks wrong, and the top layer still ends 16.5 timesthe wandering run, worked outmultiply the 80 gains: 0.98 × 0.99 × 0.96 × … × 1.03 = 16.5the first twelve are mostly below 1, so the spread dips to 0.89 by layer 12from layer 17 on, most gains sit between 1.03 and 1.06, and those compoundAn illustration with fixed made-up gains, not a measured network. as wide as the input. Step through it, then pick a gain:
Why does a drifting spread hurt? Because numbers leave the range where arithmetic behaves. Huge values overflow. In float16, the 16-bit format many networks run in, the largest number is 65,504. A spread of 2,048 puts typical values in the thousands, and the moment anything squares them, as a variance, a loss or an attention score does, they pass the ceiling: 2,048² is about 4.2 million2,048², worked out2,048 × 2,048 = 4,194,3044,194,304 ÷ 65,504 ≈ 64 times float16’s largest number.
Huge values also saturate. A tanh or a sigmoid fed numbers in the thousands sits flat against its ends, where its slope is zero, and a zero slope passes no gradient back. At an input of just 5, tanh’s slope is already 0.00018tanh’s slope at 5, worked outthe slope of tanh is 1 − tanh²(x)tanh(5) = 0.999911 − 0.99991² = 0.00018.
Tiny values underflow. float16’s smallest normal number is about 0.00006float16’s floor, worked outfloat16 has 5 exponent bits: its smallest normal number is 2⁻¹⁴ = 0.000061below that, subnormal numbers keep fewer and fewer bits, down to 2⁻²⁴ ≈ 0.00000006below that, a value rounds to 0. A signal shrunk to 0.0002 of its size is already close to it, and a few more shrinking layers push it below, where precision drains away bit by bit until the values round to zero.
The backward pass compounds the same way. The gradient travels back through the same 80 layers, and each layer multiplies it by its weights again. Gains of 1.1 send the first layers a gradient thousands of times too big; gains of 0.9, one thousands of times too small. The first layers then take wild steps or stop learning: the exploding and vanishing gradients that made deep networks hard to train.
What would stop it? Something at every layer that measures how big the numbers have become and puts them back to a standard size, whatever the weights did. Re-measure at every layer. The next question is what re-measuring means.
What does it mean to normalise a set of numbers?
Start with the smallest case. A layer’s output is a list of features, one number each per example: feature 1 might be the first of a layer’s 4,096 outputs. A batch is the handful of examples the network processes together. Take one feature, measured on a batch of four examples: 3, 9, 11 and 17.
Two moves standardise them. First, subtract their mean, 10the mean, worked out3 + 9 + 11 + 17 = 4040 ÷ 4 = 10: they become −7, −1, 1 and 7subtracting the mean3 − 10 = −79 − 10 = −111 − 10 = 117 − 10 = 7, centred on zero. Second, divide by their standard deviation, 5the standard deviation, worked outsquare each distance from the mean: 49, 1, 1, 49average the squares: (49 + 1 + 1 + 49) ÷ 4 = 100 ÷ 4 = 25, the variancetake the square root: √25 = 5: they become −1.4, −0.2, 0.2 and 1.4dividing by the spread−7 ÷ 5 = −1.4−1 ÷ 5 = −0.21 ÷ 5 = 0.27 ÷ 5 = 1.4.
As a formula, for $n$ numbers:
\[z = \frac{x - \mu}{\sigma}, \qquad \mu = \frac{1}{n}\sum_{i=1}^{n} x_i, \qquad \sigma^2 = \frac{1}{n}\sum_{i=1}^{n} (x_i - \mu)^2\]In words: $z$ counts how many standard deviations $x$ sits above or below the mean. The mean $\mu$ is the numbers’ balance point. The variance $\sigma^2$ is the average squared distance from it, and the standard deviation $\sigma$, its square root, is the spread back in the units of $x$.
Each move has one job. Subtracting the mean removes any offset the layer below added: the centre lands on zero. Dividing by $\sigma$ removes any gain: the spread lands on 1. Check it on our four: their squares average to 1the new spread, worked outthe mean of −1.4, −0.2, 0.2, 1.4 is 0their squares: 1.96 + 0.04 + 0.04 + 1.96 = 44 ÷ 4 = 1, and √1 = 1, so their standard deviation is exactly 1.
The variance here divides by $n$, not $n - 1$. This is the spread of these four numbers, not an estimate of some larger population’s, and it is what normalisation layers compute.
Now the surprise. Multiply all four inputs by 100: 300, 900, 1,100 and 1,700. The mean becomes 1,000the mean, times 100300 + 900 + 1,100 + 1,700 = 4,0004,000 ÷ 4 = 1,000 and the standard deviation 500the spread, times 100distances from 1,000: −700, −100, 100, 700squares: 490,000 + 10,000 + 10,000 + 490,000 = 1,000,000; ÷ 4 = 250,000√250,000 = 500−700 ÷ 500 = −1.4, and so onThe 100 scales both the distances and the spread, so it cancels., and out come −1.4, −0.2, 0.2 and 1.4 again.
Scaling every input scales the mean and the spread by the same factor, and it cancels. Add 1,000 to every input instead and the mean absorbs it. Whatever the layer below did to the scale or the offset, the next layer sees numbers centred on 0 with a spread of 1.
What if the layer needs those numbers big, or off centre?
Forcing every feature to mean 0 and spread 1 is a strong constraint. Maybe a feature should be large, because the next layer reads it as a strong signal. Maybe it should sit off centre, because a ReLU comes next. Standardising alone would take that choice away from the network.
So the layer gets the choice back. After standardising, each feature is scaled by its own learned number $\gamma$ (gamma) and shifted by its own learned number $\beta$ (beta):
\[y = \gamma z + \beta\]In words: stretch the standardised numbers to a spread of $\gamma$, then move their centre to $\beta$. Both are parameters, learned by gradient descent like the weights, one pair per feature. They start at $\gamma = 1$ and $\beta = 0$, so a fresh normalisation layer passes $z$ straight through.
Does the layer lose anything? No. With $\gamma = 5$ and $\beta = 10$ the standardised numbers come back exactly: 3, 9, 11 and 17undoing the norm, worked out5 × (−1.4) + 10 = −7 + 10 = 35 × (−0.2) + 10 = −1 + 10 = 95 × 0.2 + 10 = 1 + 10 = 115 × 1.4 + 10 = 7 + 10 = 17γ = the old spread and β = the old mean undo standardising exactly.. Setting $\gamma$ to the old spread and $\beta$ to the old mean always undoes the two moves, so normalisation never takes away something the layer cannot learn back. Ioffe and Szegedy made exactly this argument when they introduced the recipe in 2015.
What changes is who sets the scale. Before, it was the product of every gain below. Now it is $\gamma$: one number per feature, set directly by training.
Why $\beta$ matters shows up as soon as a ReLU follows the norm. A ReLU keeps positive numbers and sets negative ones to 0. Standardised numbers are centred on zero, so with $\beta = 0$ everything below the mean is zeroed: two of our foura ReLU after the norm, β = 0with γ = 5, β = 0: y = 5z = −7, −1, 1, 7ReLU: 0, 0, 1, 7, and about half of any roughly symmetric feature.
$\beta$ moves that threshold. The output $\gamma z + \beta$ is negative exactly when $z \lt -\beta/\gamma$. With $\gamma = 5$ and $\beta = 2$ only values below −0.4the cut, worked out5z + 2 < 0 when z < −2 ÷ 5 = −0.4y = 5z + 2 = −5, 1, 3, 9ReLU: 0, 1, 3, 9: only −1.4 was below the cut are zeroed, one of our four. For a bell-shaped feature with $\gamma = 1$ and $\beta = 2$, the ReLU cuts only what lies more than two spreads below the mean, about 2%the share below −2, worked outthe cut sits at z < −2 ÷ 1 = −2for a standard Gaussian, the chance of z < −2 is 0.023Rule of thumb: 16% lies below −1 spread, 2.3% below −2, 0.1% below −3..
One more detail, for safety. If every value in the set is the same, say 4, 4, 4 and 4, the standard deviation is 0 and the division is 0 ÷ 0. So every normalisation layer divides by $\sqrt{\sigma^2 + \epsilon}$ instead of $\sigma$, with $\epsilon$ (epsilon) a tiny constant:
\[z = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}\]PyTorch’s batch, layer and group norm layers default to $\epsilon = 10^{-5}$ (their eps argument). For our four numbers it changes nothing you could see: the spread becomes 5.000001ε at work, worked out√(25 + 0.00001) = 5.000001. For four equal values every $x - \mu$ is 0 and the divisor is about 0.0032ε alone, worked out√(0 + 0.00001) ≈ 0.00320 ÷ 0.0032 = 0, so the output is a clean 0 instead of an error.
What happens to the drift with a norm after every layer?
Put a norm after every layer and the drift has nowhere to compound. Each layer can still stretch or shrink what it hands on by its gain. But the norm right after it divides by the spread it measures, so the next layer starts again from a spread of 1 (times that norm’s $\gamma$). One layer’s error is undone before the next layer sees it.
With a gain of 1.1, the spread goes 1, then 1.1, back to 1, then 1.1, back to 1: a sawtooth instead of a climb to 2,048. The top layer’s output has whatever scale the last norm’s $\gamma$ has learned, not $g^{80}$.
The backward pass gets the same protection. A norm divides by $\sigma$ on the way forward, so it multiplies the gradient by $1/\sigma$ on the way back. If the layer below made its output 1.1 times wider, its $\sigma$ is 1.1 times bigger, and the gradient through the norm shrinks by the same 1.1 that the layer’s weights then multiply it by. The gains cancel in both directions.
There is a quieter gift. Multiply a layer’s weights and bias by any positive number and the normalised output does not change: the mean and the spread scale with them, and the factor cancels, exactly as the times-100 surprise did. A normalised layer ignores the overall scale of its weights. Part 2 shows why that makes training faster.
What does all this cost? For every set of numbers that shares a mean: one mean and one variance, a pass of additions over numbers already in memory. For every feature: two learned numbers, $\gamma$ and $\beta$. A 4,096-wide layer gains 8,192 parametersa norm’s parameters, worked out2 per feature × 4,096 features = 8,192the layer’s matrix: 4,096 × 4,096 = 16,777,216 weights8,192 ÷ 16,777,216 ≈ 0.05%A norm’s parameters are about 2/d of a d × d matrix’s., next to the 16.8 million weights of its 4,096 × 4,096 matrix.
Which numbers should share one mean?
So far “the numbers” were four values of one feature, one per example. But a layer’s output for a batch is a whole grid, and the recipe does not say which of its numbers to average together. Here is a small one: a batch of 4 examples (rows), each with 6 features (columns).
| feature 1 | feature 2 | feature 3 | feature 4 | feature 5 | feature 6 | |
|---|---|---|---|---|---|---|
| example 1 | 3 | 10 | −2 | 6 | 0 | 1 |
| example 2 | 9 | 6 | 4 | 2 | 6 | 7 |
| example 3 | 11 | 10 | 12 | 6 | 0 | 9 |
| example 4 | 17 | 6 | 6 | 2 | 6 | 15 |
Our four numbers, 3, 9, 11 and 17, were the first column: feature 1, down the batch. That is one choice of which numbers share a mean. There are three, and each one is a different technique.
Down a column: one feature, averaged over the examples in the batch. Feature 1’s mean is 10. This is batch norm, Part 2.
Along a row: one example, averaged over its own features. Example 1’s mean is 3example 1’s mean, worked out3 + 10 − 2 + 6 + 0 + 1 = 1818 ÷ 6 = 3. This is layer norm, and its leaner cousin RMSNorm, Part 3.
Within a group: one example, averaged over a group of its features. In an image network the features are channels (one detector’s response at every pixel), and a group is a few channels together. This is group norm, with instance norm as its one-channel case, Part 4.
Each choice is a bet about which numbers are alike enough to be measured together, and each has a catch that the next one fixes. A column needs a batch, which a language model generating one token at a time does not have. A row lumps together features of very different kinds, which costs image networks. Part 5 asks what models actually use in 2026.
Rapid fire: can you do these from memory?
- Each of 60 layers multiplies the spread by 1.1. Roughly how much wider is the top layer's output, and how would you estimate it in your head?
- Standardise 2, 4, 4, 4, 5, 5, 7, 9 by hand.
- Why does multiplying every input by 100 leave the standardised values unchanged, and what does that say about a normalised layer's weights?
- A feature's norm has learned γ = 3 and β = −1. What are the mean and the spread of that feature's output?
- Why does every normalisation layer put an ε under the square root, and what comes out for four equal values?
- A ReLU follows a norm. What share of a symmetric feature is zeroed at β = 0, and where does the cut sit for any γ and β?
- Why do gradients stop compounding once a norm follows every layer?
- In a grid of examples by features, which numbers share a mean under batch norm, and which under layer norm?
Where can you go deeper?
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Sergey Ioffe and Christian Szegedy, ICML 2015, arXiv:1502.03167. Section 3 introduces the recipe this part built: standardise each feature, then scale and shift it by a learned γ and β, with the argument that setting γ to the standard deviation and β to the mean recovers the original activations. Algorithm 1 is the whole transform, ε included.
- Dive into Deep Learning, Aston Zhang, Zachary Lipton, Mu Li and Alexander Smola, Section 8.5, Batch Normalization: the same recipe with code, and a second explanation of why it helps.
- Layer Normalization, Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, 2016, arXiv:1607.06450: the row answer, which Part 3 takes apart.
- Group Normalization, Yuxin Wu and Kaiming He, ECCV 2018, arXiv:1803.08494. Figure 2 draws batch, layer, instance and group norm as cubes of numbers and shades the ones that share one mean and variance: this part’s grid, with a pixel axis added.