What does batch norm average over?

A deep network’s numbers drift. If each of 80 layers hands on an output 10% wider than its input, the top layer’s output is about 2,048 times1.1⁸⁰, worked outeach layer multiplies the spread by 1.1, so 80 layers multiply it by 1.1⁸⁰1.1⁷ ≈ 1.95, so the spread nearly doubles every 7 layers, and exactly every 7.2780 ÷ 7.27 ≈ 11 doublings, and 2¹¹ = 2,048Growth that compounds per layer is a power, not a sum: 80 steps of 10% added would give only 9. as wide as its input. The fix is to re-measure at every layer: standardise (subtract the mean, divide by the standard deviation), then hand the layer back a learned scale γ and shift β of its own.

That recipe leaves one question open: which numbers share one mean? Part 1 builds the recipe from scratch if you want it. This part takes the first answer, the one from 2015 that let image networks train in an order of magnitude fewer steps: average down the column.

The puzzle A batch norm layer sees a batch of two images whose feature 1 values are 3 and 9. What does it output? And for 3 and 300?
The 30-second version −1 and +1, both times. With two numbers the mean sits halfway between them, both are half the gap away from it, and half the gap is also the standard deviation. So the smaller becomes −1 and the larger +1, whatever they are. Batch norm never sees 3 itself, only where 3 sits among the rest of its batch. That is its strength with big batches, and the source of every one of its catches: test time, small batches, examples that influence each other, and many GPUs.

Start with what a layer hands on. A batch is a handful of inputs pushed through the network together; for each one, the layer produces a row of numbers, its features. Here is a batch of 4 examples with 6 features each:

feature 1feature 2feature 3feature 4feature 5feature 6
example 1310−2601
example 2964267
example 3111012609
example 417662615

A column is one quantity measured on every example: how strongly some pattern fires, say. A row mixes six different quantities with six different scales. So the natural thing to standardise is the column: ask each feature to have mean 0 and spread 1 across the batch.

That is batch norm. Feature 1 holds 3, 9, 11 and 17: mean 10, standard deviation 5Feature 1’s mean and spreadmean: (3 + 9 + 11 + 17) ÷ 4 = 40 ÷ 4 = 10distances from it: −7, −1, 1, 7variance: (49 + 1 + 1 + 49) ÷ 4 = 100 ÷ 4 = 25standard deviation: √25 = 5Batch norm divides by the batch size, not batch size − 1, when it standardises.. Standardised, it becomes −1.4, −0.2, 0.2, 1.4Feature 1, standardised(3 − 10) ÷ 5 = −1.4(9 − 10) ÷ 5 = −0.2(11 − 10) ÷ 5 = 0.2(17 − 10) ÷ 5 = 1.4Check: the four add to 0, and their squares, 1.96 + 0.04 + 0.04 + 1.96 = 4, average to 1..

Feature 2 holds 10, 6, 10, 6. Its mean is 8, every value sits 2 away, so its spread is 2, and it becomes 1, −1, 1, −1Feature 2, standardisedmean: (10 + 6 + 10 + 6) ÷ 4 = 8distances: 2, −2, 2, −2, so variance (4 + 4 + 4 + 4) ÷ 4 = 4 and spread 2(10 − 8) ÷ 2 = 1 and (6 − 8) ÷ 2 = −1. Its own mean, its own spread: nothing from feature 1 is used. Every column gets the same treatment:

feature 1feature 2feature 3feature 4feature 5feature 6
example 1−1.41−1.41−1−1.4
example 2−0.2−1−0.2−11−0.2
example 30.211.41−10.2
example 41.4−10.2−111.4
mean, spread before10, 58, 25, 54, 23, 38, 5

In symbols, with $i$ counting the $B$ examples of the batch and $j$ the features, each feature gets a mean and a variance:

\[\mu_j = \frac{1}{B}\sum_{i=1}^{B} x_{ij}, \qquad \sigma_j^2 = \frac{1}{B}\sum_{i=1}^{B} \left(x_{ij} - \mu_j\right)^2\]

In words: go down column $j$, average it, then average the squared distances from that average.

\[\hat x_{ij} = \frac{x_{ij} - \mu_j}{\sqrt{\sigma_j^2 + \epsilon}}, \qquad y_{ij} = \gamma_j\,\hat x_{ij} + \beta_j\]

In words: standardise every cell with its own column’s statistics, then let each feature rescale and shift itself with two learned numbers. The epsilon $\epsilon$ is a tiny constant (PyTorch uses $10^{-5}$) that keeps the division safe when a column has no spread at all. $\gamma_j$ starts at 1 and $\beta_j$ at 0, so at the start the layer hands on exactly the table above.

In a convolutional layer a feature is a channel, and a channel is measured at every pixel as well as on every image. Batch norm then takes one mean per channel over the batch and every pixel: 32 images of 56 × 56 pixels give each channel’s mean 100,352 numbersOne channel’s batch32 images × 56 × 56 pixels each32 × 3,136 = 100,352 values of that channelIoffe and Szegedy call this the effective mini-batch: batch size times height times width.. Part 4 draws that cube.

Why did batch norm make training so much faster?

Ioffe and Szegedy put it into Inception, Google’s ImageNet classifier at the time, and raised the learning rate. Their abstract: “Batch Normalization achieves the same accuracy with 14 times fewer training steps.” In the paper’s numbers, Inception took 31.0 million steps to reach 72.2% accuracy, and the batch-normalised version with a learning rate 5 times higher, plus a few other tweaks, took 2.1 million: about 14 times fewer14 times fewer stepsInception: 31.0 × 10⁶ steps to 72.2%BN with 5 × the learning rate: 2.1 × 10⁶ steps31.0 ÷ 2.1 ≈ 14.8.

The abstract also says why: “Batch Normalization allows us to use much higher learning rates and be less careful about initialization.” The second half is the drift from Part 1, fixed at every layer. The first half has a mechanism you can check with four numbers.

Multiply a layer’s weights by 10. Every output of the layer is 10 times larger, so each feature’s mean and spread are 10 times larger too, and standardising cancels the 10. The layer’s output after batch norm does not depend on the size of its weights at all, only on their direction:

\[\mathrm{BN}\big((aW)u\big) = \mathrm{BN}(Wu) \quad \text{for any } a > 0\]

Here is the smallest case. A layer with two weights computes $h = w_1 u + w_2 v$. With $w = (1, 2)$ and four examples whose inputs are (1, 1), (3, 3), (5, 3) and (7, 5), it hands on 3, 9, 11 and 17The tiny layer’s outputs1 × 1 + 2 × 1 = 31 × 3 + 2 × 3 = 91 × 5 + 2 × 3 = 111 × 7 + 2 × 5 = 17: feature 1 again. With $w = (10, 20)$ it hands on 30, 90, 110 and 170. Batch norm turns both into −1.4, −0.2, 0.2, 1.4.

The gradient does notice. Take a loss that reads only example 1’s output. Its gradient on $w_1$ is 0.016 at w = (1, 2)The gradient on w₁the gradients reaching h are 0.052, −0.064, −0.036, 0.048 (worked out in the next section)w₁ multiplies u = 1, 3, 5, 7, so its gradient is their weighted sum0.052 × 1 − 0.064 × 3 − 0.036 × 5 + 0.048 × 7 = 0.052 − 0.192 − 0.180 + 0.336 = 0.016 and 0.0016 at w = (10, 20)Ten times the weightsthe outputs are 30, 90, 110, 170: mean 100, spread 50nudging w₁ moves h as much as before, but batch norm divides that by 50 instead of 5so every gradient is 10 times smaller: 0.016 ÷ 10 = 0.0016. Ten times the weights, a tenth of the gradient. Ioffe and Szegedy prove the general case in §3.3: the gradient with respect to $aW$ is $1/a$ times the gradient with respect to $W$.

So a weight that grows takes smaller steps, and the effect is squared. With one learning rate for all weights, a weight vector 10 times longer gets a step 10 times shorter, so its direction, which is all batch norm lets through, turns 100 times more slowlyWhy 100the step is learning rate × gradient: 10 times shorterthe angle it turns the weights through is about step ÷ weight length(1/10) ÷ 10 = 1/100.

And the weights grow on their own. Because scaling them changes nothing, the gradient always points at right angles to them: here w · ∇ = 0At right anglesthe gradient on (w₁, w₂) is (0.016, −0.008)(1, 2) · (0.016, −0.008) = 0.016 − 0.016 = 0True for any loss: if stretching w changes nothing, the gradient has no part along w.. A step at right angles always lengthens a vector, by Pythagoras. A learning rate that is too large lengthens the weights a lot, which shrinks every later step: the layer turns its own learning rate down.

Why batch norm helps so much is still argued over. The paper’s title gives its own story: each layer’s inputs keep changing distribution as the layers below them train, which it calls internal covariate shift, and pinning every mean and variance should stop that.

Santurkar, Tsipras, Ilyas and Madry tested that story in 2018. They injected noise after every batch norm layer, with a mean and variance that changed at every step, so the next layer’s inputs shifted more than in a network with no normalisation at all. It trained almost exactly as well as plain batch norm.

Their conclusion: that stability “has little to do with the success of BatchNorm”, which instead “makes the optimization landscape significantly smoother”, so gradients stay predictive over a larger step. The scale argument above is one concrete piece of that.

Why does the gradient through batch norm involve the whole batch?

In most layers the batch is just many copies run side by side: example 2’s gradient never touches example 1. Batch norm breaks that. Every output depends on the batch’s mean and spread, and those depend on every input, so a change to one input moves every output.

To see by how much, follow one feature. Its $B$ inputs are $x_1, \dots, x_B$, with mean $\mu$ and spread $\sigma$. The standardised values are $z_i = (x_i - \mu)/\sigma$, and the outputs are $y_i = \gamma z_i + \beta$.

Now nudge one input, call it $x_j$, up by a small amount $h$. Three things move: the input itself, the mean and the spread. Each derivative below answers one question: when $x_j$ moves by $h$, how far does this quantity move, per unit of $h$? Feature 1 (3, 9, 11, 17, with mean 10 and spread 5) checks every answer with numbers.

The mean. Write it out in full:

\[\mu = \frac{x_1 + x_2 + \dots + x_B}{B}\]

The input $x_j$ appears in that sum exactly once, multiplied by $1/B$. Raise it by $h$ and the sum rises by $h$, so the mean rises by $h/B$. The change in $\mu$ divided by the change in $x_j$ is the derivative:

\[\frac{\partial \mu}{\partial x_j} = \frac{1}{B}\]

In words: every input owns one $B$-th of the mean, wherever it sits. On feature 1, raise the 9 to 13: the mean goes from 10 to 11The new mean, worked out3 + 13 + 11 + 17 = 44, up from 4044 ÷ 4 = 11, up from 10the input moved 4 and the mean moved 1 = 4 × ¼. Raise the 17 by 4 instead and the mean moves by the same 1. The mean is a plain average, so this holds for a nudge of any size, not only a small one.

The spread. Start one step earlier, with the variance, the average squared distance from the mean:

\[\sigma^2 = \frac{1}{B}\sum_{i=1}^{B} \left(x_i - \mu\right)^2\]

Raising $x_j$ by $h$ changes those distances in two ways. Its own distance, $x_j - \mu$, grows by $h$. And the mean rises by $h/B$, which shortens every distance, its own included, by $h/B$.

The first change grows one square. The square of a distance $d$ changes about $2d$ times as fast as $d$ does, so $(x_j - \mu)^2$ grows by about $2(x_j - \mu)\,h$, and after dividing by $B$ the variance grows by $2(x_j - \mu)\,h/B$.

The second change adds nothing, and it is the step worth slowing down for. Shortening every distance by the same small $h/B$ changes the sum of their squares by about $2 \times h/B$ times the sum of the distances themselves. Distances from a mean always add to zero: on feature 1 they are −7, −1, 1 and 7, which add to 0Distances from the mean3 − 10 = −7, 9 − 10 = −1, 11 − 10 = 1, 17 − 10 = 7−7 − 1 + 1 + 7 = 0True of any numbers: the mean is defined as the point their distances balance around.. So the moving mean leaves the variance where it was, to first order.

There is a picture behind that. The mean is the point that makes the sum of squared distances smallest, the bottom of a bowl, and at the bottom of a bowl a small step sideways changes the height by nothing. Only the first change counts:

\[\frac{\partial \sigma^2}{\partial x_j} = \frac{2\,(x_j - \mu)}{B}\]

The spread is the square root of the variance, and a square root moves $1/(2\sigma)$ times as much as the number under it (the derivative of $\sqrt{v}$ is $1/(2\sqrt{v})$). The two 2s cancel:

\[\frac{\partial \sigma}{\partial x_j} = \frac{1}{2\sigma}\cdot\frac{2\,(x_j - \mu)}{B} = \frac{x_j - \mu}{B\,\sigma} = \frac{z_j}{B}\]

In words: an input drags the spread by its own standardised value, divided by $B$. Raise an input that sits far above the mean and the spread widens; raise one near the mean and the spread barely moves; raise one below the mean and the spread narrows, because the raise pulls it in towards the mean.

On feature 1, the 17 sits at 1.4 standard deviations, so raising it by 0.1 should widen the spread by about 0.0350.035, worked out and checkedpredicted: z ÷ B × h = 1.4 ÷ 4 × 0.1 = 0.035checked: 3, 9, 11, 17.1 have mean 10.025 and squared distances adding to 101.4075variance 101.4075 ÷ 4 = 25.352, spread √25.352 ≈ 5.035, and it does: 5 becomes 5.035. The 9 sits at −0.2, so raising it by 0.1 should narrow the spread by 0.0050.005, worked out and checkedpredicted: z ÷ B × h = −0.2 ÷ 4 × 0.1 = −0.005checked: 3, 9.1, 11, 17 have mean 10.025 and squared distances adding to 99.8075variance 99.8075 ÷ 4 = 24.952, spread √24.952 ≈ 4.995: 5 becomes 4.995.

With the safety constant, $\sigma$ stands for $\sqrt{\sigma^2 + \epsilon}$ everywhere. The constant does not depend on any input, so every step above goes through unchanged.

Each output. Now combine the two moving parts. Each standardised value is a numerator over a denominator, $(x_i - \mu)$ over $\sigma$, and nudging $x_j$ moves both. By the product rule, count each part’s change with the other held still:

\[\frac{\partial z_i}{\partial x_j} = \frac{1}{\sigma}\cdot\frac{\partial (x_i - \mu)}{\partial x_j} \;-\; \frac{x_i - \mu}{\sigma^2}\cdot\frac{\partial \sigma}{\partial x_j}\]

The numerator: $x_i$ itself moves only when it is the input being nudged, and the mean moves by $1/B$ whichever input it is. Write $\delta_{ij}$ for a switch that is 1 when $i = j$ and 0 otherwise, and the numerator moves by $\delta_{ij} - 1/B$.

The denominator: put in the spread’s derivative, $z_j/B$, and note that $(x_i - \mu)/\sigma$ is just $z_i$:

\[\frac{x_i - \mu}{\sigma^2}\cdot\frac{z_j}{B} = \frac{1}{\sigma}\cdot\frac{z_i\, z_j}{B}\]

Together:

\[\frac{\partial z_i}{\partial x_j} = \frac{1}{\sigma}\left(\delta_{ij} - \frac{1}{B} - \frac{z_i\, z_j}{B}\right)\]

In words, three effects, each scaled by $1/\sigma$. A direct push, felt only by the nudged input’s own output. The moved mean, which lowers every output by the same amount. And the changed spread, which pulls every output towards 0 in proportion to how far out it already sits and how far out the nudged input sits.

Here is example 1’s output, at −1.4, as each input is nudged in turn:

input nudgedits zdirect pushmoved meanchanged spreadtotal ÷ 5
example 1 (3)−1.41−0.25−0.490.052
example 2 (9)−0.20−0.25−0.07−0.064
example 3 (11)0.20−0.250.07−0.036
example 4 (17)1.40−0.250.490.048

The moved mean is $-1/B = -0.25$ in every row. The changed spread is 0.35 times the nudged input’s zThe spread column, worked out−z₁ × zⱼ ÷ 4 with z₁ = −1.4, so 1.4 × zⱼ ÷ 4 = 0.35 × zⱼ0.35 × (−1.4) = −0.49 and 0.35 × (−0.2) = −0.070.35 × 0.2 = 0.07 and 0.35 × 1.4 = 0.49, and the last column adds the row and divides by the spread, 5: (1 − 0.25 − 0.49) ÷ 5 = 0.052The totals, worked outexample 1: (1 − 0.25 − 0.49) ÷ 5 = 0.26 ÷ 5 = 0.052example 2: (0 − 0.25 − 0.07) ÷ 5 = −0.32 ÷ 5 = −0.064example 3: (0 − 0.25 + 0.07) ÷ 5 = −0.18 ÷ 5 = −0.036example 4: (0 − 0.25 + 0.49) ÷ 5 = 0.24 ÷ 5 = 0.048, and so on down.

The signs make sense. Raising example 2’s 9 lifts the mean, which pushes example 1’s output, already below the mean, further down, and the spread barely moves: negative. Raising example 4’s 17 lifts the mean too, but mostly it widens the spread, which pulls example 1’s −1.4 in towards 0, up: positive. I checked all four against finite differences, nudging each input by $10^{-6}$: they agree to within $10^{-9}$.

What gradient reaches each input?

The loss reads every output, and nudging one input moves every output, so the gradient on that input adds up one path per output. Let $g_i = \partial L/\partial y_i$ be the gradient arriving at output $i$ from the layers above. Each output is $\gamma$ times its standardised value plus $\beta$, so it moves $\gamma$ times as much, and the chain rule gives:

\[\frac{\partial L}{\partial x_j} = \sum_{i=1}^{B} g_i\,\gamma\,\frac{\partial z_i}{\partial x_j} = \frac{\gamma}{\sigma}\sum_{i=1}^{B} g_i\left(\delta_{ij} - \frac{1}{B} - \frac{z_i\, z_j}{B}\right)\]

Split the sum into its three terms. The switch $\delta_{ij}$ is 0 except when $i = j$, so the first term keeps only $g_j$. The second is $1/B$ times the sum of all the $g_i$: their average, $\mathrm{mean}(g)$. In the third, $z_j$ does not change with $i$, so it comes out of the sum and leaves $z_j$ times the average of $g_i z_i$, written $\mathrm{mean}(g\,z)$:

\[\frac{\partial L}{\partial x_j} = \frac{\gamma}{\sigma}\Big(g_j - \mathrm{mean}(g) - z_j\,\mathrm{mean}(g\,z)\Big)\]

In words: take the gradient that arrived, remove its average over the batch, remove the part of it that lines up with $z$, and scale by $\gamma/\sigma$.

Back to feature 1, with $\gamma = 1$. Let the loss read only example 1’s output, so the gradient arriving is $g = (1, 0, 0, 0)$. Only the first output’s path carries anything, so the gradient on each input is exactly the last column of the table above.

The formula reaches the same numbers in one line each: mean(g) = 0.25The average of g(1 + 0 + 0 + 0) ÷ 4 = 0.25 and mean(g z) = −0.35The average of g times z1 × (−1.4) + 0 × (−0.2) + 0 × 0.2 + 0 × 1.4 = −1.4−1.4 ÷ 4 = −0.35, so the inputs’ gradients come out as 0.052, −0.064, −0.036 and 0.048The four input gradientsexample 1: (1 − 0.25 + 0.35 × (−1.4)) ÷ 5 = (0.75 − 0.49) ÷ 5 = 0.052example 2: (0 − 0.25 + 0.35 × (−0.2)) ÷ 5 = −0.32 ÷ 5 = −0.064example 3: (0 − 0.25 + 0.35 × 0.2) ÷ 5 = −0.18 ÷ 5 = −0.036example 4: (0 − 0.25 + 0.35 × 1.4) ÷ 5 = 0.24 ÷ 5 = 0.048Subtracting −0.35 × z is adding 0.35 × z.. Example 1’s error reached all four inputs.

And the four sum to zeroTwo sums that vanish0.052 − 0.064 − 0.036 + 0.048 = 0.100 − 0.100 = 0weighted by z: −1.4 × 0.052 − 0.2 × (−0.064) + 0.2 × (−0.036) + 1.4 × 0.048 = −0.0728 + 0.0128 − 0.0072 + 0.0672 = 0. That is not luck: adding the same amount to all four inputs changes no output, because the mean absorbs it, so the gradient can have no part along (1, 1, 1, 1). For the same reason the sum weighted by $z$ is zero too: stretching the inputs away from their mean changes no output either. Those are exactly the two parts the formula removes.

Autograd works all of this out for you. What is worth carrying in your head is the coupling. An example’s output, and its gradient, carry a little noise from whichever examples it happened to be batched with. Ioffe and Szegedy noticed the side effect: batch norm “also acts as a regularizer, in some cases eliminating the need for Dropout”. The same coupling is behind every catch in the rest of this part.

What happens at test time, when one example arrives?

Training sees batches. A deployed model often sees one input at a time, and a batch of one has its own mean, which is the example itself, and a spread of 0. Standardising it gives $(x - x)/\sqrt{0 + \epsilon} = 0$: every feature of every input erased.

The fix is to stop using the batch at test time. During training, each batch norm layer also keeps a running average of the means and variances it has seen, a smoothed history updated once per batch:

\[\hat\mu \leftarrow (1 - m)\,\hat\mu + m\,\mu_{\text{batch}}, \qquad \hat\sigma^2 \leftarrow (1 - m)\,\hat\sigma^2 + m\,s^2_{\text{batch}}\]

In words: keep 90% of the old estimate and mix in 10% of the new batch, with PyTorch’s default $m = 0.1$. PyTorch calls $m$ “momentum”, and its documentation warns that this is not the optimiser’s momentum: it is the weight on the new value, not on the old.

One detail interviewers like: the two variances differ. The batch is standardised with its variance divided by $B$, but the running average stores it divided by $B - 1$, the unbiased estimate. For feature 1’s batch of four that is 33.3 rather than 25Biased against unbiasedsquared distances: 49 + 1 + 1 + 49 = 100divided by B = 4: 25, used to standardise the batchdivided by B − 1 = 3: 33.3, stored in the running varianceIoffe and Szegedy’s Algorithm 2 uses the same B/(B − 1) correction for inference..

In the figure, feature 1 has mean 10 and spread 5 over all the data and arrives in batches of 32. The running mean starts at 0, as PyTorch’s does. After 60 batches the start still weighs 0.9⁶⁰ ≈ 0.0018How fast the start fadeseach update keeps 0.9 of the old valueafter 60 updates: 0.9⁶⁰ = e^(60 × ln 0.9) = e^(−6.32) ≈ 0.0018With m = 0.1 the average mostly remembers the last 20 or so batches., and the averages read 10.0 and a spread of 5.0. A test example of 13 is then standardised as 0.613 at test time(13 − 10) ÷ 5 = 3 ÷ 5 = 0.6The same answer whatever else is in the test batch., whatever else arrives with it.

So batch norm has two modes. In training mode (model.train() in PyTorch) it standardises with the batch’s own statistics and updates the running averages. In evaluation mode (model.eval()) it uses the stored averages and leaves them alone. It is then a fixed scale and shift, which can be folded into the previous layer’s weights: the paper’s Algorithm 2 calls it “a single linear transform”.

The classic bugs follow. Evaluate in training mode, and a prediction depends on what else is in the test batch; with a batch of one, every feature is 0. Worse, the running averages drift towards the test data, because training mode keeps updating them even when no gradient is computed. Fine-tune with tiny batches in training mode, and both the statistics the model trains with and the averages it will test with get noisy.

What goes wrong when batches are small?

Shrink the batch and batch norm degrades in steps. A batch of one is the test-time disaster again: every output 0.

A batch of two is stranger. For any two different numbers $a$ and $b$, the mean is $(a + b)/2$, both sit $\vert a - b\vert/2$ away from it, and that distance is also the spread. So the smaller always becomes −1 and the larger +1. 3 and 9 give −1 and +1; so do 3 and 3003 and 300mean: (3 + 300) ÷ 2 = 151.5spread: (300 − 3) ÷ 2 = 148.5(3 − 151.5) ÷ 148.5 = −1 and (300 − 151.5) ÷ 148.5 = +1. The output says nothing about the example, only whether its neighbour was bigger.

Bigger batches are not exempt, just milder. The batch mean is an estimate of the feature’s mean over all the data, and an average of $B$ independent draws misses by about $\sigma/\sqrt{B}$. For feature 1, with $\sigma = 5$, that is 2.5 at B = 4The batch mean’s error at B = 4σ ÷ √B = 5 ÷ √4 = 5 ÷ 2 = 2.5 and 0.88 at B = 32The batch mean’s error at B = 32σ ÷ √B = 5 ÷ √32 = 5 ÷ 5.66 ≈ 0.88Halving the error takes four times the batch.. With a batch of four, the mean that standardises everything can be off by half a spread.

That noise lands on every example. In the figure, an example whose feature 1 is 13 is placed in 40 random batches of four, and its output ranges from −1.35 to 1.58 depending on its three neighbours, against 0.6 measured on the whole data. The same input, a different output every step.

Detection and segmentation live here. They need high-resolution images, so models like Mask R-CNN train with 1 or 2 images per GPU and usually freeze batch norm into a fixed scale and shift. Wu and He measured the damage on ImageNet: a ResNet-50 on 8 GPUs, with batch norm computed per GPU, had 23.6% validation error at 32 images per GPU, 27.3% at 4 and 34.7% at 2 (their Figure 1 and Table 2).

Their replacement, group norm, stayed at 24.1% with 2 images per GPU, 10.6 points betterBatch norm against group norm at 2 images per GPU34.7% − 24.1% = 10.6 percentage points of error. It averages inside each image instead of down the batch, which is Part 4.

How do eight GPUs share one batch norm?

Multi-GPU training usually means data parallel: every GPU holds the whole model and runs its own slice of the batch. By default each GPU’s batch norm sees only its own slice. Wu and He put it plainly: the statistics are “computed for each worker, but not broadcast across workers”. Eight GPUs with 4 images each train with batch norm over 4, not 32.

SyncBatchNorm makes the GPUs agree. Each GPU adds up two sums per channel, $\sum x$ and $\sum x^2$, and counts its values. The GPUs add those sums together across the machine, and everyone computes the same statistics:

\[\mu = \frac{\sum x}{N}, \qquad \sigma^2 = \frac{\sum x^2}{N} - \mu^2\]

In words: the mean of the squares minus the square of the mean is the variance, so three numbers per channel are all a GPU needs from the others.

Split feature 1 over two GPUs. GPU A holds 3 and 9: Σx = 12, Σx² = 90GPU A’s sumsΣx = 3 + 9 = 12Σx² = 9 + 81 = 90count: 2. GPU B holds 11 and 17: Σx = 28, Σx² = 410GPU B’s sumsΣx = 11 + 17 = 28Σx² = 121 + 289 = 410count: 2. Together 40 and 500Adding the two GPUs’ sumsΣx: 12 + 28 = 40Σx²: 90 + 410 = 500count: 2 + 2 = 4, so the mean is 10 and the variance 25The variance from the sumsmean: 40 ÷ 4 = 10mean of the squares: 500 ÷ 4 = 125variance: 125 − 10² = 125 − 100 = 25, spread √25 = 5: the same as one GPU holding all four. Alone, each GPU would have had a batch of two and output −1 and +1.

What crosses the wire is tiny: three numbers per channel, 768 for a 256-channel layerNumbers shared by one layer3 per channel: Σx, Σx² and the count3 × 256 = 768 numbersNext to the gradients data parallel training already sends, this is nothing. The wait is what costs.. The cost is the wait. Every GPU has to reach this layer and hear from all the others before it can go on, once per batch norm layer on the way forward. The way back waits again, because the gradient formula above needs $\mathrm{mean}(g)$ and $\mathrm{mean}(g\,z)$ over the whole batch: two more sums per channel, added up across the GPUs.

PyTorch’s torch.nn.SyncBatchNorm does exactly this, with one twist: on the way forward it gathers each GPU’s mean, spread and count and combines them. That carries the same information as the raw sums and avoids subtracting two large numbers, a classic source of round-off error. The way back is an all-reduceAll-reduceA collective operation in which every GPU contributes some numbers and every GPU ends up with their sum. It is how data parallel training adds up gradients, and how SyncBatchNorm adds up its sums. of the two gradient sums. You switch it on with SyncBatchNorm.convert_sync_batchnorm(model) before wrapping the model for data parallel training.

Language models break the column in a different way. They generate one token at a time, and their sentences have different lengths, so “the same feature across the batch” stops being a sensible group at all. Part 3 averages along the row instead.

Rapid fire: can you do these from memory?

  1. A batch norm layer sees one feature with values 10, 6, 10, 6 across the batch. What does it output, and what are its mean and spread?
  2. Why does multiplying a batch-normalised layer's weights by 10 leave its output unchanged, and what happens to the gradient on those weights?
  3. Write the gradient through batch norm for one feature. Which two parts of the incoming gradient does it remove, and why do the input gradients sum to zero?
  4. Your model gives a different prediction for the same image depending on what else is in the test batch. What did you forget?
  5. PyTorch's batch norm momentum defaults to 0.1. Is that the weight on the new batch or on the old running average, and which variance estimate does the running average store?
  6. Any two different numbers in a batch of two come out as −1 and +1. Show why in two lines.
  7. A detector trains with 2 images per GPU on 8 GPUs. What does plain batch norm compute its statistics over, and what are two ways out?
  8. Under SyncBatchNorm, which numbers per channel cross between GPUs on the way forward and on the way back, and why is the cost a wait rather than bandwidth?

Where can you go deeper?

  • Sergey Ioffe and Christian Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, ICML 2015, arXiv:1502.03167. §3 has the algorithm, §3.1 and Algorithm 2 the inference statistics (the unbiased variance and the single linear transform), §3.2 the convolutional version, §3.3 the scale argument, and §4.2 the 14 times result.
  • Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas and Aleksander Madry, How Does Batch Normalization Help Optimization?, NeurIPS 2018, arXiv:1805.11604. §2.1 is the noise experiment; §3 the smoothness argument.
  • Yuxin Wu and Kaiming He, Group Normalization, ECCV 2018, arXiv:1803.08494. Figure 1 and Table 2 are the small-batch numbers quoted above.
  • Dive into Deep Learning, §8.5, Batch Normalization: a from-scratch implementation with the training and prediction modes side by side.
  • PyTorch documentation: torch.nn.BatchNorm2d for the momentum rule and the biased and unbiased variances, and torch.nn.SyncBatchNorm for the multi-GPU version.