Why do images need something between the column and the row?
An object detector trains on big photographs, and big photographs eat memory. Wu and He point out, in the paper that introduced group norm, that the Fast R-CNN, Faster R-CNN and Mask R-CNN frameworks train with 1 or 2 images per GPU for exactly that reason. Two images is not a batch anyone should take an average over.
Here is where this series stands. A deep network’s numbers drift: each layer rescales them a little, and over many layers that compounds. The fix is to re-measure at every layer: subtract a mean, divide by a standard deviation, then give the layer a learned scale $\gamma$ and shift $\beta$ so it can choose its own size again. The open question is always the same one: which numbers share that mean?
The 30-second version
Something in between: group norm. Batch norm's mean for each channel would come from just 2 images, so it jumps around from step to step; Wu and He measured ResNet-50's ImageNet error rising from 23.6% at 32 images per GPU to 34.7% at 2. Layer norm needs no batch, but it measures every channel of an image with one mean and one spread, although the channels are different detectors. Group norm averages one image over a small group of neighbouring channels: no batch, and no lumping of everything together. In the same runs it held 24.1% at both 32 and 2 images per GPU.Two answers so far. Batch norm averages each feature down the batch, across the examples, so it needs a big batch (Part 2). Layer norm averages each example across its own features, so it needs no batch at all (Part 3). A detector with 2 images per GPU fits neither well.
Batch norm’s mean from 2 images is mostly noise. Wu and He trained a ResNet-50 on ImageNet with batch norm at 32, 16, 8, 4 and 2 images per GPU, and its validation error went from 23.6% to 34.7% (their Figure 1). Group norm, where this part is heading, stayed between 24.0% and 24.2% at every batch size and ended at 24.1%, 10.6 points10.6 points, worked outbatch norm at 2 images per GPU: 34.7% errorgroup norm at 2 images per GPU: 24.1% error34.7 − 24.1 = 10.6 percentage pointsWu and He’s Table 2. At 32 images per GPU group norm is 0.5 points behind batch norm instead. better at 2 images.
That is why detectors “freeze” batch norm. They keep the statistics from pre-training as a fixed linear layer and never normalise with their own images at all.
Layer norm needs no batch, but it treats an image like a token. An image’s numbers come in channels, and each channel is a different detector: one for vertical edges, one for a colour, one for a texture, each with its own typical size.
Layer norm pools all of them into one mean and one spread, so the loudest channels set the ruler for the quiet ones. Even at a comfortable 32 images per GPU it trails batch norm, 25.3% against 23.6%.
So this part is about the middle. First the picture: with images, the flat grid of examples and features becomes a cube.
What does a batch of images look like as numbers?
A convolutional layer slides each of its filters across the image and writes down how strongly the filter responds at every position. One filter’s responses form a grid of numbers laid out like the image: a feature map. Each filter is a channel, and a layer with 64 filters hands the next layer 64 feature maps. The raw image is the same thing with three channels: red, green and blue.
So a batch of images has four axes, not two: $N$ images, $C$ channels, and a feature map of $H$ rows by $W$ columns of pixels. Earlier parts drew a batch as rows of examples and columns of features. The pixels add a third direction, and the grid becomes a cube.
Real cubes are huge. This one is small enough to read: 2 images, 4 channels, and a 2 × 2 map per channel, so 4 pixels and 32 numbers32 numbers, worked out2 images × 4 channels = 8 feature mapseach map is 2 × 2 = 4 pixels8 × 4 = 32 numbersIn general N × C × H × W..
Read each 2 × 2 map row by row and it becomes a row of 4. Then each image is a small grid, a row per channel and a column per pixel, which is how Wu and He’s Figure 2 draws the cube flat.
Look at the rows of image 1. Channel 1 reads 7, 15, 7, 15: a vertical edge, bright on the right. Channel 2 swings between 9 and 1. Channel 4 barely moves, 13 and 15 everywhere. Those differences are what the four norms disagree about.
Which cells share a mean in each of the four norms?
Every norm in this part does the same two moves to a number $x$. Standardise it with the mean $\mu$ and standard deviation $\sigma$ of some set of numbers, then rescale it with a learned $\gamma$ and $\beta$:
\[y = \gamma\,\frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta\]In words: count how many standard deviations $x$ sits from the mean of its set, then stretch and shift to taste. The $\epsilon$ is a tiny constant that keeps the division safe when a set has no spread at all.
The four norms differ in one thing only: which set. Wu and He write each of them as a single line saying which cells share it. Follow one cell, the red 7: image 1, channel 1, pixel 1.
Batch norm takes one channel across every image and every pixel. That is how Ioffe and Szegedy defined it for convolutional layers: all positions of a feature map share one mean, so the images and the pixels are pooled together. Here that is 8 numbers8 numbers, worked outone channel, in every image, at every pixel2 images × 4 pixels = 8 numbersIn general N × H × W: the batch size is in it., channel 1 in both images, with mean 14 and sd 5mean 14, sd 5, worked outchannel 1, both images: 7, 15, 7, 15, 13, 21, 13, 21sum 112, so the mean is 112 ÷ 8 = 14gaps from 14: −7, 1, −7, 1, −1, 7, −1, 7squares: 49 + 1 + 49 + 1 + 1 + 49 + 1 + 49 = 200variance 200 ÷ 8 = 25, so the sd is √25 = 5, so the 7 becomes −1.4−1.4, worked out(7 − 14) ÷ 5 = −7 ÷ 5 = −1.41.4 standard deviations below its channel’s mean across the batch..
Layer norm takes one image, every channel and every pixel: 16 numbers16 numbers, worked outone image: 4 channels × 4 pixels = 16 numbersIn general C × H × W: no batch in it.. Image 1 has mean 10 and sd 5mean 10, sd 5, worked outchannel sums: 44 + 20 + 40 + 56 = 160, so the mean is 160 ÷ 16 = 10squared gaps from 10, channel by channel: 68 + 164 + 100 + 68 = 400variance 400 ÷ 16 = 25, so the sd is 5, so the 7 becomes −0.6−0.6, worked out(7 − 10) ÷ 5 = −3 ÷ 5 = −0.6.
Instance norm takes one image and one channel: just its 4 pixels4 pixels, worked outone image, one channel: a 2 × 2 map2 × 2 = 4 numbersIn general H × W.. Channel 1 of image 1 is 7, 15, 7, 15, with mean 11 and sd 4mean 11, sd 4, worked out(7 + 15 + 7 + 15) ÷ 4 = 44 ÷ 4 = 11gaps from 11: −4, 4, −4, 4, each squared 16variance 64 ÷ 4 = 16, so the sd is 4, so the 7 becomes −1−1, worked out(7 − 11) ÷ 4 = −4 ÷ 4 = −1.
Group norm takes one image and a group of neighbouring channels. With 2 groups of 2, the 7 shares a mean with the rest of channels 1 and 2 of image 1: 8 numbers again8 numbers, worked out4 channels in 2 groups: 4 ÷ 2 = 2 channels per group2 channels × 4 pixels = 8 numbersIn general (C ÷ G) × H × W, with G groups., with mean 8 and sd 5mean 8, sd 5, worked outchannels 1 and 2 of image 1: 7, 15, 7, 15, 9, 9, 1, 1sum 64, so the mean is 64 ÷ 8 = 8gaps from 8: −1, 7, −1, 7, 1, 1, −7, −7squares: 1 + 49 + 1 + 49 + 1 + 1 + 49 + 49 = 200variance 200 ÷ 8 = 25, so the sd is 5, so it becomes −0.2−0.2, worked out(7 − 8) ÷ 5 = −1 ÷ 5 = −0.2.
| norm | averages over | numbers per mean here | needs the batch? | the red 7 becomes |
|---|---|---|---|---|
| batch norm | one channel, every image, every pixel | 8 | yes | −1.4 |
| layer norm | one image, every channel, every pixel | 16 | no | −0.6 |
| instance norm | one image, one channel | 4 | no | −1 |
| group norm, 2 groups | one image, a group of channels | 8 | no | −0.2 |
In general a mean covers $NHW$ numbers under batch norm, $CHW$ under layer norm, $HW$ under instance norm and $(C/G)HW$ under group norm with $G$ groups. Only the first has $N$ in it. That one letter is the whole small-batch story: shrink the batch and only batch norm’s statistics get worse.
All four learn their $\gamma$ and $\beta$ one per channel (Wu and He’s equation 6). So whichever set a channel was measured with, it gets its own scale and shift back afterwards.
How does group norm choose its groups?
By position. Group norm takes a count $G$ and cuts each image’s $C$ channels into $G$ runs of $C/G$ neighbours: the first $C/G$ channels form group 1, the next $C/G$ group 2, and so on. Each group in each image gets its own mean and standard deviation, and nothing is ever shared between images.
The dial has two ends, and both are norms you already know. $G = 1$ puts every channel in one group: one mean per image over everything, which is layer norm. $G = C$ gives every channel a group of its own: one mean per image and channel, which is instance norm. Wu and He say exactly this: the two extreme cases of group norm are layer norm and instance norm.
Why should neighbouring channels share a mean at all? Wu and He’s argument is that a layer’s channels are not independent: they come in families. Their example is the first layer of a network, where a filter and its mirror image respond to natural images in much the same way, so their two channels can be normalised together.
Higher up the families are harder to name, but they list frequency, shape, illumination and texture as things that tie channels together. The idea is old. Classical vision features such as SIFT and HOG are group-wise by design: HOG normalises each small cell’s histogram of edge directions on its own. Group norm puts that habit back inside a deep network.
How many groups? Wu and He’s default is 32, and in their sweep it was also the best. On ResNet-50 at 32 images per GPU, 32 groups gave 24.1% error, and every group count from 2 to 64 landed within 0.6 points0.6 points, worked outbest: 32 groups, 24.1% errorworst of 2 to 64 groups: 2 groups, 24.7%24.7 − 24.1 = 0.6 pointsWu and He’s Table 3, top panel. of it. One group, which is layer norm, gave 25.3%.
They also tried a fixed number of channels per group. One channel per group, which is instance norm, was the worst at 28.4%, and even 2 channels per group brought that down to 25.6%. So the usual recipe is 32 groups whatever the width: a layer of 256 channels gets groups of 8groups of 8, worked out256 channels ÷ 32 groups = 8 channels per group.
There is one place where the dial bites. At the end of a classifier, global average poolingGlobal average poolingReplacing each whole feature map by its average, so every channel becomes a single number. Classifiers such as ResNets do it just before the last layer, which then sees one number per channel, whatever the image size. replaces each feature map by its mean: a map of one pixel. Image 1’s four maps become 11, 5, 10 and 14the pooled maps, worked outchannel 1: (7 + 15 + 7 + 15) ÷ 4 = 11channel 2: (9 + 9 + 1 + 1) ÷ 4 = 5channel 3: (3 + 9 + 11 + 17) ÷ 4 = 10channel 4: (13 + 15 + 15 + 13) ÷ 4 = 14.
Put instance norm there and each mean covers one number. A number minus itself is 0, and the $\epsilon$ keeps 0 divided by almost nothing at 0. Every channel of every image comes out 0, whatever the picture showed.
A group of two channels is barely better. Two different numbers always become −1 and +1, because their mean sits halfway between them and their standard deviation is half the gap. 11 and 5 have mean 8 and sd 3, so they become +1 and −1+1 and −1, worked outmean (11 + 5) ÷ 2 = 8gaps +3 and −3: variance (9 + 9) ÷ 2 = 9, sd 3(11 − 8) ÷ 3 = +1 and (5 − 8) ÷ 3 = −1Any two different numbers: the mean is halfway, the sd half the gap, so always −1 and +1.. It is the trap of batch norm with a batch of 2, moved from the batch axis to the channel axis.
What does instance norm wipe out of an image?
Instance norm measures each image, and each of its channels, on its own. The mean of a channel is its overall level in that image, and the standard deviation is how much it varies across the image. On the raw pixels of a grayscale photograph those two numbers have everyday names: the mean is the brightness, and the standard deviation is the contrast.
Standardising subtracts one and divides by the other, so both are gone. Take a 4 × 4 grayscale image with light falling from the top-left corner: five pixels of 12, five of 4 and six of 0. Its mean is 5 and its sd is 5mean 5, sd 5, worked outsum: 5 × 12 + 5 × 4 + 6 × 0 = 60 + 20 + 0 = 80mean 80 ÷ 16 = 5gaps 7, −1 and −5: squares 5 × 49 + 5 × 1 + 6 × 25 = 400variance 400 ÷ 16 = 25, so the sd is 5.
Now a copy with every pixel turned into $3x + 20$: brighter by 20, with every gap three times wider. Its mean is 35 and its sd is 15mean 35, sd 15, worked outthe copy’s pixels: 56, 32 and 20mean 3 × 5 + 20 = 35adding 20 leaves the gaps alone; multiplying by 3 triples them: sd 3 × 5 = 15check: 5 × 21² + 5 × 3² + 6 × 15² = 3,600, and 3,600 ÷ 16 = 225 = 15².
After instance norm the two are one grid. 12 and its copy 56 both become 1.4the two grids, worked outimage: (12 − 5) ÷ 5 = 1.4, (4 − 5) ÷ 5 = −0.2, (0 − 5) ÷ 5 = −1copy: (56 − 35) ÷ 15 = 1.4, (32 − 35) ÷ 15 = −0.2, (20 − 35) ÷ 15 = −1, 4 and 32 both become −0.2, and 0 and 20 both become −1. The algebra says it for any $a > 0$ and any $b$:
\[\frac{(ax + b) - (a\mu + b)}{a\sigma} = \frac{x - \mu}{\sigma}\]Adding $b$ moves the mean by $b$, which is subtracted. Multiplying by $a$ multiplies the standard deviation by $a$, which is divided out.
That is exactly what fast style transfer wanted. Ulyanov, Vedaldi and Lempitsky noticed that a stylised image’s contrast should come from the style image, not from the photo being stylised, so the generator network should throw the photo’s contrast away. Swapping its batch norm layers for instance norm, and keeping them at test time, did that in one move. The results improved markedly, and the paper’s title calls instance norm the missing ingredient for fast stylization.
Deeper in a network the same move acts on feature channels rather than pixels, and there it removes more than contrast. Huang and Belongie argued that the mean and spread of each feature channel carry an image’s style: a painting full of one kind of brushstroke has a high average on the channel that detects it. On their reading, instance norm is style normalisation.
If two numbers per channel are the style, you can hand an image a different one. Instance norm ends, like every norm here, with a learned $\gamma$ and $\beta$, the same for every image. Huang and Belongie’s adaptive instance norm, AdaIN, learns no $\gamma$ or $\beta$ at all: it reads them off a second image. $\gamma$ is the style’s standard deviation and $\beta$ the style’s mean, channel by channel:
\[\mathrm{AdaIN}(x, y) = \sigma(y)\,\frac{x - \mu(x)}{\sigma(x)} + \mu(y)\]In words: strip the content $x$ of its own mean and spread, then dress it in the mean and spread of the style $y$.
On the 4 × 4 image: normalised, it is 1.4, −0.2 and −1. Take a checkerboard of 30 and 50 as the style, with mean 40 and sd 10mean 40, sd 10, worked outeight 30s and eight 50s: mean (30 + 50) ÷ 2 = 40every pixel sits 10 from 40, so the variance is 100 and the sd 10. AdaIN turns 1.4 into 10 × 1.4 + 40 = 54, and the rest into 38 and 3038 and 30, worked out10 × (−0.2) + 40 = −2 + 40 = 3810 × (−1) + 40 = −10 + 40 = 30The result’s mean is 10 × 0 + 40 = 40 and its sd 10 × 1 = 10: the style’s..
The result keeps the content’s pattern, light from the top-left, with the style’s brightness and contrast. The checkerboard itself does not come across: AdaIN moves two numbers per channel, not a picture.
In Huang and Belongie’s network this happens on every channel of a fixed VGG-19 encoder’s features, where those two numbers carry textures and strokes rather than brightness, and a trained decoder turns the result back into an image. One network, any style image, one forward pass.
Why do diffusion models still use group norm?
Stable Diffusion’s denoiser is a U-NetU-NetA convolutional network that shrinks its feature maps step by step and then grows them back to full size, with shortcuts carrying each resolution across. In a diffusion model it takes a noisy image and predicts the noise; Stable Diffusion’s also has attention layers inside., and every normalisation layer in its ResNet blocks is group norm with 32 groups. In Hugging Face’s diffusers library, UNet2DConditionModel sets norm_num_groups=32 by default, and the published configs of Stable Diffusion 1.5, 2.1 and XL all use 32. The original CompVis code builds every norm as GroupNorm32(32, channels).
Its blocks are 320, 640 and 1,280 channels wide, so 32 groups means 10, 20 and 40 channels per groupchannels per group, worked out320 ÷ 32 = 10640 ÷ 32 = 201,280 ÷ 32 = 40.
Why not batch norm? The detector’s reason, plus one of diffusion’s own. The U-Net works on large feature maps, so each image’s activations are big and few images fit on one GPU. And a diffusion model is used differently from how it is trained: training sees random batches of images at random noise levels, while sampling walks one image through a chain of denoising steps.
Batch norm would have to swap its batch statistics for running averages between those two settings. Group norm computes the same thing in both, from the image in front of it.
The timestep has to get in somewhere, and in diffusers’ ResnetBlock2D it enters right beside a group norm. Stable Diffusion uses the default mode: the timestep’s embedding is projected to one number per channel and added to the features just before the block’s second group norm.
The other mode, scale_shift, normalises first, then multiplies by one plus a predicted scale and adds a predicted shift. That is AdaIN’s move with the timestep in place of a style image, and OpenAI’s guided-diffusion models switch it on (use_scale_shift_norm).
The default mode leans on the choice of norm. A number added to a whole channel is exactly what instance norm subtracts, so under instance norm the timestep would vanish without a trace. Group norm subtracts one mean per group, so only the part of the shift that a group’s channels share is lost, and the differences between them survive.
One more detail is worth knowing. The CompVis GroupNorm32 casts its input to float32 before normalising and back afterwards, so the statistics are computed in full precision even when the rest of the network runs in half.
Diffusion transformers replaced the U-Net, and with it group norm: they normalise each token with layer norm and predict its $\gamma$ and $\beta$ from the timestep and the condition, such as the prompt: adaptive layer norm. That, and what the 2026 models settled on, is Part 5.
Rapid fire: can you do these from memory?
- A batch holds 8 images with 64 channels of 32 × 32 pixels. How many numbers share one mean under batch norm, layer norm, instance norm, and group norm with 32 groups?
- Why does batch norm care how many images sit on one GPU, when the other three do not?
- Group norm with G = 1, and with G = C: which norms are these?
- After global average pooling, what does instance norm output, and what does a group of two channels output?
- Every pixel of an image is doubled and brightened by 50. What changes after instance norm?
- Where do AdaIN's γ and β come from, and which of the style's statistics does each one carry?
- Give two reasons Stable Diffusion's U-Net uses group norm rather than batch norm.
- The timestep is added to each channel just before a norm. What would happen if that norm were instance norm?
Where can you go deeper?
- Yuxin Wu and Kaiming He, Group Normalization, ECCV 2018, arXiv 1803.08494. Figure 1 and Table 2 for batch size, Figure 2 for the four cubes, §3 for the definitions and the two extreme cases, Table 1 for all four norms at 32 images per GPU, Table 3 for the group sweep, and the introduction for detectors at 1 or 2 images per GPU.
- Dmitry Ulyanov, Andrea Vedaldi and Victor Lempitsky, Instance Normalization: The Missing Ingredient for Fast Stylization, 2016, arXiv 1607.08022. Equation 3 and the contrast argument around it.
- Xun Huang and Serge Belongie, Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization, ICCV 2017, arXiv 1703.06868. §4 reads instance norm as style normalisation; §5, equation 8, is AdaIN.
- Sergey Ioffe and Christian Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, ICML 2015, arXiv 1502.03167. §3.2 is the convolutional version: one mean per feature map over the batch and every location.
- The diffusers source on GitHub:
src/diffusers/models/unets/unet_2d_condition.pyfor thenorm_num_groups=32default, andsrc/diffusers/models/resnet.pyforResnetBlock2Dand its two timestep modes. Theunet/config.jsonfiles of Stable Diffusion 1.5 and SDXL base 1.0 on Hugging Face (2.1’s checked on the sd2-community mirror), andGroupNorm32in CompVis’sldm/modules/diffusionmodules/util.py.