Where should the norm sit in a transformer block?
Four parts in, here is the whole story so far. A deep network’s numbers drift: 80 layers that each widen their output by 10% leave it 2,048 times1.1⁸⁰, worked outln 1.1 ≈ 0.095380 × 0.0953 ≈ 7.62e to the 7.62 ≈ 2,048At 0.9 per layer the same 80 layers shrink it to about 0.0002. wider at the top. The fix is to re-measure at every layer: subtract a mean, divide by a standard deviation, then give the layer back a scale $\gamma$ and a shift $\beta$ of its own.
\[y = \gamma\,\frac{x - \mu}{\sigma} + \beta\]The norms differ only in which numbers share one mean. Batch norm averages down the column: one feature across the batch (Part 2). Layer norm and RMSNorm average along the row: one token across its own features (Part 3). Group norm and instance norm average inside one image, over a group of channels or one channel’s pixels (Part 4).
This last part asks what models actually do with all of that as of October 2026. The first question is not which norm, but where it goes.
The 30-second version
The one with the norm before each sublayer, pre-norm. Its residual stream is a plain running sum, so the gradient has a road from the top of the network down to the first layer with no norm on it, and it arrives at a sensible size from the first step. Post-norm puts a norm after every addition, two per layer, so 160 of them sit right on that road. Xiong et al. (2020) showed that at initialisation this leaves large gradients on the parameters near the output, so a full learning rate on step one destabilises training, and the warm-up is the patch. Pre-norm's price is a stream that keeps growing with depth, which is why every pre-norm model ends with a final norm.The 2017 transformer built each layer from two sublayers, attention and the MLP (the small two-layer network after it), and wired both into a residual streamResidual streamThe running sum that carries a token through the network. Every sublayer reads it, computes something, and adds its output back, so the stream after the last layer is the embedding plus every sublayer’s contribution.. Call the stream $x$ and a sublayer $F$. The question is where the norm goes.
Post-norm, the 2017 choice, normalises after adding $F$’s output back. Pre-norm, the choice of almost every model since, normalises only the copy that $F$ reads, and leaves the stream alone:
\[\text{post-norm: } x \leftarrow \mathrm{Norm}\big(x + F(x)\big) \qquad\qquad \text{pre-norm: } x \leftarrow x + F\big(\mathrm{Norm}(x)\big)\]Why does the place matter? Follow the gradient back down. In pre-norm the stream is a plain sum: at the top it is the stream at any lower layer plus every sublayer output in between. So its derivative with respect to that lower layer holds an identity:
\[x_{\text{top}} = x_{l} + \sum_{\text{sublayers above } l} F(\dots) \quad\Longrightarrow\quad \frac{\partial x_{\text{top}}}{\partial x_{l}} = I + \sum \frac{\partial F(\dots)}{\partial x_{l}}\]That $I$ is the residual highway. Whatever the sublayers do, the gradient from the top reaches layer 1 along a road with no norm on it, below the final one.
In post-norm every addition is followed by a norm on the stream itself. An 80-layer model has two sublayers per layer, so the gradient crosses 160 norms160 norms, worked outeach layer has 2 sublayers: attention and the MLPpost-norm puts a norm after each addition, on the stream2 × 80 = 160Pre-norm’s stream crosses only the final norm. on its way down. Each one multiplies it by its own derivative, which divides by the spread that norm measured. That is 160 factors in a row, and nothing makes them multiply to 1.
Xiong et al. (2020) worked out what this does at initialisation. In post-norm, the expected gradients of the parameters near the output layer are large, so a full learning rate on the first steps makes training unstable. The warm-upLearning-rate warm-upStarting the learning rate near zero and raising it to its full value over the first steps of training, so that early updates stay small while the gradients are still badly scaled. was the patch. Pre-norm’s gradients are well-behaved from the start, and they showed the warm-up can go.
Pre-norm’s price is the stream itself. Nothing ever re-measures it, so it grows as sublayers add to it. Idealise: the embeddings have variance 1, and every sublayer adds an independent output of variance 1. Variances of independent things add, so after 160 sublayers the stream’s variance is 161161, worked outthe embeddings: variance 1160 sublayers, each adding an independent output of variance 1variances of independent terms add: 1 + 160 × 1 = 161 and its spread about 12.712.7, worked outspread = √variance√161 ≈ 12.69After layer l the spread is √(1 + 2l): layer 40 gives √81 = 9..
That is an idealisation, not a measurement, but the direction is right. The sublayers do not mind, since each one reads a normalised copy. What does mind is the output head, which turns the stream into scores for the next token. So every pre-norm model ends with one more norm before the head, the final norm, which divides the stream by its spread.
The growth has a second cost: the 160th sublayer adds a spread of 1 to a stream of 12.7, so late layers have a quieter voice. That is one reason post-norm keeps coming back in research. Keel (Chen and Wei, January 2026) pairs post-norm with a highway-style connection and trains at “depths exceeding 1000 layers”. Transformer Part 4 has more on this trade.
What do 2026 language models put around each sublayer?
We read the code or config of about 35 open checkpoints released from 2024 to October 2026. Every one of them uses RMSNorm: divide the row by its root mean square, with no mean subtracted. And every one ends with a final norm before the output head.
Placement is where they differ. Pre-norm is the majority: Llama, Qwen, DeepSeek, Kimi, GLM, MiniMax, Mistral, GPT-OSS and Phi are all pure pre-norm. GPT-OSS’s model card says it in one line: “Similar to GPT-2 we use Pre-LN placement.”
A minority also normalise what the sublayer returns. OLMo 2 and 3 normalise only the output, $x + \mathrm{RMSNorm}(F(x))$. Gemma 2, 3 and 4, and Meta’s Muse Spark Open, normalise both sides, a sandwich: $x + \mathrm{Norm}(F(\mathrm{Norm}(x)))$. Kim et al. studied the sandwich as Peri-LN at ICML 2025.
Why normalise the output too? Read it against the last section’s idealisation. An RMSNorm on the output with gain 1 hands the stream a vector of RMS exactly 1, however large the sublayer’s own weights grow. The output norm turns “each sublayer adds variance 1” from an assumption into a rule.
| model (on Hugging Face) | norm | placement | QK-norm | note |
|---|---|---|---|---|
| Gemma 2 (June 2024) | RMSNorm, gain 1 + w | sandwich | no: logits soft-capped at 50 | “normalize the input and output of each transformer sub-layer” |
| Llama 3.1 (July 2024) | RMSNorm | pre-norm | no | the plain recipe |
| OLMo 2 (late 2024) | RMSNorm | after the sublayer only | yes, across all heads | switched from layer norm without a gain after ablations showed no difference |
| DeepSeek-V3 (December 2024) | RMSNorm | pre-norm | no (MLA): RMSNorm on the compressed latents | recomputes RMSNorm in the backward pass |
| Gemma 3 (March 2025) | RMSNorm, gain 1 + w | sandwich | yes, per head | replaced Gemma 2’s soft-capping |
| Qwen3 (April 2025) | RMSNorm | pre-norm | yes, per head | added “to ensure stable training” |
| Kimi K2 (July 2025) | RMSNorm | pre-norm | no (MLA): QK-Clip | 15.5 trillion tokens “with zero loss spike” |
| GPT-OSS (August 2025) | RMSNorm | pre-norm | no | attention sinks instead |
| Qwen3-Next (September 2025) | RMSNorm, gain 1 + w | pre-norm | yes, per head | “zero-centered and weight-decayed layernorm” |
| Gemma 4 (March 2026) | RMSNorm, gain starts at 1 | sandwich | yes, per head, plus a norm on the values | dropped the 1 + w gain |
| DeepSeek-V4 (April 2026) | RMSNorm | pre-norm, inside four residual streams | RMS on each query head, RMSNorm on the shared key-value head | norm weights kept in fp32 |
| Muse Spark Open (October 2026) | RMSNorm | sandwich, plus a residual gate | yes, without a learned gain, times a fixed scale | the gate keeps the stream’s variance fixed |
Dates are when the weights appeared on Hugging Face, which is close to, not always equal to, the release.
Look at the norm column. A norm’s gain $\gamma$ starts at 1, and by convention it is excluded from weight decayWeight decayA small pull of every weight toward 0 at each optimiser step, which keeps weights from growing without need. Applied to a norm’s gain, it would pull the gain toward 0 and quietly turn its layer down., because a gain pulled toward 0 would silence its layer.
Gemma 2 and 3, Qwen3-Next and the later Qwen models, MiniMax-M3 and Muse Spark store the gain as $w$ and apply $1 + w$. Since $w$ starts at 0, the gain still starts at 1, but weight decay now pulls the gain toward 1 instead of 0. Qwen3-Next’s model card calls its norms “zero-centered and weight-decayed”. Gemma 4 went back to gains that start at 1.
Why RMSNorm everywhere? It needs one reduction, the sum of squares, where layer norm needs the mean first and then the spread around it. And in practice it trains as well. OLMo 2’s report is blunt about its switch from layer norm without a gain: “Our ablations show no difference between the two, so we switch back to RMSNorm.” Part 3 has the comparison on our grid.
Why normalise the queries and keys as well?
Attention scores each key by its dot product with the query, the logit $q \cdot k$, and a softmax turns the scores into weights that add up to 1. Take three keys with logits 2, 1 and 0. Exponentiate and divide by the total, and the weights come out 0.665, 0.245 and 0.090softmax of 2, 1, 0, worked oute² ≈ 7.389, e¹ ≈ 2.718, e⁰ = 1total ≈ 11.1077.389 ÷ 11.107 ≈ 0.665, 2.718 ÷ 11.107 ≈ 0.245, 1 ÷ 11.107 ≈ 0.090.
Now let training grow the query and key weights. The logits grow with them. At three times the size, 6, 3 and 0, the top key takes 0.9500.950, worked oute⁶ ≈ 403.4, e³ ≈ 20.09, e⁰ = 1total ≈ 424.5403.4 ÷ 424.5 ≈ 0.950 of the attention. At ten times, 20, 10 and 0, it takes 0.999950.99995, worked outdivide top and bottom by e²⁰: the top share is 1 ÷ (1 + e⁻¹⁰ + e⁻²⁰)e⁻¹⁰ ≈ 0.0000454, e⁻²⁰ ≈ 0.0000000021 ÷ 1.0000454 ≈ 0.99995. The attention has gone one-hot.
Why is one-hot bad? The gradient that reaches a logit through the softmax is scaled by the slope of its weight, and for the top key that slope is
\[\frac{\partial p}{\partial z} = p\,(1 - p)\]At $p = 0.665$ the slope is 0.2230.223, worked outp = 0.665, so 1 − p = 0.3350.665 × 0.335 ≈ 0.223The largest a slope p(1 − p) can be is 0.25, at p = 0.5.. At three times it is 0.0470.047, worked outp = 0.950, so 1 − p = 0.0500.950 × 0.050 ≈ 0.047, and at ten times 0.000050.00005, worked outp = 0.99995, so 1 − p ≈ 0.00004540.99995 × 0.0000454 ≈ 0.000050.223 ÷ 0.0000454 ≈ 4,900: almost five thousand times less gradient.. A weight stuck at 1 cannot be moved by its logit, so almost nothing flows back.
This is not a thought experiment. ViT-22B (Dehghani et al., 2023) saw training diverge in models around 8B parameters, “caused by extremely large values in attention logits, which lead to (almost one-hot) attention weights with near-zero entropy”. SD3 (Esser et al., 2024) saw its loss diverge at high resolutions under mixed precision, in the last transformer blocks. Wortsman et al. (2024) reproduced the logit growth in small models trained at high learning rates.
The fix is a norm in a new place. QK-norm normalises $q$ and $k$ before the dot product, per head, with RMSNorm and a learned gain. A factor $c$ on the weights then cancels, because the RMS carries the same factor:
\[\mathrm{RMSNorm}(c\,\mathbf q) = \frac{c\,\mathbf q}{\mathrm{RMS}(c\,\mathbf q)} = \frac{c\,\mathbf q}{c\,\mathrm{RMS}(\mathbf q)} = \mathrm{RMSNorm}(\mathbf q)\]It also caps the logit. With gains of 1, a normalised head of 128 numbers has length √128, so the usual scaled logit $q \cdot k / \sqrt{d}$ can be at most about 11.3the cap, worked outRMS 1 over 128 numbers: length √128Cauchy-Schwarz: q · k ≤ √128 × √128 = 128divided by √128: at most √128 ≈ 11.3. To sharpen attention past that, the model has to grow the gains themselves, on purpose.
Henry et al. proposed normalising queries and keys in 2020. From 2025 on it is standard in models without MLA: Qwen3 onward, Gemma 3 and 4, OLMo 2 and 3, GLM-4.5 and 4.6, MiniMax-M2 and M3, Llama 4 Scout (without a learned gain, and only on its RoPE layers), Muse Spark and DeepSeek-V4. Gemma 3 replaced Gemma 2’s soft-capping of the logits with it, and Qwen3’s report says it added QK-norm “to ensure stable training”.
Models built on MLAMulti-head latent attentionDeepSeek’s attention variant: each token’s keys and values are stored as one small compressed vector, a latent, and the full keys are never built during inference. It shrinks the KV cache. cannot use it: DeepSeek-V3, the Kimi K2 family, GLM-5 and Mistral Small 4. In the Kimi K2 report’s words, QK-norm “is not applicable to multi-head latent attention (MLA), because its Key matrices are not fully materialized during inference”. These models put RMSNorm on the compressed latents instead.
Kimi K2 added a clip: QK-Clip. It watches each head’s largest logit in the batch, and when that passes τ = 100, it scales down that head’s query and key weights after the optimiser step, leaving the step itself untouched. It works per head because “only a small subset of heads exhibit exploding logits”.
Without it, a test model with 9B active and 53B total parameters, trained with the Muon optimiser, saw its largest logits “quickly exceed a magnitude of 1000”. K2 itself trained on 15.5 trillion tokens with zero loss spikes.
How does a diffusion model steer an image through its norm?
A diffusion model removes noise from an image over many steps, and one network does every step. So it has to be told which step it is on, the timestep (how much noise is left), along with the class or the prompt. The standard way to tell it goes through the norm.
Recall $\gamma$ and $\beta$: after standardising, every norm hands the layer a learned scale and shift. Adaptive layer norm, adaLN, drops the learned ones and predicts them instead. A small layer reads the timestep and the condition and outputs a scale and a shift; the block normalises with no $\gamma$ or $\beta$ of its own, then applies
\[(1 + \text{scale})\cdot z + \text{shift}\]Take row 1 of the grid as one token, 3, 10, −2, 6, 0, 1. Layer norm turns it into 0, 1.75, −1.25, 0.75, −0.75, −0.5layer norm of row 1, worked outmean (3 + 10 − 2 + 6 + 0 + 1) ÷ 6 = 3squared deviations 0 + 49 + 25 + 9 + 9 + 4 = 96; ÷ 6 = 16; spread 4(x − 3) ÷ 4 for each number. With illustrative predictions of scale 1 and shift 0.5, it becomes 0.5, 4, −2, 2, −1, −0.5the modulated row, worked out(1 + 1) × z + 0.5 = 2z + 0.52z = 0, 3.5, −2.5, 1.5, −1.5, −1add 0.5: 0.5, 4, −2, 2, −1, −0.5. At another timestep, other numbers: one layer, a different behaviour at every step.
DiT (Peebles and Xie, ICCV 2023) added a third number, a gate on the sublayer’s output before it joins the stream, $x + \text{gate} \times F(\dots)$. That makes six vectors per block, a scale, a shift and a gate for the attention and again for the MLP, all predicted from the timestep and the class.
adaLN-Zero is the initialisation that made it train well: the layers that predict them start at zero. Scale 0, shift 0, gate 0, so every block outputs exactly its input, and the whole network starts as the identity.
The diffusion transformers we checked all modulate this way: SD3 (in diffusers its blocks are AdaLayerNormZero), FLUX.1 (layer norm without a gain, then $(1 + \text{scale})\cdot\mathrm{LN}(x) + \text{shift}$) and Qwen-Image (a shift, scale and gate for both sublayers).
They also put RMSNorm on the queries and keys. SD3 added it after the high-resolution divergence from the last section, SD3.5’s card credits it with better training stability, and FLUX.2 keeps both the gainless layer norm and the RMSNorm on q and k.
The U-Net diffusion models, Stable Diffusion 1.5, 2.1 and SDXL, still use group norm with 32 groups (Part 4).
The idea is older than diffusion. AdaIN (Huang and Belongie, 2017) transferred style by normalising a content image channel by channel, then putting back a style image’s mean and spread through $\gamma$ and $\beta$. adaLN is the same move, with a network predicting what to put back.
Can a model train without any norm layer?
Every norm computes a statistic over many numbers, a reduction, before it can write a single output. That costs memory traffic, and it ties every output to all its neighbours. Could a point-wise function, one that looks at each number alone, do the job?
DyT, dynamic tanh (Zhu, Chen, He, LeCun and Liu, CVPR 2025), replaces every norm, the final one included, with
\[\mathrm{DyT}(x) = \gamma \odot \tanh(\alpha x) + \beta\]where $\alpha$ is one learned number per layer and $\gamma$, $\beta$ are per feature, as before. No averages: a large input is squashed toward ±1 and a small one passes almost unchanged.
The difference shows with one outlier. Turn the 6 in row 1 into 66. Layer norm’s mean jumps to 13 and its spread to 24mean 13, spread 24, worked out3 + 10 − 2 + 66 + 0 + 1 = 78; 78 ÷ 6 = 13deviations −10, −3, −15, 53, −13, −12squares 100 + 9 + 225 + 2,809 + 169 + 144 = 3,456; ÷ 6 = 576; √576 = 24, so the other five collapse into a band 0.5 widethe squeezed five, worked out(3, 10, −2, 0, 1 each minus 13) ÷ 24= −0.417, −0.125, −0.625, −0.542, −0.5from −0.625 to −0.125: 0.5 wideWithout the outlier they ran from −1.25 to 1.75: 3 wide, six times more.. DyT caps the outlier near 1 and leaves the other five exactly where they were: with $\alpha = 0.25$, the 3 becomes 0.6350.635, worked outαx = 0.25 × 3 = 0.75tanh 0.75 ≈ 0.635, outlier or not.
On LLaMA it held up. Trained for 200B tokens at 7B, 13B, 34B and 70B parameters, RMSNorm and DyT reached training losses of 1.59 against 1.60, 1.53 against 1.54, 1.50 against 1.50, and 1.45 against 1.45: within 0.01the gaps, worked out7B: 1.60 − 1.59 = 0.0113B: 1.54 − 1.53 = 0.0134B: 1.50 − 1.50 = 070B: 1.45 − 1.45 = 0 at every size.
The catch is $\alpha$. Its starting value had to be tuned: 0.8 in attention and 0.2 elsewhere at 7B, down to 0.2 and 0.05 at 70B. The paper found that larger models need a smaller one, and that width, not depth, sets it. The figure’s chips show why it matters: too small and tanh is a straight line, too large and every number becomes ±1.
Others followed. Derf (Chen, Lu, Zhu, Sun and Liu, CVPR 2026) swaps tanh for the error function, $\mathrm{erf}(\alpha x + s)$, and its abstract reports beating layer norm, RMSNorm and DyT on visual recognition and generation, speech and DNA. nGPT (Loshchilov et al., ICLR 2025) goes the other way and normalises everything: hidden states and weight matrices all live on the unit sphere, and it reports 4 to 20 times fewer training steps to the same accuracy.
Status as of October 2026: none of the ~35 open models we checked ships without norm layers. The frontier is changing the residual stream instead, and keeping the norms.
DeepSeek-V4 (April 2026) runs mHC, manifold-constrained hyper-connections: four copies of the residual stream, mixed between layers by matrices that Sinkhorn iterations keep constrained. Its model card credits mHC with “enhancing stability of signal propagation across layers”.
Qwen3.8-Flash-Next (August 2026) also widens the stream to four copies and adds a “Gated Residual”. Its card says the quiet part out loud: “Residual streams with normalization are what make deep LLM training manageable.”
Muse Spark Open (October 2026) gates the stream at every sublayer: the output is $a$ times the residual plus $b$ times the sublayer’s normalised output, with $b = \mathrm{sigmoid}(g/T)$, $T = 0.3$, and $a = \sqrt{(1 - b)(1 + b)}$. Then a² + b² = 1a² + b² = 1, worked outa² = (1 − b)(1 + b) = 1 − b²a² + b² = 1 − b² + b² = 1Two independent parts of variance v, weighted a and b, add to (a² + b²) v = v., so if the residual and the branch are independent with the same variance, the sum keeps it. The 161 from the first section never happens.
What else does a 2026 training recipe do with its norms?
The statistics are computed in fp32, even when everything else runs in bf16. bf16 keeps 8 significant bits, so a running sum stops absorbing small terms: in bf16, 256 + 1 = 256256 + 1 in bf16, worked out8 significant bits: between 256 and 512, bf16 numbers are 2 apart257 sits exactly halfway between 256 and 258round to even: 256Add 1 to a bf16 total a thousand times and it stops at 256.. A sum of squares over thousands of features would stall long before it finished.
Every reference RMSNorm in Hugging Face’s code for Llama, Qwen, Gemma, OLMo, GPT-OSS and DeepSeek upcasts to fp32 to compute the RMS. They differ on one detail: Llama and Qwen cast back to bf16 before multiplying by the gain, while Gemma, OLMo 2 and GPT-OSS multiply in fp32 and then cast.
DeepSeek-V4’s reference code keeps the norm weights themselves in fp32. DeepSeek-V3 trained in FP8 but kept its normalisation operators in BF16 or FP32, and it recomputes every RMSNorm in the backward pass instead of storing its output, trading a little arithmetic for memory.
No weight decay on $\gamma$ and $\beta$, by default. nanoGPT’s comment says it plainly: “all weight tensors in matmuls + embeddings decay, all biases and layernorms don’t”. The Hugging Face Trainer excludes norm parameters by name, and Megatron-LM puts zero decay on every 1-D parameter. The exception is the zero-centred gain, where decay pulls toward 1; Megatron has an option for what it calls “the Qwen3-Next rule”, decay on the QK-norm gains.
Norms do almost no arithmetic per byte: read a row, reduce it, write it back. They are memory-bound and never touch the tensor cores, and PyTorch’s July 2026 post “Towards Free Normalization” puts them at “roughly 10% of total latency” in a typical LLM.
So kernels fuse them. Apex, Transformer Engine, Liger Kernel and Quack ship fused RMSNorms, and the same PyTorch post fuses the norm into the neighbouring matrix multiply, hiding up to 90% of the norm kernel’s latency. For an RMSNorm without a gain the trick is one line: scaling each row of A commutes with the multiply, so multiply first and scale the rows of the result.
In vision, batch norm is still the default: torchvision’s ResNet, EfficientNet and MobileNetV3, and the convolution block of Ultralytics’ YOLO, all use BatchNorm2d. When each GPU holds only a few images, SyncBatchNorm computes the statistics over every GPU’s share of the batch (Part 2). ConvNeXt replaced batch norm with layer norm, and ViT uses pre-norm layer norm, like the language models.
Which norm should you reach for?
| situation | reach for | why | covered in |
|---|---|---|---|
| a CNN for images with a healthy batch (ResNet, EfficientNet, YOLO) | batch norm | many examples per feature; running averages at test time | Part 2 |
| the same CNN with a few images per GPU | SyncBatchNorm, or group norm | add up the statistics across GPUs, or stop needing the batch | Parts 2 and 4 |
| detection, segmentation, a diffusion U-Net | group norm, 32 groups | statistics inside one image, any batch size | Part 4 |
| style transfer | instance norm, AdaIN | wipe one image’s contrast and colour cast; put a style’s back | Part 4 |
| ConvNeXt or ViT | layer norm | per token, no batch | Part 3 |
| a language model | RMSNorm, pre-norm, plus a final norm | one reduction, no batch, a clean road for the gradient | Parts 3 and 5 |
| loss spikes from attention | QK-norm, or QK-Clip under MLA | cancels the growth of the logits | Part 5 |
| a diffusion transformer | layer norm without a gain, adaLN-Zero, RMSNorm on q and k | the timestep and prompt set the scale, shift and gate | Part 5 |
| a research model without reductions | DyT or Derf | point-wise, no statistics; tune α | Part 5 |
Part 1 started with a drift: 80 layers, 10% each, 2,048 times wider. Every part since has re-measured the numbers over a different set: the column, the row, the group inside an image.
2026 adds two more answers. Where the measurement goes decides whether the gradient has a clean road home, and the scale and shift can be set by the timestep rather than learned once. The numbers still drift, and every model we checked still re-measures them.
Rapid fire: can you do these from memory?
- Write post-norm and pre-norm for one sublayer F. In an 80-layer post-norm model, how many norms sit on the residual path, and why does that call for a warm-up?
- Why does a pre-norm model need a final norm, and roughly how wide is its idealised stream after 80 layers?
- Softmax of 2, 1, 0: roughly what weights? What happens at 20, 10, 0, and why does the gradient vanish?
- Why does QK-norm cancel the growth of the query and key weights, and why can an MLA model not use it?
- What is a zero-centred gain, and how does it change the rule about weight decay?
- In adaLN-Zero, which three numbers are predicted, what do they start at, and what does a block compute at step 0?
- One number in a row becomes an outlier. What happens to the other outputs of layer norm, and to those of DyT?
- Why are RMSNorm statistics computed in fp32 during bf16 training?
Where can you go deeper?
The model facts above come from the released code, configs, model cards and reports, read in October 2026. The papers, by section:
Placement.
- Ruibin Xiong et al., “On Layer Normalization in the Transformer Architecture”, ICML 2020: arXiv 2002.04745. Post-norm’s large gradients near the output at initialisation, and pre-norm without a warm-up.
- Jeonghoon Kim et al., “Peri-LN: Revisiting Normalization Layer in the Transformer Architecture”, ICML 2025: arXiv 2502.02732. The sandwich.
- Team OLMo, “2 OLMo 2 Furious”, 2024: arXiv 2501.00656. Norms after the sublayer, and the switch to RMSNorm.
- Chen and Wei, “Post-LayerNorm Is Back: Stable, ExpressivE, and Deep” (Keel), 2026: arXiv 2601.19895.
- The reports and cards behind the table: Gemma 2 (arXiv 2408.00118), Gemma 3 (arXiv 2503.19786), Gemma 4 (arXiv 2607.02770), Qwen3 (arXiv 2505.09388), GPT-OSS (arXiv 2508.10925), DeepSeek-V3 (arXiv 2412.19437), and the Qwen3-Next model card.
Queries and keys.
- Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar and Yuxuan Chen, “Query-Key Normalization for Transformers”, Findings of EMNLP 2020: arXiv 2010.04245.
- Mostafa Dehghani et al., “Scaling Vision Transformers to 22 Billion Parameters”, ICML 2023: arXiv 2302.05442.
- Mitchell Wortsman et al., “Small-scale proxies for large-scale Transformer training instabilities”, ICLR 2024: arXiv 2309.14322.
- Kimi Team, “Kimi K2: Open Agentic Intelligence”, 2025: arXiv 2507.20534. Section 2.1 is QK-Clip.
Diffusion.
- William Peebles and Saining Xie, “Scalable Diffusion Models with Transformers” (DiT), ICCV 2023: arXiv 2212.09748. adaLN-Zero.
- Patrick Esser et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis” (SD3), ICML 2024: arXiv 2403.03206. Section 5.3.2 is the QK-norm fix.
- Xun Huang and Serge Belongie, “Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization”, ICCV 2017: arXiv 1703.06868.
- FLUX.1’s modulation and QK-norm in black-forest-labs/flux, layers.py.
Without norm layers.
- Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun and Zhuang Liu, “Transformers without Normalization” (DyT), CVPR 2025: arXiv 2503.10622.
- Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun and Zhuang Liu, “Stronger Normalization-Free Transformers” (Derf), CVPR 2026: arXiv 2512.10938.
- Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun and Boris Ginsburg, “nGPT: Normalized Transformer with Representation Learning on the Hypersphere”, ICLR 2025: arXiv 2410.01131.
- DeepSeek-AI, “mHC: Manifold-Constrained Hyper-Connections”, 2025: arXiv 2512.24880, and the DeepSeek-V4 model card.
- The Qwen3.8-Flash-Next model card and the Muse Spark Open model card.
In practice.
- PyTorch, “Towards Free Normalization: Fusing Normalization into GEMM and Attention Kernels”, July 2026: pytorch.org/blog.
- nanoGPT’s weight-decay groups in model.py, and PyTorch’s SyncBatchNorm documentation.
- Zhuang Liu et al., “A ConvNet for the 2020s” (ConvNeXt), CVPR 2022: arXiv 2201.03545.