The PreNorm Dilution Problem

As depth increases, each layer's relative contribution diminishes

Hidden State Magnitude (grows with depth)
Layer 1
Layer 20
Per-layer contribution (opacity = relative weight)
Hidden state magnitude grows O(L)
Each layer's contribution shrinks as 1/L