In the backward pass, what does each block receive, keep and hand back?
Part 3 ended with two rules, one for each kind of block. In both, $\delta$ is the gradient of the loss at the block’s output, shaped like that output:
\[\text{linear } z = xW + b:\qquad \frac{\partial L}{\partial W} = x^\top\delta,\qquad \frac{\partial L}{\partial b} = \delta,\qquad \frac{\partial L}{\partial x} = \delta W^\top\] \[\text{activation } h = f(z):\qquad \frac{\partial L}{\partial z} = \delta\odot f^\prime(z)\]In words: a linear block records its input stood up times $\delta$ for its weights, and $\delta$ itself for its bias, then hands back $\delta W^\top$. An activation records nothing and hands back each entry times its own slope. Every network is a chain of these blocks, and backpropagation is these two rules run in one particular order, from the loss back to the input.
The 30-second version
Each block receives δ, the gradient of the loss with respect to its output: a row shaped like that output. A linear block z = xW + b keeps xᵀδ as its weight gradient, an outer product shaped like W, and δ itself as its bias gradient, then hands back δWᵀ. An activation block keeps nothing and hands back δ ⊙ f′(z), entry by entry. Start at the loss with δ = ∂L/∂y and repeat until you reach the input: that is all of backpropagation.The network has three linear blocks, a ReLU after the first and a tanh after the second, and a squared loss with target 1. The first block is Part 3’s layer:
\[x = \begin{bmatrix}1 & 2\end{bmatrix}, \qquad W_1 = \begin{bmatrix}1 & -1 & 0\\ 1 & 1 & -1\end{bmatrix}, \qquad b_1 = \begin{bmatrix}0 & 1 & 0\end{bmatrix}\] \[W_2 = \begin{bmatrix}-0.5 & -1\\ 1 & 1\\ 1 & -1\end{bmatrix}, \qquad b_2 = \begin{bmatrix}0.5 & -0.5\end{bmatrix}, \qquad W_3 = \begin{bmatrix}2\\ -1\end{bmatrix}, \qquad b_3 = 0, \qquad L = (y - 1)^2\]The forward pass runs left to right. Block 1 gives $z_1 = [3, 2, -2]$ and the ReLU zeroes the negative entry, $h_1 = [3, 2, 0]$. Block 2 gives z₂ = [1, −1.5]z₂ = h₁W₂ + b₂, worked outentry 1: 3 × (−0.5) + 2 × 1 + 0 × 1 + 0.5 = 1entry 2: 3 × (−1) + 2 × 1 + 0 × (−1) − 0.5 = −1.5, the tanh squashes it to $h_2 = [0.7616, -0.9051]$, and block 3 gives y = 2.4283y and L, worked outy = 0.7616 × 2 + (−0.9051) × (−1) + 0 = 2.4283L = (2.428337 − 1)² = 1.428337² = 2.0401The film computes every number unrounded and shows four decimals, so a hand product can differ in the last digit. and $L = 2.0401$.
Each block keeps what its backward rule will need: $x$, which units the ReLU let through, $h_1$, $h_2$ and $y$. Those kept rows are most of the memory training uses. The backward pass then starts at the loss, whose slope gives $\delta_3 = \partial L/\partial y = 2(y - 1) = 2.8567$: positive, because $y$ is too big.
How does the gradient walk back, block by block?
From here every step is one of the two rules. The film names the rule at each block and checks every shape; here is the same walk, from the loss back to the input:
| block | rule | records | hands back |
|---|---|---|---|
| $L = (y - 1)^2$ | the loss's slope | nothing | $\delta_3 = 2(y - 1) = 2.8567$ |
| $y = h_2W_3 + b_3$ | linear | $h_2^\top\delta_3 = [2.1756, -2.5857]^\top$, and $\delta_3$ | $\delta_3W_3^\top = [5.7133, -2.8567]$ |
| $h_2 = \tanh(z_2)$ | activation | nothing | $\odot\,(1 - h_2^2) = \odot\,[0.4200, 0.1807]$, giving $\delta_2 = [2.3995, -0.5162]$ |
| $z_2 = h_1W_2 + b_2$ | linear | $h_1^\top\delta_2$, 3 × 2, row 3 zero; and $\delta_2$ | $\delta_2W_2^\top = [-0.6835, 1.8832, 2.9157]$ |
| $h_1 = \mathrm{ReLU}(z_1)$ | activation | nothing | $\odot\,[1, 1, 0]$, giving $\delta_1 = [-0.6835, 1.8832, 0]$ |
| $z_1 = xW_1 + b_1$ | linear | $x^\top\delta_1$, 2 × 3, column 3 zero; and $\delta_1$ | $\delta_1W_1^\top = [-2.5667, 1.1997]$ |
Each row uses the gradient handed back by the row above it. Three things in it are worth saying out loud.
The tanh lets only 18% of unit 2’s gradient through, because unit 2 sits near tanh’s floor. Ten units that flat in a row would leave about 4 × 10⁻⁸Ten slopes of 0.18070.1807² = 0.032650.1807⁴ = 0.03265² = 0.0010660.1807⁸ = 0.001066² = 1.137 × 10⁻⁶0.1807¹⁰ = 1.137 × 10⁻⁶ × 0.03265 = 3.7 × 10⁻⁸: the vanishing gradient.
Unit 3 receives the biggest gradient in its row, 2.9157, and the ReLU throws all of it away: its input was −2, and a small nudge leaves its output at 0. A unit that is off for every example never learns, the dead ReLU.
Unit 3 also costs a row of $\partial L/\partial W_2$ and a column of $\partial L/\partial W_1$. An outer product is zero along a row wherever its input is zero, and along a column wherever its $\delta$ is.
Why walk backwards at all? MML writes each layer’s gradient as its own chain of Jacobians (eqs. 5.117 and 5.118), and the chains for different layers share everything but their last factor. Carry one running product down from the loss and each shared factor is multiplied once: for 80 layers, 159 products instead of 3,240Products for 80 layerseach layer’s chain on its own: 1 + 2 + … + 80 = 80 × 81 ÷ 2 = 3,240one sweep: 80 records + 79 hand-backs = 159.
What does one step of learning do?
Every weight and bias steps against its own gradient at the same time, $W \leftarrow W - \eta\,\partial L/\partial W$. Each gradient already has its weight’s shape, so the step needs no transpose: that was the point of the shape rule.
With $\eta = 0.02$ one step lands almost on the target: y = 0.9644 and L = 0.0013One step at η = 0.02every number minus 0.02 × its gradient, e.g. (W₂)₁₁ = −0.5 − 0.02 × 7.1984 = −0.6440forward again: z₁ = [3.0820, 1.7740, −2], z₂ = [0.0710, −1.6656]h₂ = [0.0709, −0.9310], W₃ = [1.9565, −0.9483]ᵀ, b₃ = −0.0571y = 0.9644, L = (0.9644 − 1)² = 0.0013. With $\eta = 0.05$ the same gradient overshoots: y = −0.9311 and L = 3.7292One step at η = 0.05forward again: z₁ = [3.2051, 1.4350, −2], z₂ = [−1.2854, −1.9220]h₂ = [−0.8579, −0.9581], W₃ = [1.8912, −0.8707]ᵀ, b₃ = −0.1428y = −0.9311, L = (−0.9311 − 1)² = 3.7292Past η ≈ 0.037, one step of this gradient raises the loss., worse than where it started. A gradient knows which way is downhill, not how far the hill goes.
So backpropagation is the two rules from Part 3, applied in reverse order, nothing else. Part 5, automatic differentiation, is the machine that does this bookkeeping for any program, not just a row of blocks.
Rapid fire: can you do these from memory?
- Write the two block rules, linear and activation, with every shape.
- A linear block saw the input [3, 0, 1] and receives δ = [2, −1]. What does it record for its weights, and what shape is it?
- Why is the tanh step an entrywise product and not a matrix product?
- Why did unit 3 cost a row of ∂L/∂W₂ but a column of ∂L/∂W₁?
- One step at η = 0.02 cut the loss from 2.04 to 0.0013, and η = 0.05 raised it to 3.73. What does that say about what a gradient knows?
Where can you go deeper?
All of this is §5.6.1, Gradients in a Deep Network, in Mathematics for Machine Learning by Deisenroth, Faisal and Ong (Cambridge University Press, 2020), free at mml-book.github.io. MML writes vectors as columns ($Ax$ where this series writes $xW$), so its formulas that involve a vector appear here with that vector turned into a row; Jacobians keep MML’s layout, outputs down the side.
- Eqs. 5.111 to 5.114: the network as a composition of layers and its squared loss, with the forward pass drawn in Fig. 5.8.
- Eqs. 5.115 to 5.118: the chain rule for every layer’s parameters, and the observation that most of each product was already computed for the layer above. The backward pass is Fig. 5.9.
- Example 5.11, the gradient of a least-squares loss in a linear model: this part’s last block and its loss, on their own.
- Eq. 5.144 in §5.6.2: the backward pass starts from 1. The rest of §5.6.2 is Part 5.