The Gradient · 4b. The backward pass, block by block
A worked example, every number computedEach linear block records its input stood up times δ and hands back δWᵀ; each activation multiplies by its slope. δ₂ = [2.3995, −0.5162] after the tanh, δ₁ = [−0.6835, 1.8832, 0] after the ReLU. One step at η = 0.02 takes the loss from 2.0401 to 0.0013; η = 0.05 overshoots to 3.7292.
Open video ↗Read the walkthrough
Block 3. ∂L/∂W₃ = h₂ᵀδ₃ = [2.1756, −2.5857]ᵀ (2 × 1), ∂L/∂b₃ = δ₃; hands back δ₃W₃ᵀ = [5.7133, −2.8567].
The tanh. Slope 1 − h₂² = [0.4200, 0.1807]; δ₂ = [5.7133, −2.8567] ⊙ [0.4200, 0.1807] = [2.3995, −0.5162].
Block 2. ∂L/∂W₂ = h₁ᵀδ₂ = [[7.1984, −1.5487], [4.7989, −1.0324], [0, 0]] (3 × 2): row 3 is zero because unit 3's output was 0. Hands back δ₂W₂ᵀ = [−0.6835, 1.8832, 2.9157].
The ReLU. The mask [1, 1, 0] gives δ₁ = [−0.6835, 1.8832, 0]: unit 3's gradient is thrown away.
Block 1. ∂L/∂W₁ = xᵀδ₁ = [[−0.6835, 1.8832, 0], [−1.3670, 3.7665, 0]] (2 × 3): column 3 is zero. Hands back δ₁W₁ᵀ = [−2.5667, 1.1997].
One step. W ← W − η ∂L/∂W for every weight. η = 0.02: y = 0.9644, L = 0.0013. η = 0.05: y = −0.9311, L = 3.7292.