The Gradient · 4a. The two rules, and the forward pass

A worked example, every number computed

A linear block records xᵀδ and δ and hands back δWᵀ; an activation hands back δ ⊙ f′(z). The network x → W₁ → ReLU → W₂ → tanh → W₃ → loss runs forward to y = 2.4283 and L = 2.0401, keeping x, the ReLU's mask, h₁, h₂ and y. The backward pass starts at the loss: δ₃ = 2(y − 1) = 2.8567.

Open video ↗
Read the walkthrough

The two rules. Linear block z = xW + b: records ∂L/∂W = xᵀδ and ∂L/∂b = δ, hands back ∂L/∂x = δWᵀ. Activation block h = f(z): records nothing, hands back δ ⊙ f′(z).

The network. x = [1, 2]; W₁ = [[1, −1, 0], [1, 1, −1]], b₁ = [0, 1, 0]; W₂ = [[−0.5, −1], [1, 1], [1, −1]], b₂ = [0.5, −0.5]; W₃ = [2, −1]ᵀ, b₃ = 0; L = (y − 1)².

Forward. z₁ = [3, 2, −2], h₁ = ReLU(z₁) = [3, 2, 0]; z₂ = h₁W₂ + b₂ = [1, −1.5], h₂ = tanh(z₂) = [0.7616, −0.9051]; y = h₂W₃ = 2.4283; L = 1.4283² = 2.0401. Kept: x, the mask [1, 1, 0], h₁, h₂, y.

The loss. δ₃ = ∂L/∂y = 2(y − 1) = 2.8567, shaped like y. Numbers are computed unrounded and shown to four decimals, so a hand product can differ in the last digit.