The Gradient · 3b. The general rule: two blocks

A worked example, every number computed

A batch stacks examples as rows. Each example brings its own outer product and XᵀΔ adds them: [[2, −1, 3], [5, −2, 5]]. The bias gets the column sums of Δ, [3, −1, 2]; the input gets ΔWᵀ, one row per example. An activation multiplies each entry by its own slope: δ ⊙ f′(z). Those two blocks are every layer.

Open video ↗
Read the walkthrough

A batch. X = [[1, 2], [0, 1]] (2 × 2) gives Z = XW + b = [[3, 2, −2], [1, 2, −1]]; the loss hands down Δ = [[2, −1, 3], [1, 0, −1]], one δ per example.

The weights. Example 1's outer product [[2, −1, 3], [4, −2, 6]] plus example 2's [[0, 0, 0], [1, 0, −1]] is [[2, −1, 3], [5, −2, 5]] = XᵀΔ: (n × B)(B × m) = n × m.

The bias and the input. ∂L/∂b = column sums of Δ = [3, −1, 2]. ∂L/∂X = ΔWᵀ = [[3, −2], [1, 2]], (B × m)(m × n) = B × n. For a mean loss, divide by B.

The activation. h = f(z) works entry by entry, so ∂L/∂z = δ ⊙ f′(z). A ReLU after the layer: [2, −1, 3] ⊙ [1, 1, 0] = [2, −1, 0].

The rule. Linear block: XᵀΔ, column sums of Δ, ΔWᵀ. Activation block: δ ⊙ f′(z). Check that δ is taken at this block's output, that x is this block's input, and that the shapes match. PyTorch's nn.Linear stores Wᵀ, so its weight.grad is δᵀx.