A CNN by hand · 2. The kernel's gradient

A worked example, one operation at a time

K₁₁ multiplied a pixel in every one of the sixteen windows, so its gradient adds sixteen terms. Written out, that sum is a window of X times Δ: the kernel's gradient is X convolved with Δ.

Open video ↗
Read the walkthrough

The chain rule. ∂L/∂K₁₁ = Σi,j (∂L/∂Zij)(∂Zij/∂K₁₁) = Σi,j ΔijXij. Twelve of the sixteen Δs are 0, which leaves Δ₁₂X₁₂ + Δ₁₃X₁₃ + Δ₃₁X₃₁ + Δ₄₃X₄₃ = 2·0 + 2·1 + (−2)·1 + 2·1 = 2.

Δ as a kernel. Those four pixels sit exactly where Δ's nonzero entries land when Δ is laid over X. Sliding Δ to the 2 × 2 positions gives ∂L/∂K = X ⋆ Δ = [[2, 4], [2, 2]].

The symmetry. Forward, Z = X ⋆ K; backward, ∂L/∂K = X ⋆ Δ. The bias sits in every window with weight 1, so ∂L/∂b = ΣΔ = 4.