Can you backpropagate through a convolution by hand?
Everyone has called loss.backward() on a CNN. Far fewer people can say what lands in the kernel’s gradient, or why the image’s gradient involves a flipped kernel.
Two results answer both, and each gets its own film. This page sets up a CNN small enough for a pencil and finds the one thing both results need: the gradient on the map.
The 30-second version
The kernel's gradient is the image convolved with Δ: slide Δ over the image exactly as the kernel slid in the forward pass, multiply and add. The image's gradient is Δ, padded with a ring of zeros, convolved with the kernel rotated 180°. The rotation appears because as the window moves right and down, a fixed pixel sits under weights further left and up: the kernel is read backwards along both axes.The network, with $b = 0$, $c = 0$ and target $t = 3$:
\[X = \begin{bmatrix} 0&0&1&0&0\\ 1&0&1&1&0\\ 1&1&1&0&0\\ 0&2&1&2&0\\ 0&1&0&1&0 \end{bmatrix}, \qquad K = \begin{bmatrix} 1 & 1\\ -1 & 2 \end{bmatrix}, \qquad w = \begin{bmatrix} 1 & 1 & -1 & 1 \end{bmatrix}^\top\]The forward pass slides $K$ over $X$. Each entry of the map is one window of $X$ times $K$, cell by cell, added. Deep learning calls this a convolution and writes it $X \star K$:
\[Z_{ij} = \sum_{a,b} X_{i+a-1,\,j+b-1}\,K_{ab} + b, \qquad Z = \begin{bmatrix} -1&3&2&-1\\ 2&2&1&1\\ 6&2&4&-2\\ 4&2&5&1 \end{bmatrix}\]A ReLU zeroes the three negatives, a 2 × 2 max pool keeps each quarter’s winner, $h = [3, 2, 6, 5]$, and one linear unit gives y = 4y, worked outh = [3, 2, 6, 5], w = [1, 1, −1, 1]ᵀy = hw + c = 3 + 2 − 6 + 5 + 0 = 4L = (y − t)² = (4 − 3)² = 1 and $L = 1$.
Backwards, the loss hands back $\partial L/\partial y = 2(y - t) = 2$. The head is a linear block, so $\partial L/\partial h = 2w^\top = [2, 2, -2, 2]$. Max pool sends each of those to its quarter’s winner and gives every other entry 0. Every winner was positive, so the ReLU passes them. That is the gradient on the map:
\[\Delta = \frac{\partial L}{\partial Z} = \begin{bmatrix} 0&2&2&0\\ 0&0&0&0\\ -2&0&0&0\\ 0&0&2&0 \end{bmatrix}\]From here on, $X$, $K$ and $\Delta$ are all we need.
What is the kernel’s gradient?
The image and the gradient on the map, again:
\[X = \begin{bmatrix} 0&0&1&0&0\\ 1&0&1&1&0\\ 1&1&1&0&0\\ 0&2&1&2&0\\ 0&1&0&1&0 \end{bmatrix}, \qquad \Delta = \begin{bmatrix} 0&2&2&0\\ 0&0&0&0\\ -2&0&0&0\\ 0&0&2&0 \end{bmatrix}\]Start from one entry of the map. Weight $K_{ab}$ multiplies exactly one pixel in window $(i, j)$, so its derivative there is that pixel:
\[\frac{\partial Z_{ij}}{\partial K_{ab}} = X_{i+a-1,\,j+b-1}\]The same weight sits in all 16 windows, so the chain rule adds 16 paths, each weighted by its window’s $\Delta_{ij}$:
\[\frac{\partial L}{\partial K_{ab}} = \sum_{i,j}\Delta_{ij}\,X_{i+a-1,\,j+b-1}\]Put that next to the forward line for $Z_{ij}$. It is the same sum with $\Delta$ where $K$ was: slide $\Delta$ over $X$ as a 4 × 4 kernel. It fits in 2 × 2 positions, so the answer has the shape of $K$:
\[\frac{\partial L}{\partial K} = X \star \Delta = \begin{bmatrix} 2 & 4\\ 2 & 2 \end{bmatrix}, \qquad \frac{\partial L}{\partial b} = \sum_{i,j}\Delta_{ij} = 4\]Only Δ’s four nonzero entries count. At the first position they sit on X₁₂, X₁₃, X₃₁ and X₄₃, which gives 2 · 0 + 2 · 1 + (−2) · 1 + 2 · 1 = 2all four positionstop left: 2·0 + 2·1 + (−2)·1 + 2·1 = 2top right: 2·1 + 2·0 + (−2)·1 + 2·2 = 4bottom left: 2·0 + 2·1 + (−2)·0 + 2·0 = 2bottom right: 2·1 + 2·1 + (−2)·2 + 2·1 = 2. The bias sits in every window with weight 1, so its gradient is the sum of $\Delta$.
In words: the kernel’s gradient is the forward pass run again, with the gradient on the map in the kernel’s place.
What is the image’s gradient?
The kernel and the gradient on the map, again:
\[K = \begin{bmatrix} 1 & 1\\ -1 & 2 \end{bmatrix}, \qquad \Delta = \begin{bmatrix} 0&2&2&0\\ 0&0&0&0\\ -2&0&0&0\\ 0&0&2&0 \end{bmatrix}\]Follow one pixel, X₂₃. It sits in four windows, under a different weight in each: K₂₂ in window (1, 2), K₂₁ in (1, 3), K₁₂ in (2, 2) and K₁₁ in (2, 3). Each window sends back its Δ times that weight:
\[\frac{\partial L}{\partial X_{23}} = \Delta_{12}K_{22} + \Delta_{13}K_{21} + \Delta_{22}K_{12} + \Delta_{23}K_{11} = 2\cdot 2 + 2\cdot(-1) + 0 + 0 = 2\]Look at the order. As the window moves right, the pixel sits further left under the kernel. As it moves down, the pixel sits further up. For every pixel $X_{pq}$:
\[\frac{\partial L}{\partial X_{pq}} = \sum_{i,j}\Delta_{ij}\,K_{p-i+1,\,q-j+1}\]As $i$ goes up, $p - i + 1$ goes down: the kernel is read backwards along both axes. Reading a grid backwards along both axes is rotating it by 180°:
\[\operatorname{rot}_{180}(K) = \begin{bmatrix} 2 & -1\\ 1 & 1 \end{bmatrix}\]That gives a recipe for all 25 pixels at once. Pad $\Delta$ with one ring of zeros, k − 1 = 1 for a 2 × 2 kernel, so that an edge pixel, which sits in fewer windows, still gets a full sum with zeros standing in for the windows it misses. Then slide the rotated kernel over it:
\[\frac{\partial L}{\partial X} = \operatorname{pad}(\Delta) \star \operatorname{rot}_{180}(K) = \begin{bmatrix} 0&2&4&2&0\\ 0&-2&2&4&0\\ -2&-2&0&0&0\\ 2&-4&2&2&0\\ 0&0&-2&4&0 \end{bmatrix}\]Row 2, column 3 is the 2 we just found by hand.
The forward pass slides $K$ without flipping it, which is a cross-correlation. Its backward pass for the image flips $K$, which makes it a true convolution: the full convolution of $\Delta$ with $K$. PyTorch calls the same operation conv_transpose2d.
Can you check it in code?
The cell rebuilds the toy, gets Δ, computes both gradients as slides, and checks them against a loop that applies the chain rule window by window. It needs only numpy and runs in your browser.
import numpy as np
def corr(A, B):
"""Slide B over A, multiply, add: deep learning's convolution."""
h, w = B.shape
return np.array([[np.sum(A[i:i+h, j:j+w] * B)
for j in range(A.shape[1] - w + 1)]
for i in range(A.shape[0] - h + 1)])
X = np.array([[0, 0, 1, 0, 0], [1, 0, 1, 1, 0], [1, 1, 1, 0, 0],
[0, 2, 1, 2, 0], [0, 1, 0, 1, 0]], dtype=float)
K = np.array([[1., 1.], [-1., 2.]])
w, t = np.array([1., 1., -1., 1.]), 3.0
QUARTERS = [(0, 0), (0, 2), (2, 0), (2, 2)]
# forward, then back to the map
Z = corr(X, K)
A = np.maximum(Z, 0)
h = np.array([A[r:r+2, s:s+2].max() for r, s in QUARTERS])
y = h @ w
dh = 2 * (y - t) * w
D = np.zeros((4, 4))
for k, (r, s) in enumerate(QUARTERS):
i, j = np.unravel_index(A[r:r+2, s:s+2].argmax(), (2, 2))
D[r + i, s + j] = dh[k]
D = D * (Z > 0)
print("y =", y, " L =", (y - t) ** 2, "\nDelta =\n", D)
# the two results of this part
dK = corr(X, D) # X convolved with Delta
dX = corr(np.pad(D, 1), np.rot90(K, 2)) # pad, rotate 180, slide
print("dL/dK =\n", dK, "\ndL/db =", D.sum())
print("dL/dX =\n", dX)
# the same, window by window, straight from the chain rule
dK2, dX2 = np.zeros((2, 2)), np.zeros((5, 5))
for i in range(4):
for j in range(4):
dK2 += D[i, j] * X[i:i+2, j:j+2]
dX2[i:i+2, j:j+2] += D[i, j] * K
print("dK matches:", np.allclose(dK, dK2),
" dX matches:", np.allclose(dX, dX2))
The notebook linked at the top does the same in PyTorch against autograd: conv2d(X, Δ) for the kernel, and conv2d on the padded Δ with K.flip(0, 1) for the image, which matches conv_transpose2d.
It also has an extra. Write each window as a row of a 16 × 4 matrix $P$ and the convolution is $z = Pk + b$, Part 3’s linear block. Its rule $\partial L/\partial k = P^\top\delta$ is $X \star \Delta$ again, and the notebook checks it on a real layer with channels, a batch, stride and padding.
Can you solve these yourself?
Try each before opening the answer.
-
Work out ∂L/∂K₂₂ as one position of X ⋆ Δ.
answer
It is Δ's last position, one row down and one column in. Its four nonzero entries land on X₂₃, X₂₄, X₄₂ and X₅₄: 2 × 1 + 2 × 1 + (−2) × 2 + 2 × 1 = 2.
-
Work out ∂L/∂X₄₂ by pad, rotate and slide. Which window does it come from?
answer
The padded Δ under that position is [[−2, 0], [0, 0]], and rot₁₈₀(K) = [[2, −1], [1, 1]], so the sum is −2 × 2 = −4. It comes from window (3, 1), where X₄₂ sits under K₂₂ = 2 and Δ₃₁ = −2. The other three windows over X₄₂ have Δ = 0.
-
Change the target from 3 to 5. What happens to the loss and to every gradient?
answer
The loss is still (4 − 5)² = 1, but ∂L/∂y becomes −2 instead of 2. Δ flips sign, and both gradients are built from Δ, so both flip: ∂L/∂K = [[−2, −4], [−2, −2]].
- Which operation gives the kernel's gradient, and what plays the kernel in it?
- Why does ∂L/∂K come out 2 × 2 when Δ is 4 × 4?
- The forward pass never flips K. Why does the image's gradient need it rotated 180°?
- How many rings of zeros does Δ need for a 3 × 3 kernel?
The chain rule this part leans on is Chapter 5 of Deisenroth, Faisal and Ong’s Mathematics for Machine Learning. For stride, padding and transposed convolutions drawn as pictures, read Dumoulin and Visin’s A guide to convolution arithmetic for deep learning.