Where does the transpose come from?
A layer computes $z = xW + b$, and training changes $W$. Ask for $\partial L/\partial W$ in an interview and most people write $x^\top\delta$ from memory. Ask why the transpose sits on $x$ there, but on $W$ in $\delta W^\top$, and the answers get vague.
This part derives both the way you would on a whiteboard: one entry at a time, with small whole numbers. It follows §5.4 of Mathematics for Machine Learning (MML). MML writes vectors as columns ($Ax$ where we write $xW$), so its formulas that involve a vector appear here with that vector turned into a row; Jacobians keep MML’s layout.
The 30-second version
∂L/∂W = xᵀδ = [[2, −1, 3], [4, −2, 6]] and ∂L/∂x = δWᵀ = [3, −2]. Neither transpose is a convention to memorise. A gradient has the shape of what it is taken with respect to, and from a 1 × 2 row x and a 1 × 3 row δ, the only 2 × 3 product is xᵀδ; from δ and W, the only 1 × 2 product is δWᵀ. One entry first says why: weight W_ij sits in one output's sum, so its gradient is x_i δ_j, while input x_i sits in all three, so its three paths add.The film works on one layer with two inputs and three outputs:
\[x = \begin{bmatrix}1 & 2\end{bmatrix},\qquad W = \begin{bmatrix}1 & -1 & 0\\ 1 & 1 & -1\end{bmatrix},\qquad b = \begin{bmatrix}0 & 1 & 0\end{bmatrix},\qquad z = xW + b = \begin{bmatrix}3 & 2 & -2\end{bmatrix}\]An example is a row, the way PyTorch holds a batch: $x$ is 1 × 2, and $W$ is $n \times m$, one row per input and one column per output. So \(W_{ij}\) is the weight from input $i$ to output $j$.
What does one entry tell you?
Write the layer out, one line per output:
\[\begin{aligned} z_1 &= x_1W_{11} + x_2W_{21} + b_1 = 3\\ z_2 &= x_1W_{12} + x_2W_{22} + b_2 = 2\\ z_3 &= x_1W_{13} + x_2W_{23} + b_3 = -2 \end{aligned}\]Two facts carry the whole part. Each weight appears in exactly one line. Each input appears in all three.
The shape rule says how to lay out the answer: the gradient of a number has the shape of the thing it is taken with respect to. So $\partial L/\partial W$ is 2 × 3, one entry per weight, sitting where its weight sits. The film’s toy loss $L = 2z_1 - z_2 + 3z_3$ hands back $\delta = \partial L/\partial z = [2, -1, 3]$, how much the loss moves per unit of each output.
One weight first. \(W_{21}\) sits only in the line for $z_1$, multiplied by $x_2$, so one path runs from it to the loss:
\[\frac{\partial L}{\partial W_{21}} = \frac{\partial L}{\partial z_1}\,\frac{\partial z_1}{\partial W_{21}} = \delta_1\,x_2 = 2\cdot 2 = 4\]In words: the gradient at the output the weight feeds, times the input it multiplies. The weight’s own value never enters. Raise \(W_{21}\) from 1 to 2 and $L$ rises by exactly 4The nudge, worked outL = 2 · 3 − 2 + 3 · (−2) = −2W₂₁: 1 → 2 moves z₁ = 1 · 1 + 2 · 2 + 0 = 5; z₂, z₃ stayL = 2 · 5 − 2 + 3 · (−2) = 22 − (−2) = 4.
Every weight sits in one line, so every entry follows the same two steps: \(\partial L/\partial W_{ij} = x_i\,\delta_j\). Laid out where the weights sit, that is a column times a row:
\[\frac{\partial L}{\partial W} = x^\top\delta = \begin{bmatrix}1\\ 2\end{bmatrix}\begin{bmatrix}2 & -1 & 3\end{bmatrix} = \begin{bmatrix}2 & -1 & 3\\ 4 & -2 & 6\end{bmatrix}\]The shapes check: (2 × 1)(1 × 3) = 2 × 3, the shape of $W$. From $x$ and $\delta$, $x^\top\delta$ is the only product with that shape. That is where the transpose comes from.
An input is different. $x_1$ sits in all three lines, so three paths reach the loss and their terms add. Each sum is $\delta$ dotted with one row of $W$, and dotting with the rows of $W$ is multiplying by $W^\top$:
\[\frac{\partial L}{\partial x} = \delta W^\top = \begin{bmatrix}2 & -1 & 3\end{bmatrix}\begin{bmatrix}1 & 1\\ -1 & 1\\ 0 & -1\end{bmatrix} = \begin{bmatrix}3 & -2\end{bmatrix}\]That is (1 × 3)(3 × 2) = 1 × 2, the shape of $x$. In a deeper network $x$ is the layer below’s output, and this row is the $\delta$ handed down to it. The bias sits in one line with coefficient 1, so $\partial L/\partial b = \delta$.
What is the general rule?
Training sees a batch. Stack $B$ examples as the rows of $X$, a $B \times n$ matrix, and their gradients as the rows of $\Delta$, $B \times m$. The second film adds the example [0, 1], whose loss weighs its outputs 1, 0 and −1:
\[X = \begin{bmatrix}1 & 2\\ 0 & 1\end{bmatrix},\qquad \Delta = \begin{bmatrix}2 & -1 & 3\\ 1 & 0 & -1\end{bmatrix}\]The batch loss is a sum over examples, so each example’s outer product adds, and one product does the adding: $X^\top\Delta$ gives [[2, −1, 3], [5, −2, 5]]XᵀΔ, worked outexample 1: [1, 2]ᵀ[2, −1, 3] = [[2, −1, 3], [4, −2, 6]]example 2: [0, 1]ᵀ[1, 0, −1] = [[0, 0, 0], [1, 0, −1]]sum: [[2, −1, 3], [5, −2, 5]]Entry (2, 1) is column 2 of X dotted with column 1 of Δ: 2 · 2 + 1 · 1 = 5.. The bias is added to every row, so it collects every example’s $\delta$: the column sums of $\Delta$, [3, −1, 2]Column sums of Δ2 + 1 = 3−1 + 0 = −13 + (−1) = 2. The input gradient stays per example: $\Delta W^\top$ has rows [3, −2] and [1, 2].
An activation $h = f(z)$ works entry by entry, so each unit has one path and its gradient is multiplied by its own slope. With a ReLU after the layer, $h = [3, 2, 0]$, the slopes are [1, 1, 0], and a gradient [2, −1, 3] arriving at $h$ leaves as [2, −1, 0].
That is the whole general rule: two blocks.
\[\text{linear block } Z = XW + b:\qquad \frac{\partial L}{\partial W} = X^\top\Delta,\qquad \frac{\partial L}{\partial b} = \textstyle\sum_{\text{rows}}\Delta,\qquad \frac{\partial L}{\partial X} = \Delta W^\top\] \[\text{activation block } h = f(z):\qquad \frac{\partial L}{\partial z} = \delta\odot f^\prime(z)\]Read them in words. The weights get the inputs stood up, times the gradients at the outputs, summed over the batch. The bias gets each output’s gradient, summed over the batch. The input gets the gradient times $W^\top$, and that is the $\Delta$ handed to the block before. An activation multiplies each entry by its own slope and records nothing. With one example, $X^\top\Delta$ is $x^\top\delta$ and the column sum is $\delta$.
Three checks catch most mistakes, by hand or in code an agent wrote: is $\delta$ taken at this block’s output, is $x$ this block’s input, and do the shapes match? One storage detail: PyTorch’s nn.Linear stores $W^\top$, so its weight.grad holds $\delta^\top x$, the same rule transposed.
MML’s §5.4 first builds the derivative of $z$ with respect to $W$ itself, before any loss: 18 partial derivatives, 12 of them zero, since each weight sits in one line. At a width of 4,096 that object would hold 275 GBThe full derivative at width 4,0964,096 outputs × (4,096 × 4,096 weights) = 2³⁶ = 68,719,476,736 entries× 4 bytes in fp32 = 274,877,906,944 bytes ≈ 275 GBAll but one in 4,096 are zero. for one layer and one example. No framework builds it: it multiplies $\delta$ in first and writes $x^\top\delta$ directly. Part 4 runs the two rules back through a whole network.
Rapid fire: can you do these from memory?
- W is n × m. Why is ∂L/∂W = xᵀδ and not δᵀx or xδᵀ?
- x = [4, 0, −1] and δ = [1, −2]. Write ∂L/∂W, and say why one of its rows is zero.
- Why does the gradient of an input need a sum over the outputs, when a weight's has one term?
- A batch of 32 examples: what does XᵀΔ add up, and what are the formula and the shape of ∂L/∂b?
- nn.Linear stores its weight as m × n. What does its weight.grad hold, in terms of x and δ?
Where can you go deeper?
Everything here is from Chapter 5, Vector Calculus, of Mathematics for Machine Learning by Deisenroth, Faisal and Ong (Cambridge University Press, 2020), free at mml-book.github.io, with its vectors turned into rows.
- §5.4, gradients of matrices: derivatives as tensors of partial derivatives, and Figure 5.7’s two ways to hold them.
- Example 5.12 (eqs. 5.85 to 5.92): the derivative of $Ax$ with respect to $A$, this part’s layer in columns.
- §5.5, useful identities for computing gradients (5.99 to 5.108); 5.106, $\partial(a^\top Xb)/\partial X = ab^\top$, is the layer rule in disguise.
- §5.6, backpropagation, which Part 4 follows.