How many numbers describe a function with two inputs and two outputs near one point?

Part 1 had one output: a surface, its two slopes, and the gradient row that collects them. Almost nothing in a network is that simple. A layer takes a row and returns a row, and so do attention, a norm and a softmax. Training needs to know how every output moves when any input moves.

This part builds the object that records it, the Jacobian, and lets the notation fall out of it: the shape of every derivative, why the chain rule is a matrix product, and where the $W^\top$ of every backward pass comes from.

The puzzle A function takes two numbers in and gives two numbers out. Pick one point. How many numbers does it take to say how the outputs respond to every possible small nudge of the inputs?
The 30-second version Four, in a 2 × 2 matrix: the Jacobian. Nudging the first input moves the outputs by one list of two changes, nudging the second by another, and every other nudge is a mix of the two, so two lists of two numbers say everything. Zoom in far enough and every smooth function looks like a matrix. That one fact is why the chain rule is a matrix product, why a gradient passes back through a layer as a row times its Jacobian, and where every transpose in matrix calculus comes from.

The film follows one small map, $f(x_1, x_2) = [x_1^2 + x_2,\ x_1x_2]$ at (1, −1), then a layer, then a quadratic, every number computed.

What is the Jacobian, and what shape does every derivative have?

Nudge $x_1$ by a little $h$ and both outputs move: $x_1^2 + x_2$ by about $2x_1h$, and $x_1x_2$ by about $x_2h$. Nudge $x_2$ and they move by $h$ and $x_1h$. Collect the four slopes with one row per output and one column per input:

\[J = \frac{\mathrm{d}f}{\mathrm{d}x} = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \dfrac{\partial f_1}{\partial x_2} \\ \dfrac{\partial f_2}{\partial x_1} & \dfrac{\partial f_2}{\partial x_2} \end{bmatrix} = \begin{bmatrix} 2x_1 & 1 \\ x_2 & x_1 \end{bmatrix} = \begin{bmatrix} 2 & 1 \\ -1 & 1 \end{bmatrix}\]

Entry $(i, j)$ is $J_{ij} = \partial f_i/\partial x_j$: outputs down the side, inputs across, so $n$ inputs and $m$ outputs make $J$ an $m \times n$ matrix. Column $j$ is the response to nudging input $j$; row $i$ is output $i$’s gradient, a row like $x$.

Zoom in and $f$ is $J$. A row nudge $\mathrm{d}x$ changes the output row by

\[f(x + \mathrm{d}x) \approx f(x) + \mathrm{d}x\,J^\top\]

each output moving by its own row of $J$ dotted with the nudge. For $\mathrm{d}x = [0.1,\ 0]$ that predicts $[0.2,\ -0.1]$ against a true [0.21, −0.1]the true changef(1.1, −1) = [1.1² − 1, 1.1 × (−1)] = [0.21, −1.1]f(1, −1) = [0, −1], so the change is [0.21, −0.1]The miss, 0.01, is the nudge squared: halve the nudge and it drops to 0.0025.. The determinant, $\det J = 2 \cdot 1 - 1 \cdot (-1) = 3$, says small regions near (1, −1) come out three times bigger.

One rule in two halves gives the shape of every derivative in the series:

you differentiate with respect to you get its shape
a number $L$ a row $x$, $1 \times n$ the gradient $\partial L/\partial x$ $1 \times n$, like $x$
a number $L$ a matrix $W$, $n \times m$ the gradient $\partial L/\partial W$ $n \times m$, like $W$
a row $f$, $1 \times m$ a row $x$, $1 \times n$ the Jacobian $J$ $m \times n$, outputs then inputs

For one output the halves agree: its Jacobian is $1 \times n$, Part 1’s gradient row.

Why is the chain rule just matrix multiplication?

Feed $x$ into $g$ and the result into $f$. A nudge leaves $g$ as $\mathrm{d}x\,J_g^\top$, and leaves $f$ as $\mathrm{d}x\,J_g^\top J_f^\top = \mathrm{d}x\,(J_fJ_g)^\top$. So the Jacobian of the composition is the product (MML 5.48):

\[J_{f\circ g} = J_f\,J_g\]

The shapes check themselves. If $g$ takes 2 numbers to 3 and $f$ takes 3 to 1, the product is $(1 \times 3)(3 \times 2) = 1 \times 2$.

When the last function is a loss, its Jacobian is one row, its gradient, so a gradient passes back through each function as a row times its Jacobian. In the film the loss $L = f_1 + 3f_2$ hands back $[1,\ 3]$:

\[\frac{\partial L}{\partial x} = \frac{\partial L}{\partial f}\,J = \begin{bmatrix}1 & 3\end{bmatrix}\begin{bmatrix} 2 & 1 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix}-1 & 4\end{bmatrix}\]

Entry 1 is one term per path from $x_1$ to $L$, $1 \cdot 2 + 3 \cdot (-1) = -1$. Read the product left to right and you have backpropagation in one line.

The Jacobian of a layer. For $z = xW$, output $j$ is $z_j = \sum_i x_iW_{ij}$, so the slope of output $j$ along input $i$ is the weight between them, $W_{ij}$. Row $j$ of the Jacobian is column $j$ of $W$:

\[\frac{\mathrm{d}(xW)}{\mathrm{d}x} = W^\top\]

$W$ is stored inputs by outputs while $J$ lists outputs first: that swap is the transpose of every backward pass. A bias changes no slope. An activation acts entry by entry, so its Jacobian is diagonal, and a ReLU wipes the rows of the units that did not fire.

In the film, $x = [1,\ 2]$ and a $W$ with rows [2, −1, 1] and [1, 1, −1] give z = [4, 1, −1]z = xWz₁ = 1 × 2 + 2 × 1 = 4z₂ = 1 × (−1) + 2 × 1 = 1z₃ = 1 × 1 + 2 × (−1) = −1. Unit 3 is off, and the row [1, 3, 2] arriving from above comes back as [−1, 4]a row through the layerWᵀ with unit 3’s row wiped: [[2, 1], [−1, 1], [0, 0]]x₁: 1 × 2 + 3 × (−1) + 2 × 0 = −1x₂: 1 × 1 + 3 × 1 + 2 × 0 = 4Without the ReLU the 2 would add [2, −2], giving [1, 2]., the shape of $x$. Part 3 asks the other question: what does $W$ itself get?

Which formula for the derivative of a quadratic is right?

Every squared loss is a quadratic, and with $x$ a row a quadratic is the number $xBx^\top = \sum_{i,j} x_iB_{ij}x_j$. Three formulas for its gradient are in circulation, $2xB$, $x(B + B^\top)$ and $(2Bx^\top)^\top$, all $1 \times n$ rows, so the shape check cannot choose.

A nudge can. Take $B$ with rows [1, 2] and [0, 3], which is not symmetric, at $x = [1,\ 1]$. Nudge by $[0.01,\ 0]$ and $f$ moves by 0.0401the nudge testf(x) = x₁² + 2x₁x₂ + 3x₂², so f(1, 1) = 6 and f(1.01, 1) = 6.0401x(B + Bᵀ) = [4, 8] predicts 4 × 0.01 = 0.042xB = [2, 10] predicts 0.02; (2Bxᵀ)ᵀ = [6, 6] predicts 0.06, and only $x(B + B^\top)$ predicts it.

One entry first gets there without a guess. $x_k$ sits in row $k$ and in column $k$ of the double sum, so $\partial f/\partial x_k = \sum_j B_{kj}x_j + \sum_i x_iB_{ik}$, which is entry $k$ of $xB^\top$ plus entry $k$ of $xB$:

\[\frac{\mathrm{d}(xBx^\top)}{\mathrm{d}x} = x(B + B^\top)\]

With a symmetric $B$ it is $2xB$, which is how the wrong formula survives in people’s notes.

Least squares (MML’s Example 5.11) puts the part to work. Stack $N$ examples as the rows of $\Phi$, $N \times D$, with the weights a $D \times 1$ column $w$, residuals $e = y - \Phi w$ and loss $L = e^\top e$. Weight $w_d$ reaches the loss through every residual with slope $-\Phi_{nd}$, so one entry is $\partial L/\partial w_d = -2\sum_n \Phi_{nd}\,e_n$, a sum down column $d$. Stacked in the shape of $w$:

\[\frac{\partial L}{\partial w} = -2\,\Phi^\top e \qquad (D \times N)(N \times 1) = D \times 1\]

The update $w \leftarrow w - \eta\,\partial L/\partial w$ needs no transpose. Fitting a line to (0, 1), (1, 2) and (2, 2) from the line $\hat y = t$, the gradient is [−4, −2]ᵀ−2Φᵀe at the startw = [0, 1]ᵀ predicts 0, 1, 2, so e = [1, 1, 0]ᵀ and L = 2Φᵀe = [1 + 1 + 0, 0 + 1 + 0]ᵀ = [2, 1]ᵀ−2 × [2, 1]ᵀ = [−4, −2]ᵀOne step with η = 0.1 gives w = [0.4, 1.2]ᵀ; the best line, w = [7/6, 1/2]ᵀ, has L = 1/6. and the loss falls from 2 toward its minimum, 1/6.

  1. A function takes a 1 × 3 row to a 1 × 2 row. What shape is its Jacobian, and what does its second column tell you?
  2. Write the Jacobian of [x₁x₂, x₁ + x₂²] and evaluate it at (1, 1).
  3. What is the Jacobian of x ↦ ReLU(xW) at a point where the second unit is off?
  4. One function takes 4 numbers to 3 and another takes 3 to 1. Which shapes multiply in the chain rule for their composition, and what comes out?
  5. A gradient passing back through a layer meets Wᵀ, yet the update of W needs no transpose. Why both?

Where can you go deeper?

One chapter covers all of this: Mathematics for Machine Learning by Deisenroth, Faisal and Ong (Cambridge University Press, 2020), Chapter 5, Vector Calculus, free at mml-book.github.io:

  • §5.3, gradients of vector-valued functions: the Jacobian (Definition 5.6, 5.56 to 5.59), its determinant as an area scale (Figure 5.5), the table of shapes (Figure 5.6), and Examples 5.9 (a linear map), 5.10 (the chain rule) and 5.11 (least squares).
  • §5.2, the chain rule written as a matrix product (5.48), and the remark that ends §5.2.2 on checking a gradient with finite differences.
  • §5.5, the identities 5.104 to 5.108, including the quadratic form.

MML writes vectors as columns ($Ax$ where we write $xW$), so its formulas that involve a vector appear here with that vector turned into a row; Jacobians keep MML’s layout, outputs down the side. Its Example 5.9 finds the Jacobian $A$ of $Ax$, the same matrix as our $W^\top$.