What is the steepest slope on a hill that rises 3 one way and 4 the other?
Every training step asks the loss one question: which way is downhill, and how steep? The answer is a gradient. This series builds it in six parts, section by section from Chapter 5 of Mathematics for Machine Learning (MML), and ends with a CNN solved by hand.
It also fixes one convention for every part, the one PyTorch uses: inputs are rows. This part says what that means, and why it hands you a free check on every gradient you write.
The 30-second version
5, toward (3, 4). The slope in any direction is the gradient [3, 4] dotted with that direction, and a dot product with a unit vector is largest when the two point the same way: √(3² + 4²) = 5. Walking diagonally gives 7/√2 ≈ 4.95, close but not the top. At a right angle to (3, 4) the slope is zero, which is why the gradient always crosses the contour lines at a right angle.The film works the puzzle on MML’s Example 5.7, every number on screen computed. The pages after it keep the notation.
What is a partial derivative, and what is the gradient?
Freeze every input but one. The surface collapses to an ordinary curve, and the slope of that curve is the partial derivative:
\[\frac{\partial f}{\partial x_1} = \lim_{h\to 0}\frac{f(x_1 + h,\, x_2) - f(x_1,\, x_2)}{h}\]In words: nudge $x_1$, hold $x_2$ still, and divide the change in $f$ by the nudge. In the film the ratio reads 3.5, 3.1 and 3.01 as the run shrinks from 0.5 to 0.01.
The frozen input is just a constant, so the rules from school still work. For $f = x_1^2x_2 + x_1x_2^3$ that gives $\partial f/\partial x_1 = 2x_1x_2 + x_2^3$ and $\partial f/\partial x_2 = x_1^2 + 3x_1x_2^2$. At (1, 1) they are 3 and 4both slopes at (1, 1)∂f/∂x₁ = 2x₁x₂ + x₂³ = 2 × 1 × 1 + 1³ = 3∂f/∂x₂ = x₁² + 3x₁x₂² = 1² + 3 × 1 × 1² = 4.
Put them side by side and you have the gradient:
\[\nabla f = \begin{bmatrix}\dfrac{\partial f}{\partial x_1} & \dfrac{\partial f}{\partial x_2}\end{bmatrix} = \begin{bmatrix}2x_1x_2 + x_2^3 & x_1^2 + 3x_1x_2^2\end{bmatrix} = \begin{bmatrix}3 & 4\end{bmatrix}\]One number per input, each saying how fast $f$ moves when that input alone is nudged. The point is the row $x = [x_1,\ x_2]$, so the gradient is a $1 \times 2$ row too: the slope along $x_1$ sits where $x_1$ sits.
Its first job is to predict small changes. Nudge the point by a small row $\mathrm{d}x$: each input moves $f$ by its slope times its nudge, and the moves add:
\[f(x + \mathrm{d}x) \approx f(x) + \nabla f\,\mathrm{d}x^\top\]The $\mathrm{d}x^\top$ is the nudge stood up as a column, so a $1 \times 2$ row times a $2 \times 1$ column gives one number. For $\mathrm{d}x = [0.01,\ 0.02]$ the rule predicts 0.11the predicted change3 × 0.01 = 0.03 and 4 × 0.02 = 0.080.03 + 0.08 = 0.11, and $f$ really moves by 0.1123the true changef(1.01, 1.02) = 1.01² × 1.02 + 1.01 × 1.02³ = 2.1123f(1, 1) = 2, so f moves by 0.1123Halve the nudge and the 0.0023 left over drops to 0.0006: it shrinks like the nudge squared.. Part 2 is built on this line.
Which way round does this series write vectors?
Every vector in matrix calculus is either a row or a column, and the formulas change with the choice. This series chooses once, the way PyTorch holds data: inputs are rows. One example with $n$ features is a $1 \times n$ row $x$, and a batch stacks $B$ examples as the rows of a $B \times n$ matrix, a tensor of shape (B, n). A layer multiplies from the right:
$W$ is $n \times m$: row $i$ belongs to input $i$ and column $j$ to output $j$, so $W_{ij}$ is the weight from input $i$ to output $j$.
The shape rule. The gradient of a number, such as the loss $L$, has the shape of the thing it is taken with respect to: $\partial L/\partial x$ is $1 \times n$, $\partial L/\partial W$ is $n \times m$, $\partial L/\partial b$ is $1 \times m$. The update $W \leftarrow W - \eta\,\partial L/\partial W$ needs matching shapes, it is exactly what PyTorch keeps in W.grad, and a gradient of the wrong shape is wrong, which is a check you get for free.
One entry first. Write one entry as a sum, take an ordinary derivative, then put each slope where its variable sits. Take $y = xw$ with $x = [1,\ 2]$ and $w$ the column of weights 3 and −1, so $y = x_1w_1 + x_2w_2 =$ 1y = xwy = 1 × 3 + 2 × (−1) = 3 − 2 = 1Nudge w₂ to −0.9 and y = 3 − 1.8 = 1.2: it moved 0.2, which is x₂ × 0.1.. Weight $w_2$ sits in one term, $x_2w_2$, so its slope is $x_2$. In place:
\[\frac{\partial y}{\partial w} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} = x^\top, \qquad \frac{\partial y}{\partial x} = \begin{bmatrix} 3 & -1 \end{bmatrix} = w^\top\]Neither transpose was memorised; the shape rule put both in place. Part 3 runs the same three moves on a whole weight matrix.
MML writes vectors as columns ($Ax$ where we write $xW$), so its formulas that involve a vector appear here with that vector turned into a row, as in $\nabla f\,\mathrm{d}x^\top$ above. The gradient of a number is a row in both.
Why does the gradient point uphill?
Walk in a unit direction, the row $u = [\cos\theta,\ \sin\theta]$. It is a nudge of length 1, so the slope you feel is the gradient dotted with it:
\[\nabla f\,u^\top = 3\cos\theta + 4\sin\theta = 5\cos(\theta - 53.1^\circ)\]It peaks when $u$ lines up with the gradient, at ‖∇f‖ = 53 cos θ + 4 sin θ as one wavewrite it as R cos(θ − φ): R cos φ = 3 and R sin φ = 4R = √(3² + 4²) = √25 = 5φ = arctan(4 ÷ 3) = 53.1°, and it is zero at a right angle to it, along the contour line. So the gradient is the steepest way up and its length is the steepest slope. Training walks the other way, along $-\nabla f$.
Let the point move and the same dot product is the chain rule, with the velocity $v = [\mathrm{d}x_1/\mathrm{d}t,\ \mathrm{d}x_2/\mathrm{d}t]$ a row too:
\[\frac{\mathrm{d}f}{\mathrm{d}t} = \frac{\partial f}{\partial x_1}\frac{\mathrm{d}x_1}{\mathrm{d}t} + \frac{\partial f}{\partial x_2}\frac{\mathrm{d}x_2}{\mathrm{d}t} = \nabla f\,v^\top\]From (1, 1) with $v = [4,\ -3]$ your height does not change at all at first, $3 \cdot 4 + 4 \cdot (-3) = 0$, because that velocity runs along the contour. MML’s Example 5.8 walks a circle, $f = x_1^2 + 2x_2$ with $x_1 = \sin t$ and $x_2 = \cos t$, and gets $2\sin t\cos t - 2\sin t$.
Part 2 gives the function several outputs, and the gradient row grows into a matrix.
- Say what a partial derivative is in one sentence, without symbols.
- Find both partial derivatives of f(x, y) = (x + 2y³)².
- The input is a 1 × 5 row and W is 5 × 3. What shapes are ∂L/∂x and ∂L/∂W, and why must they be?
- The partial slopes at a point are 5 and 12. What is the steepest slope there, and the slope along the contour line?
- You derived a gradient by hand. How do you check it using nothing but the function itself?
Where can you go deeper?
All of this part is from Mathematics for Machine Learning by Deisenroth, Faisal and Ong (Cambridge University Press, 2020), Chapter 5, Vector Calculus, free at mml-book.github.io:
- §5.1, the derivative as a limit, and the chain rule in §5.1.2.
- §5.2, partial derivatives (Definition 5.5) and the gradient (5.40), with Example 5.7, this part’s surface.
- §5.2.2, the chain rule with two inputs (5.49) and Example 5.8; the remark that closes the section checks a gradient with finite differences.
- §5.8, the linear approximation (5.148), which with rows reads $f(x) \approx f(x_0) + \nabla f(x_0)(x - x_0)^\top$.