The Gradient · 1. Partial derivatives and the gradient

One surface, two slices, one arrow

MML's surface f(x₁, x₂) = x₁²x₂ + x₁x₂³ at (1, 1). Nudging one input at a time gives slopes 3 and 4; side by side they make the gradient [3, 4], a row the shape of the point. It predicts small changes, points straight uphill with steepest slope 5, and the chain rule dots it with a velocity row.

Open video ↗
Read the walkthrough

The question. Along x₁ the ground rises with slope 3, along x₂ with slope 4. What is the steepest slope?

Nudge one input. Freeze x₂ = 1: f(x₁, 1) = x₁² + x₁. Rise over run for runs 0.5, 0.1, 0.01 gives 3.5, 3.1, 3.01, settling on 3 = ∂f/∂x₁ = 2x₁x₂ + x₂³. Along x₂ the same gives 4.

The gradient. ∇f = [∂f/∂x₁, ∂f/∂x₂] = [3, 4], a 1 × 2 row like the point x = [x₁, x₂]. The nudge [0.01, 0.02] changes f by about ∇f dxᵀ = 0.03 + 0.08 = 0.11; the true change is 0.1123, and the miss (0.0023) drops four times when the nudge is halved.

Rows. Inputs are rows: one example is 1 × n, a batch is B × n. A layer is z = xW + b with W of shape n × m, and the gradient of a number has the shape of what it is taken with respect to: ∂L/∂x is 1 × n, ∂L/∂W is n × m.

One entry first. For y = xw with x = [1, 2] and w = [3, −1]ᵀ, y = 1. Nudging w₂ by 0.1 moves y by 0.2, so ∂y/∂w₂ = x₂. Put each slope where its weight sits: ∂y/∂w = xᵀ, a column like w; ∂y/∂x = wᵀ, a row like x.

Uphill. The slope along a unit row u = [cos θ, sin θ] is ∇f uᵀ = 3 cos θ + 4 sin θ = 5 cos(θ − 53.1°). It peaks at ‖∇f‖ = 5 along [3, 4]; diagonally it is 4.95; at a right angle it is 0, along the contour. A step of 0.01 along the gradient rises 0.0505, more than any other direction tried.

The chain rule. df/dt = ∇f vᵀ. With v = [4, −3]: 3·4 + 4·(−3) = 0, a walk along the contour (after t = 0.01 the true change is −0.0017). With v = [3, 4]: 25. MML's Example 5.8 gives 2 sin t cos t − 2 sin t.

One card. A partial derivative, the gradient row, the steepest slope, and the chain rule as a dot product of rows. Part 2 gives the function several outputs: the Jacobian.