The Gradient · 2. The Jacobian
Outputs down the side, inputs acrossThe map f(x) = [x₁² + x₂, x₁x₂] at (1, −1). Nudging each input gives a list of output changes; together they make the Jacobian J = [[2, 1], [−1, 1]], outputs down the side. A gradient row passes back as a row times J, the chain rule multiplies Jacobians, a layer xW has Jacobian Wᵀ, and the gradient of xBxᵀ is x(B + Bᵀ).
Open video ↗Read the walkthrough
Two in, two out. f(x₁, x₂) = [x₁² + x₂, x₁x₂] sends (1, −1) to (0, −1). A grid of points comes out curved: no single matrix does what f does.
Nudge one input. Nudging x₁ by h moves the outputs by about h·(2, −1); at h = 0.1 the true change is [0.21, −0.1]. Nudging x₂ moves them by h·(1, 1). Every other nudge is a mix: four numbers.
The Jacobian. J_ij = ∂f_i/∂x_j, one row per output, one column per input: J = [[2, 1], [−1, 1]]. Down a column: the response to one input. Along a row: one output's gradient.
Zoom in. A small square's image straightens onto J's parallelogram. f(x + dx) ≈ f(x) + dx Jᵀ: for dx = [0.1, 0] it predicts [0.2, −0.1] against a true [0.21, −0.1]; the miss 0.01 drops to 0.0025 when the nudge is halved. det J = 3: area grows three times.
A row passes back. With L = f₁ + 3f₂, ∂L/∂f = [1, 3]. One entry first, one term per path: ∂L/∂x₁ = 1·2 + 3·(−1) = −1, ∂L/∂x₂ = 4. That is the row times J: ∂L/∂x = (∂L/∂f) J = [−1, 4].
The chain rule. A nudge leaves g as dx J_gᵀ and f as dx (J_f J_g)ᵀ, so J of f∘g = J_f J_g. Shapes: (1 × 3)(3 × 2) = 1 × 2. With a loss at the end, ∂L/∂x = (∂L/∂h) J_h J_g ⋯: backprop.
A layer: Wᵀ. For z = xW with x = [1, 2] and W = [[2, −1, 1], [1, 1, −1]], z = [4, 1, −1]. ∂z_j/∂x_i = W_ij, so row j of J is column j of W: J = Wᵀ. A ReLU wipes unit 3's row, and δ = [1, 3, 2] comes back as δJ = [−1, 4] (without the ReLU, [1, 2]).
The quadratic. For x B xᵀ with B = [[1, 2], [0, 3]] at x = [1, 1], three formulas compete: 2xB = [2, 10], x(B + Bᵀ) = [4, 8], (2Bxᵀ)ᵀ = [6, 6]. A nudge of [0.01, 0] changes f by 0.0401, and only x(B + Bᵀ) predicts it. One entry first: row sums plus column sums.