The Gradient · 3a. Rows, the shape rule, and one entry first
A worked example, every number computedAn example is a 1 × n row and W is n × m: one row per input, one column per output. The shape rule lays ∂L/∂W out like W. One entry first: W₂₁ sits in one line, so ∂L/∂W₂₁ = δ₁x₂ = 4. Every entry is xᵢδⱼ, which is the outer product xᵀδ. An input sits in every line, so its paths add: δWᵀ = [3, −2]. The bias gets δ.
Open video ↗Read the walkthrough
Rows. x = [1, 2] is one example, a 1 × 2 row; a batch stacks examples as the rows of X (B × n), as PyTorch holds them. W = [[1, −1, 0], [1, 1, −1]] is 2 × 3: row i holds input i's weights, column j feeds output j.
The layer. z = xW + b = [3, 2, −2], written out: z₁ = x₁W₁₁ + x₂W₂₁ + b₁, and so on. Each weight sits in one line; each input sits in all three.
The shape rule. The gradient of a number has the shape of what it is taken with respect to: ∂L/∂W is 2 × 3, ∂L/∂x is 1 × 2, ∂L/∂b is 1 × 3. The toy loss L = 2z₁ − z₂ + 3z₃ gives δ = [2, −1, 3].
One entry first. W₂₁ reaches L along one path, so ∂L/∂W₂₁ = δ₁ · x₂ = 2 · 2 = 4. Every entry is xᵢδⱼ, giving [[2, −1, 3], [4, −2, 6]] = xᵀδ: (2 × 1)(1 × 3) = 2 × 3, the shape of W.
An input. x₁ sits in all three lines: ∂L/∂x₁ = δ₁W₁₁ + δ₂W₁₂ + δ₃W₁₃ = 2 + 1 + 0 = 3, and ∂L/∂x₂ = −2. Dotting with the rows of W is multiplying by Wᵀ: δWᵀ = [3, −2], (1 × 3)(3 × 2) = 1 × 2.
The bias. bⱼ sits in one line with coefficient 1: ∂L/∂b = δ = [2, −1, 3].