The Gradient · 5b. Why training runs backwards

Part 3’s layer and the tape, every number computed

Part 3’s layer z = xW + b with the toy loss L = 2z₁ − z₂ + 3z₃ has eleven inputs and one output: forward mode needs eleven passes, reverse mode one. The tape records every operation and what its backward step needs; read bottom-up, it fills each gradient slot, twice where a value was used twice. In a network each line is a block, and its backward step is xᵀδ, δWᵀ, δ or δ ⊙ f′(z).

Open video ↗
Read the walkthrough

One loss, eleven inputs. x = [1, 2], W with rows [1, −1, 0] and [1, 1, −1], b = [0, 1, 0], so z = [3, 2, −2] and L = −2. Forward mode seeds one input per pass: ẋ = [1, 0] gives ż = [1, −1, 0] and L̇ = 3. Reverse mode seeds L̄ = 1 and gets all eleven gradients at once. Llama 3 8B has 8.03 billion inputs to its loss.

The tape. Each line saves what its backward step needs (exp saves its output, a sum saves nothing). Read bottom-up, c’s slot gets 0.545 then 0.259, a’s gets 0.805 then 2.187, and x̄ ≈ 5.983.

Every line is a block. A layer’s backward step is the linear rule: x̄ = δWᵀ, W̄ = xᵀδ, b̄ = δ. An activation’s is δ ⊙ f′(z): a ReLU on z = [3, 2, −2] turns [2, −1, 3] into [2, −1, 0].

The layer, in numbers. δ = [2, −1, 3]. W̄ = xᵀδ has rows [2, −1, 3] and [4, −2, 6]. x̄ = δWᵀ = [3, −2]: x₁ reaches all three outputs, so three terms add. b̄ = δ.

nn.Linear. nn.Linear(2, 3) stores weight = Wᵀ (3 × 2) and computes x @ weight.T + bias = xW + b. weight.grad = δᵀx has rows [2, 4], [−1, −2] and [3, 6]: the same numbers, transposed. For a batch it is ΔᵀX.