How much more work is a derivative than the function itself?
Part 4 ran backpropagation by hand. Nobody does that for a real model: you write the forward pass, call loss.backward(), and every weight gets its gradient. This part opens that call.
MML’s answer is that backpropagation is one case of automatic differentiation, which differentiates any program built from small operations (§5.6.2). The first film walks one of the chapter’s examples, the second Part 3’s layer.
The 30-second version
About the same: a small constant factor, however long the formula gets. Split f into six elementary steps and keep every intermediate value. Then walk the steps backwards, multiplying in each step’s own derivative, and add wherever two paths meet. The backward walk has as many steps as the forward one, and at x = 1 it gives 5.983. That walk is reverse-mode automatic differentiation, and backpropagation is the same walk through a neural network.How does reverse mode walk the graph?
The film’s program is MML 5.123 to 5.128:
\[\begin{aligned} a &= x^2, & b &= \exp(a), & c &= a + b,\\ d &= \sqrt{c}, & e &= \cos(c), & f &= d + e. \end{aligned}\]Each line is a node, each input a wire, and the repeated piece $x^2 + \exp(x^2)$ is $c$, computed once. Every wire knows only its own derivative: $2x$, $\exp(a)$, 1 into a sum, $1/(2\sqrt{c})$ and $-\sin c$ out of $c$.
Reverse mode seeds $\bar f = 1$ and asks every node how much $f$ would move if it moved: the adjoint, $\bar c = \partial f/\partial c$. Each node collects from the nodes it feeds (MML 5.145):
\[\frac{\partial f}{\partial x_i} = \sum_{x_j:\; x_i \in \mathrm{Pa}(x_j)} \frac{\partial f}{\partial x_j}\,\frac{\partial x_j}{\partial x_i}\]In words: my effect on $f$ is the sum, over everything I feed, of its effect on $f$ times my effect on it.
At $x = 1$ the node $c$ feeds two others, and the two wires out of it carry 0.259 and 0.5450.259 and 0.545, worked outc = a + b = 1 + exp(1) = 1 + 2.7183 = 3.7183the wire to d: 1 ÷ (2√c) = 1 ÷ (2 × 1.9283) = 0.2593the wire to e: −sin c = −sin 3.7183 = 0.54533.7183 radians is past π, where sine is negative, so −sin c comes out positive.. Two answers come back and add, and the walk ends at 5.9835.983, worked out∂f/∂c = 1 × 0.2593 + 1 × 0.5453 ≈ 0.8045: c feeds d and e, and the two paths add∂f/∂a = 0.8045 × exp(1) + 0.8045 × 1 = 2.1870 + 0.8045 = 2.9915: a feeds b and c∂f/∂x = 2.9915 × 2x = 2.9915 × 2 = 5.983The closed form 5.110 at x = 1 gives the same 5.983.:
\[\begin{aligned} \bar c &= 1\cdot 0.259 + 1\cdot 0.545 \approx 0.805, \\[6pt] \bar a &= 0.805\cdot e^{1} + 0.805\cdot 1 \approx 2.992, \\[6pt] \bar x &= 2.992 \cdot 2 \approx 5.983. \end{aligned}\]The machine keeps every digit; rounded to three places, the last ones need an $\approx$.
Forward mode seeds $\dot x = 1$ and pushes a tangent, how fast each value moves when $x$ moves. It lands on the same 5.983: one chain rule, brackets in different places (MML 5.120 and 5.121):
\[\underbrace{\left(\frac{dy}{db}\,\frac{db}{da}\right)\frac{da}{dx}}_{\text{reverse}} \;=\; \frac{dy}{dx} \;=\; \underbrace{\frac{dy}{db}\left(\frac{db}{da}\,\frac{da}{dx}\right)}_{\text{forward}}\]The walk back costs one multiply-add per wire, so the derivative costs about as much as $f$. The saving is reuse: $c$ is computed once, and so is $\bar c$. Here is the whole machine for four operations, in about 25 lines:
import math
class Var:
def __init__(self, value, parents=()):
self.value = value # computed on the way forward
self.parents = parents # (parent, local derivative) pairs
self.grad = 0.0 # df/d(this node), filled in on the way back
def __add__(self, other):
return Var(self.value + other.value, [(self, 1.0), (other, 1.0)])
def square(u): return Var(u.value ** 2, [(u, 2 * u.value)])
def exp(u): return Var(math.exp(u.value), [(u, math.exp(u.value))])
def sqrt(u): return Var(math.sqrt(u.value), [(u, 0.5 / math.sqrt(u.value))])
def cos(u): return Var(math.cos(u.value), [(u, -math.sin(u.value))])
def backward(out):
order, seen = [], set()
def visit(v): # every node after its parents
if id(v) not in seen:
seen.add(id(v))
for p, _ in v.parents:
visit(p)
order.append(v)
visit(out)
out.grad = 1.0 # df/df = 1 (MML 5.144)
for v in reversed(order): # children before parents
for p, local in v.parents:
p.grad += v.grad * local # sum over children (MML 5.145)
x = Var(1.0)
a = square(x); b = exp(a); c = a + b # MML 5.123 to 5.128
d = sqrt(c); e = cos(c); f = d + e
backward(f)
print(f"f(1) = {f.value:.4f}")
for name, v in [("c", c), ("b", b), ("a", a), ("x", x)]:
print(f"df/d{name} = {v.grad:.4f}")
Why does training run backwards?
With one input and one output the direction never matters. With many, it decides everything: forward mode gets one column of the Jacobian per pass and reverse mode one row, so $n$ inputs and $m$ outputs cost $n$ passes against $m$.
A training loss has $m = 1$. Part 3’s layer, $z = xW + b$ with the toy loss $L = 2z_1 - z_2 + 3z_3$, already depends on eleven numberseleven inputs, countedx is 1 × 2: 2 numbersW is 2 × 3: 6 numbersb is 1 × 3: 3 numbers2 + 6 + 3 = 11, and one number, L, comes out: eleven forward passes, or one backward. For Llama 3 8B it is 8.03 billion against one backward pass, which costs about twice a forward pass.
The backward pass needs a record of the forward one. With requires_grad=True, every operation writes a line on a tape: its inputs, what its backward step will need, and a backward function. loss.backward() reads the tape bottom-up, each line adding into its inputs’ gradient slots. A value used twice gets two deposits: the p.grad += line of the Var cell.
Bottom-up works because every user of a value was recorded after it, so it is read first. The price is memory: a deep network holds every layer’s activations until the sweep returns. Activation checkpointing keeps a few and recomputes the rest.
What does each backward function compute?
Each backward function receives $\delta$, the gradient of the loss at the operation’s output, shaped like that output. It returns $\delta$ times the local Jacobian without building the Jacobian: a vector-Jacobian product. In a network every line on the tape is a block, and its backward step is one of Part 3’s two rules.
The linear block $z = xW + b$, with $x$ a $1 \times n$ row and $W$ of shape $n \times m$, hands back three things, each shaped like what it belongs to:
\[\frac{\partial L}{\partial x} = \delta\,W^\top, \qquad \frac{\partial L}{\partial W} = x^\top\delta, \qquad \frac{\partial L}{\partial b} = \delta\]In words: the input gets the gradient times $W^\top$, the weights get the input, transposed, times the gradient, and the bias gets the gradient. An activation $h = f(z)$ hands back $\delta \odot f^\prime(z)$, the gradient times its own slope.
With $x = [1, 2]$, $W$ with rows $[1, -1, 0]$ and $[1, 1, -1]$, $b = [0, 1, 0]$ and the toy loss, $\delta = [2, -1, 3]$. The weights get the rows [2, −1, 3] and [4, −2, 6]xᵀδ, worked outxᵀδ is a 2 × 1 column times a 1 × 3 row: entry (i, j) is xᵢ δⱼrow 1: x₁ = 1 times δ = [2, −1, 3]row 2: x₂ = 2 times δ = [4, −2, 6]Each weight sits in one output’s sum, so its gradient is a single product: input times the gradient at the output it feeds., and the input gets [3, −2]δWᵀ, worked outinput 1 reaches every output through row 1 of W, [1, −1, 0]: 2 × 1 + (−1) × (−1) + 3 × 0 = 3input 2 reaches them through row 2, [1, 1, −1]: 2 × 1 + (−1) × 1 + 3 × (−1) = −2An input sits in every output’s sum, so its gradient adds one term per output.. A ReLU after this layer would see $z = [3, 2, -2]$ and pass back $[2, -1, 0]$.
One twist: nn.Linear(2, 3) stores its weight as a $3 \times 2$ matrix, out by in, our $W^\top$, and computes x @ weight.T + bias, which is $xW + b$. So weight.grad holds $\delta^\top x$, our $x^\top\delta$ stored transposed: the rows [2, 4], [−1, −2] and [3, 6]nn.Linear’s weight.grad, worked outnn.Linear(2, 3).weight is 3 × 2: row j belongs to output jits grad is δᵀx: row j is δⱼ times x = [1, 2]2 × [1, 2] = [2, 4], −1 × [1, 2] = [−1, −2], 3 × [1, 2] = [3, 6]The transpose of xᵀδ: the same numbers, stored the other way round.. For a batch $X$ with one example per row it holds $\Delta^\top X$.
Autograd differentiates the program that ran: an if contributes only the branch it took, and ReLU’s kink at 0 gets the convention 0. A hand-written backward in a fused kernel can simply be wrong, so check it against a finite difference, as MML suggests at the end of §5.2.2. The nudge has a sweet spot:
import math
def f(x):
s = x * x + math.exp(x * x)
return math.sqrt(s) + math.cos(s)
def closed_form(x): # MML 5.110, written out by hand
s = x * x + math.exp(x * x)
return 2 * x * (1 / (2 * math.sqrt(s)) - math.sin(s)) * (1 + math.exp(x * x))
exact = closed_form(1.0)
print(f"closed form {exact:.12f}")
for k in [1, 3, 5, 7, 9, 11, 13]:
h = 10.0 ** -k
fd = (f(1.0 + h) - f(1.0 - h)) / (2 * h)
print(f"h = 1e-{k:<2} {fd:.12f} error {abs(fd - exact):.1e}")
The error bottoms out near $h = 10^{-7}$, then climbs as rounding takes over: good enough to catch a wrong gradient, never to train with. The next part puts these rules to work on a small CNN, solved by hand.
- Break f(x) = √(x² + exp(x²)) + cos(x² + exp(x²)) into intermediate variables and give the local derivative on every wire.
- A node feeds three others. How does reverse mode combine what comes back to it, and why?
- A function has 10 inputs and 10,000 outputs. Which mode gives its full Jacobian in fewer passes, and how many?
- A linear layer computes z = xW + b for a row x. What does its backward function return for x, W and b, and what shape is
nn.Linear(4, 3).weight.grad? - What does reverse mode cost that forward mode does not, and what does activation checkpointing trade for it?
Where can you go deeper?
All of this is §5.6.2 of Mathematics for Machine Learning by Deisenroth, Faisal and Ong (Cambridge University Press, 2020), free at mml-book.github.io. Example 5.14 has only scalars, so its formulas appear here exactly as MML prints them.
- Equations 5.119 to 5.121 with Figure 5.10: the two modes as two bracketings.
- Example 5.14 with Figure 5.11: the variables (5.123 to 5.128), local derivatives (5.129 to 5.134) and backward sweep (5.135 to 5.142); the function is 5.109, its derivative 5.110.
- Equations 5.143 to 5.145: the procedure for any graph. The end of §5.2.2: gradient checking.