The Gradient · 5a. The graph and the two sweeps

MML’s Example 5.14 at x = 1, every number computed

MML’s Example 5.14 becomes six lines of a program and a graph of seven nodes. Each wire carries its local derivative. Reverse mode walks back from f, adding wherever a value was used twice, and lands on df/dx = 5.983. Forward mode pushes a tangent from x and lands on the same number.

Open video ↗
Read the walkthrough

The program. a = x², b = exp(a), c = a + b, d = √c, e = cos(c), f = d + e. At x = 1: a = 1, b = 2.718, c = 3.718, d = 1.928, e = −0.838, f = 1.090. The repeated piece x² + exp(x²) is c, computed once.

Local derivatives. 2x on x → a, exp(a) on a → b, 1 on both wires into c, 1/(2√c) = 0.259 on c → d, −sin(c) = 0.545 on c → e, 1 on both wires into f.

Reverse mode. Seed f̄ = 1. d̄ = ē = 1. c feeds two nodes, so c̄ = 1 × 0.259 + 1 × 0.545 ≈ 0.805. a feeds two nodes too: ā ≈ 2.187 + 0.805 = 2.992. x̄ = ā × 2 ≈ 5.983 (MML 5.145).

Forward mode. Seed ẋ = 1 and push: ȧ = 2, ḃ ≈ 5.437, ċ ≈ 7.437, ḋ ≈ 1.928, ė ≈ 4.055, ḟ ≈ 5.983: the same answer.

Two bracketings. Both modes are one chain rule, multiplied from opposite ends (MML 5.120 and 5.121). The backward walk has one multiply-add per wire, so the derivative costs a small multiple of f itself.