The Math Under the Model
Every time I take a new technique apart I find old math inside it. LoRA is a low-rank factorisation, Muon orthogonalises a gradient, RoPE is a rotation, FlashAttention is a softmax rearranged so it fits on chip. Coding agents write most of the code around these ideas now, which makes it easy to let the math fade, and the math is exactly what you need to check what the agent wrote. Each topic here is a series. Every part opens on a question worth getting wrong first, the figures are meant to be dragged rather than watched, and where a part has code, it runs twice: once in the page, and once as a PyTorch notebook in Colab.
-
APR 2024topic 01 · 6 parts
The Gradient
Vector calculus for deep learning in six parts, following Chapter 5 of Mathematics for Machine Learning: partial derivatives and gradients, the Jacobian, gradients of matrices, backpropagation and automatic differentiation, then a CNN solved by hand, each told in short films where every number on screen is computed.
-
APR 2024topic 02 · 6 parts
Probability
A language model reads “the cat sat on the” and says mat, 70%. This series follows that one number until it becomes the training loss: what it means, how a sentence multiplies it, how Bayes turns it around, how noisy it is to measure, where the model’s Gaussians come from, and why training minimises its negative log.
-
APR 2024topic 03 · 5 parts
Normalisation
A deep network’s numbers drift as they pass through its layers, so something has to re-measure them. This series asks which numbers to re-measure together: down the batch (batch norm), along each example (layer norm and RMSNorm), within groups of channels (group norm and instance norm), and what models actually use in 2026.