The Math Under the Model

drag everything, apr 2024

Every time I take a new technique apart I find old math inside it. LoRA is a low-rank factorisation, Muon orthogonalises a gradient, RoPE is a rotation, FlashAttention is a softmax rearranged so it fits on chip. Coding agents write most of the code around these ideas now, which makes it easy to let the math fade, and the math is exactly what you need to check what the agent wrote. Each topic here is a series. Every part opens on a question worth getting wrong first, the figures are meant to be dragged rather than watched, and where a part has code, it runs twice: once in the page, and once as a PyTorch notebook in Colab.

back to the notebook