Knowledge Gradient Ascent

Working notes on what I am currently learning: generative models, efficient ML systems, and the occasional fight with a GPU cluster. I write them to understand things properly, and leave them here in case they are useful to you.

• SEP 2026

  entry 05 · 5 topics

  How to Prepare for an ML Interview in 2026

  A running syllabus for ML interviews: the questions that actually get asked, answered from first principles, one topic at a time, with the diagrams and animations I wanted while I was preparing.

  Interrogating the Transformer6 parts Interrogating Mixture of Experts6 parts Interrogating the KV Cache6 parts Interrogating the GPU7 parts Interrogating Parallelism7 parts
• MAY 2026

  entry 04 · 5 parts

  Catching up on Flow based LLMs

  A five-part build from first principles: why generation is transport, how it survives contact with discrete tokens, and how flow matching, diffusion, and masked diffusion turn out to be one framework.

  p.1generation as transport p.2tokens on the simplex p.3jumps, not velocities p.4it was one process all along p.5how fast, and how far
• 22 MAR 2026

  entry 03

  From DenseFormer to AttnRes: The Convergent Evolution of Depth-Wise Attention

• MAR 2026

  entry 02 · 3 parts

  Distributed Training from Scratch

  A three-part, from-scratch tour of training large models across many GPUs: the batch-size hierarchy, data and model sharding with DDP and FSDP, and what actually breaks at 256 GPUs.

  p.1the batch size hierarchy p.2ddp, fsdp, and the data problem p.3surviving 256 gpus on frontier
• 1 OCT 2023

  entry 01

  SSH Keys Demystified: A Researcher's Guide to Cluster Access

  Picture this. You’ve just been granted access to a shiny HPC cluster. Four GPUs waiting for you. You open your terminal, type ssh cluster: