How to Prepare for an ML Interview in 2026

I am interviewing at ML companies this year, and the preparation splits cleanly into topics. Each topic here is a series I wrote while working through it: the questions an interviewer opens with, the derivation underneath, and the figures that make the answer stick. Start anywhere. More topics land as I get to them.

• SEP 2026

  topic 01 · 6 parts

  Interrogating the Transformer

  The questions ML interviews actually ask about transformers, answered from first principles: one pass through the block, why the square root is there, what residuals and norms buy you, how position gets in, what the FFN knows, and what changed since 2017.

  p.1one pass through the block p.2attention and the square root p.3residuals and the norm p.4where am i? p.5what the ffn knows p.6what changed since 2017
• SEP 2026

  topic 02 · 6 parts

  Interrogating Mixture of Experts

  What a Mixture of Experts actually is and why every frontier model became one: total versus active parameters counted on DeepSeek-V3, how a token picks its experts, why routers collapse and how three different mechanisms stop them, what an MoE layer costs in memory and network traffic, and what the field still has not settled in September 2026.

  p.1why make a model sparse p.2how a token picks its experts p.3keeping the experts busy p.4the bill for sparsity p.5what is unsolved p.6the question set
• SEP 2026

  topic 03 · 6 parts

  Interrogating the KV Cache

  The one object that decides how an LLM serves: why the cache exists, exactly how many bytes it is, every way people make it smaller, how vLLM and SGLang manage it, and what it does to tokens per second.

  p.1why the cache exists p.2how many bytes, exactly p.3making it smaller p.4how a server keeps it p.5what it does to throughput p.6the question set
• SEP 2026

  topic 04 · 7 parts

  Interrogating the GPU

  How a model actually meets the hardware: the four bottlenecks, the roofline, the memory hierarchy inside a GPU, where every byte goes during training and serving, FlashAttention from first principles, the September 2026 toolkit, and what to watch when building at scale.

  p.1where the time goes p.2the roofline p.3inside the memory hierarchy p.4where the bytes go p.5flashattention p.6the 2026 toolkit p.7building at scale
• SEP 2026

  topic 05 · 7 parts

  Interrogating Parallelism

  The five ways a model gets split across a cluster, derived one at a time: the two budgets that force it, data parallelism and the ZeRO ladder, tensor and sequence parallelism, pipeline schedules and their bubbles, context and expert parallelism, how the axes compose onto a real topology, and a graded question set.

  p.1why one gpu cannot do it p.2splitting the batch p.3splitting a matrix multiply p.4splitting the layers p.5long sequences and sparse models p.6putting them together p.7the question set

back to the notebook