Can you answer these without looking anything up?

Five parts of films build an understanding; this part tests it. Three levels: the mechanism (shapes, what is computed and stored), the trade-offs, where most interviews live, and the frontier, where a good answer names what the question turns on and then commits.

Each sketch is what a strong answer contains, and every number in it comes from Parts 1 to 5.

They will ask Tell me about Mixture of Experts.
The 30-second version Pick a frame and say why. Mine: MoE separates parameter count from arithmetic per token, buying capacity with memory instead of compute. Then three consequences: a hard top-k leaves most experts without gradient, so balancing needs its own mechanism; experts on other devices turn a layer into two all-to-alls; and a decoding batch touches nearly every expert, reading all the weights for little work. Each comes with a number I can work out.

Level 1: can you state the mechanism?

level 1the mechanism
  1. Mixtral 8x7B is eight seven-billion-parameter experts, and it has 47 billion parameters rather than 56. Where did the other nine billion go?

    answer sketch

    They were never there: only the feed-forward sublayer is copied eight times. Attention, the norms, the embeddings and the output head exist once and serve every token, so eight whole 7B models double-count everything outside the FFN.

  2. A token arrives at an MoE layer as a vector of length $d$. Name every tensor computed on the way to the layer's output, with its shape.

    answer sketch

    Router scores, $N$ values from a $d \times N$ product; a top-$k$ pick of $k$ indices, with no gradient; $k$ gates from renormalising the chosen scores. Each chosen expert's SwiGLU returns a $d$-vector, and their gate-weighted sum, plus any shared expert, leaves as a $d$-vector: the shape it arrived in.

  3. During the backward pass of an MoE layer, which quantities receive gradient and which receive none?

    answer sketch

    The chosen experts get gradient through their outputs, scaled by their gates, and the router only through those $k$ gates, since the top-$k$ pick has no derivative. Unchosen experts get nothing, and the router never learns whether one of them was better, which is why balancing lives outside the task loss.

  4. Given 61 layers of which the first 3 are dense, $d = 7168$, experts of width 2048, 256 routed plus 1 shared, top-8, and a 129k vocabulary with untied embeddings: compute total and active parameters.

    answer sketch

    One SwiGLU expert is $3 \times 7168 \times 2048$, about 44 million; an MoE layer holds 257 of them plus a $7168 \times 256$ router, about 11.3 billion, of which about 398 million run per token. Fifty-eight such layers give 657 billion stored and 23 billion active; the rest of the model brings it to DeepSeek-V3's published 671 billion and 37 billion.

  5. Does top-8 routing mean eight forward passes through the layer? Describe what a real kernel does instead.

    answer sketch

    No. Route the whole batch, sort token rows by expert into one buffer, run one grouped matrix multiply split at the expert offsets, then scatter back and weight by the gates. The two permutations do no arithmetic but move many bytes, which is where a naive implementation loses its throughput.

  6. Define expert capacity and the capacity factor, and say precisely what happens to a token that overflows.

    answer sketch

    Capacity is tokens per batch over experts, times the capacity factor, fixed before routing so the experts run as fixed-shape multiplies. A token reaching a full expert is dropped: its expert work is skipped and it passes on through the residual, silently. Dropless kernels with variable group sizes remove the factor.

  7. Write the auxiliary load-balancing loss, define both of its vectors, and give its value at perfect balance and at total collapse.

    answer sketch

    $L = \alpha N \sum_i f_i P_i$, with $f_i$ the fraction of tokens sent to expert $i$ and $P_i$ its mean router probability. Both sum to one, so the factor $N$ makes the minimum, at perfect balance, exactly 1; collapse onto $k$ experts gives $N/k$, which is $N$ for Switch Transformer's top-1. Switch used $\alpha = 0.01$.

  8. Where does DeepSeek's per-expert bias enter the computation, and where does it deliberately not?

    answer sketch

    It is added to the affinity only for the top-$k$ comparison; the gate that weights the output uses the original affinity. So it changes which experts are chosen without changing how much they count, and adds no loss term. Each step moves it by a small constant, down for busy experts and up for idle ones.

Level 2: can you reason about the trade-offs?

level 2the trade-offs
  1. At fixed active parameters, would you rather have 8 experts of width $W$ or 64 of width $W/8$? What changes and what does not?

    answer sketch

    Active parameters and FLOPs per token are identical by construction; what grows, combinatorially, is the number of expert teams a token can assemble. The cost is systems work, not arithmetic: a router eight times larger, a longer sort, and more destinations per token in the all-to-all.

  2. Does a Mixture of Experts reduce the memory needed to serve a model of a given quality?

    answer sketch

    No: every expert must sit in HBM, because routing is decided per token in the middle of the forward pass, with no time to fetch weights. Memory tracks total parameters; only speed tracks active ones. Offloading cold experts works only near batch one, as the next question shows.

  3. A 256-expert top-8 model decodes 128 tokens together. How many experts does one layer read, and what does that do to arithmetic intensity?

    answer sketch

    Tokens pick independently, so a layer reads $N(1 - (1 - k/N)^B)$ experts, about 252 of 256. Bytes scale with $N$ and FLOPs with $k$, so the intensity is worse than a dense model holding the same weights by a factor that settles at $N/k = 32$. To reach the H100 ridge of 295 the layer needs about 9,450 tokens in flight, against 295 for a dense one.

  4. Your training run is dropping 15% of tokens. A colleague proposes doubling the capacity factor. Why is that the wrong first move?

    answer sketch

    Dropping is a symptom of skew; the capacity factor only decides how much skew you buffer, so doubling it fills the grouped multiply with padding and hides the problem. Read the per-layer load histogram first, then check the balancer: is it on, is it per layer, and over which batch are the fractions averaged?

  5. At step 40,000 your MoE's loss curve looks healthy. What could already be badly wrong, and how would you know?

    answer sketch

    Router collapse: a few experts doing all the work still make a working model, just a smaller one than you pay for, so the loss hides it. Log each layer's load histogram and its count of idle experts every step: collapse usually begins in one layer, so a model-wide average misses it.

  6. You profile a training step and the all-to-all is 2.5 times the expert compute. List the levers, ordered by how much they buy.

    answer sketch

    Overlap first, because it turns a sum into a maximum: run another micro-batch's arithmetic under this one's transfer. Then shrink the payload with FP8 dispatch and BF16 combine, then bound the fan-out with node-limited routing. A bigger batch does not help: traffic and arithmetic both scale with tokens.

  7. Why does top-1 routing with a renormalised gate leave the router with no gradient from the output, and what are the two ways out?

    answer sketch

    Renormalising over one expert makes its gate identically 1, so the task loss gives the router no gradient and only the balancing term trains it. Keep the raw probability as the multiplier, as Switch Transformer did, or use $k \ge 2$, as nearly every large model now does.

  8. A colleague computes the balance loss per micro-batch instead of over the global batch, to save an all-reduce. What does that change about the model, not just the speed?

    answer sketch

    Same formula, different scope: a micro-batch is a few sequences, so the term asks every sequence to spread over the whole bank, against specialisation. The global batch costs one all-reduce of an $N$-vector, and Qiu and colleagues (2025) found better perplexity and more specialisation with it, in models up to 42.8 billion parameters.

Level 3: can you defend a design decision?

level 3the frontier
  1. Argue against Mixture of Experts. You have a fixed compute budget per token and you may build either a sparse model or a dense one with the same active parameters.

    answer sketch

    The dense model needs a fraction of the memory and has static shapes, no all-to-all, no routing to collapse and no balancer to tune. In decoding a sparse layer needs about 32 times the batch to use its GPUs as well, since 128 tokens already read 252 of 256 experts. When memory, not compute, is the limit, dense is the right answer.

  2. Shared expert or no shared expert? What would change your mind?

    answer sketch

    For: what every token needs is learned once, and DeepSeek and Kimi K2 ship one. Against: an always-on expert removes combinations, OLMoE's matched-compute ablation came out slightly against it, and Qwen3 dropped its shared experts. With hundreds of experts the lost combinations cost little, so granularity would move me.

  3. Auxiliary loss or the per-expert bias controller? What does the evidence actually say?

    answer sketch

    The Loss-Free Balancing paper reports perplexity 9.50 against 9.56 and a worst overload of 0.04 against 0.72, because the bias adds no gradient unrelated to the task. But that is one comparison at 1B parameters, and DeepSeek still ship a small sequence-wise loss beside the bias. A good default, on thinner evidence than its adoption suggests.

  4. A product manager says your MoE should have a medical expert and a legal expert. What do you tell them?

    answer sketch

    That subject experts do not reliably emerge. Mixtral's routing looks the same for arXiv, PubMed and philosophy and tracks syntax and neighbouring tokens instead, and ST-MoE found no decoder specialisation and no language experts. OLMoE, trained from scratch, does find domain experts: a medical expert is something you would impose, not something to expect.

  5. You have a strong dense checkpoint and budget for more training. Upcycle it into an MoE, or start over?

    answer sketch

    It turns on the remaining budget, and the published break-even points sit 4.8 times apart: about 120% of the dense budget in Google's sparse upcycling, 25% in OLMoE. Copied experts also give the router no signal until they drift apart, and they inherit the dense model's settings. Upcycle for a short run; start over for a long one.

  6. Reinforcement learning on your MoE diverges. The dense model of the same active size is fine. Diagnose it.

    answer sketch

    Routing first: top-$k$ is discontinuous, so small differences between the old and new policy, or the rollout and training engines, change which experts run, and the importance ratio compares two networks. Qwen measured about 10% of activated experts changing per update on a 30B model. Log how often routing differs, then try routing replay, R3 or GSPO.

  7. DeepSeek-V4 removed the node-limited routing that V3 needed. What has to be true for that to be a good idea, and would it be a good idea on your cluster?

    answer sketch

    The all-to-all must hide under compute, since the cap only bounded exposed traffic. DeepSeek counts $6hd$ FLOPs against $3h$ bytes per token and expert: $2d = 6{,}144$ FLOPs per byte for V4-Pro, so each GB/s of network hides 6.1 TFLOP/s. If your GPUs compute more per byte of network, or your experts are narrower, keep the cap.

  8. You must tune a model that activates one parameter in sixty-four, and you can afford only a dense proxy sweep. What do you do, and what do you tell your manager to expect?

    answer sketch

    Expect the transfer to be wrong: a 2026 study of 1,800 runs found the best learning rate falls and the best batch grows with sparsity, by 0.57 and 1.42 times at one expert in 64, and neither parameter count predicts it. Bracket with the dense sweep, then sweep at the target sparsity on the smallest model you can afford.

What do you do with a list like this?

Not memorise it. Each sketch carries one load-bearing idea; with the idea, the sentences come in the room.

Level 1 is arithmetic from the config alone. Level 2 nearly always starts with “at fixed what?”. Level 3 rewards naming the axis the answer turns on and then committing to a default anyway.