How does a token choose its experts?
Part 1 established the trade. This part is the mechanism, and it is the part an interviewer will make you write on a whiteboard, because there is nowhere to hide: either you know what shape the router is and what its output multiplies, or you do not.
The whole routing apparatus is one matrix. There is no separate network, no attention over experts, no learned assignment table, and no communication between tokens. A vector goes in, $N$ scores come out, the top $k$ win.
What makes the mechanism interesting is not its complexity. It is that a component holding 0.016% of a layer’s parameters decides where 100% of that layer’s capacity goes, and it makes that decision for one token at a time with no knowledge of what the rest of the batch is doing.
Everything difficult in Parts 3 and 4 is a consequence of that last sentence.
The 30-second version
The hidden state, a $d$-vector, is multiplied by a $d \times N$ router matrix to give one score per expert. Modern large models pass those through a sigmoid rather than a softmax, take the top $k$, and renormalise those $k$ scores so they sum to one, sometimes times a fixed scaling factor. Each chosen expert runs the full feed-forward computation on the token, and the outputs are summed with the renormalised scores as weights. Models with a shared expert run that one unconditionally and add it too. The gate is a single scalar per expert per token, and it is the only path by which the router's weights ever see the loss: the top-$k$ selection itself is a hard argmax with no gradient, so an expert that is never chosen for a token gets nothing from that token, and neither does the router's opinion about it. In a real implementation you do not run $k$ separate forward passes. You compute the routing for the whole batch, permute the tokens into expert order, run one grouped matrix multiply, and permute back.Step through the figure with the sliders. It runs on DeepSeek-V3’s actual shapes, and the two controls in the second step let you see the difference between a softmax gate and a sigmoid gate on identical logits.
What does the router actually compute?
One matrix-vector product. For DeepSeek-V3 the router is $7168 \times 256$, which is 1.84 million parameters and 3.67 megaflops per token.
Set that against what it commits the layer to. The eight chosen experts are 352 million parameters and 705 megaflops. The decision costs half a percent of the arithmetic it authorises.
The router sees one token. It does not see the sequence, it does not see the batch, and it certainly does not see what the other layers decided. Whatever coordination exists between routing decisions is emergent, not designed, and Part 3 is largely about the consequences.
Why did the large models switch from softmax to sigmoid?
The 2017 formulation and everything through Mixtral used a softmax over the expert scores. DeepSeek-V3’s config says scoring_func: sigmoid, and Kimi K2 and several others followed.
The reason is the size of the bank. A softmax spreads one unit of probability across all $N$ experts, so as $N$ grows every individual score shrinks. On the token in the figure, a softmax over 256 experts gives the winner about 5%; a sigmoid gives it about 0.75. The sigmoid scores each expert on its own merits and does not care how many others exist.
That has two consequences. Adding experts to the bank does not rescale every existing gate, which matters if you ever want to grow a model. And the scores stay in a range where the numerics are comfortable, rather than drifting toward a regime where an exponential of a drifting logit becomes a precision problem.
After the top-$k$ renormalisation the two look similar, because both get divided by their own sum. The differences live in the gradients, in what happens when $N$ changes, and in the fact that the softmax’s exponential exaggerates whatever gap the logits had.
What is the gate value and what does it multiply?
\[g_i = \frac{s_i}{\sum_{j \in \text{TopK}} s_j} \cdot \lambda, \qquad y = \sum_{i \in \text{TopK}} g_i \, \text{FFN}_i(x)\]The gate is one scalar per chosen expert. It multiplies that expert’s entire $d$-dimensional output before the sum. DeepSeek-V3 renormalises over the chosen $k$ and then multiplies by a fixed routed scaling factor of 2.5, both of which are in the published config.
Being precise about the order matters, because it is a common interview trip. Some models take the softmax over all $N$ and use those probabilities directly as gates; others take the top $k$ first and normalise within them. The second gives gates that sum to one regardless of how confident the router was, which throws away a signal and gains a scale that does not drift.
Notice what the gate is not. It is not a probability that gets sampled. It is not a mask. It is a deterministic weight in a weighted sum, and its entire job during the backward pass is to be the road the router’s gradient travels down.
Why does the router get so little gradient?
Because the only differentiable thing in the whole selection is those $k$ scalars.
The top-$k$ operation is a hard argmax. It has no useful derivative, so no gradient flows through the choice itself. What does flow is $\partial L / \partial g_i$ for the $k$ experts that were chosen, and from there back into the router’s weights. The 248 experts that were not chosen contribute nothing to the output and receive nothing from the backward pass, and the router learns nothing about whether one of them would have been better.
This is the exact reason routing is hard, and it has a sharp corollary worth having ready. If you use top-1 and renormalise the gate over the chosen set, the gate is identically 1, a constant, and the router receives no gradient at all from the output. It would learn only from the balancing loss.
Switch Transformer, which is the model that made top-1 respectable, avoids this by keeping the raw softmax probability as the multiplier rather than renormalising. Shazeer’s original 2017 formulation had argued you needed $k \ge 2$ for the router to get a meaningful signal; Switch’s answer was that you need $k \ge 2$ only if you throw the magnitude away.
Why 256 tiny experts instead of eight large ones?
This is the design move that separates the 2024 generation from Mixtral, and it comes from the DeepSeekMoE paper, where it is called fine-grained expert segmentation.
Hold the arithmetic fixed and cut each expert into $m$ narrower pieces, then select $m$ times as many. Active parameters are unchanged to the digit. FLOPs are unchanged. What changes is the number of distinct expert teams a token can assemble, and it changes by a lot: $\binom{8}{2}$ is 28, while $\binom{256}{8}$ is about $4 \times 10^{14}$.
The bet is that expressiveness in the combination is worth something, and the published models all took it. Mixtral had 8 experts and used 2. DeepSeek-V3 has 256 and uses 8. Kimi K2’s config lists 384 and uses 8. Qwen3-235B lists 128 and uses 8.
What it costs is not FLOPs. It is a router matrix that grows with $N$, a longer sort in the kernel, and $mk$ network destinations per token instead of $k$, which is the whole subject of Part 4. Fine-grained routing is cheap in arithmetic and expensive in systems work.
What is a shared expert for, and does it help?
DeepSeekMoE’s other idea: hold out one or two experts that run on every token, outside the top-$k$. The argument is that whatever every token needs, common syntax, the shape of the residual stream, does not need to be learned redundantly in all 256 experts, so isolating it frees the routed experts to be different from each other.
DeepSeek, Moonshot and Z.ai all ship one. Their published configs list n_shared_experts: 1 alongside 256, 384 and 160 routed experts respectively.
And it is genuinely contested. AI2 ran the matched-compute ablation for OLMoE and reported that sharing an expert performed slightly worse than using two routed ones, with a neat combinatorial argument: with 32 experts and 4 active, making one of them shared cuts the choices from $\binom{32}{4} = 35{,}960$ to $\binom{31}{3} = 4{,}495$, removing almost 90% of the combinations. Alibaba shipped shared experts in Qwen2.5-MoE and removed them in Qwen3.
If asked, the honest answer is that it is a small effect either way, that the labs disagree, and that it probably interacts with granularity, because the more experts you have the less a single always-on one costs you in combinations.
What happens when an expert runs out of room?
To run the experts as batched matrix multiplies you need fixed-size buffers, and you have to size them before you know the routing. That is what the capacity factor is for. Switch Transformer defines expert capacity as tokens per batch divided by the number of experts, times a capacity factor, and a token arriving at a full expert is simply dropped: its computation is skipped and it passes to the next layer through the residual connection.
Note what that means. A dropped token is not an error and does not raise an exception. It is a token that silently did not get a feed-forward layer, and it happens more the more skewed the routing is.
The capacity factor is the knob and it is a bad one. Too low and you drop tokens; too high and your buffers sit half empty and the matrix multiply is mostly padding.
The modern answer is to stop doing it. Block-sparse kernels, of which MegaBlocks is the reference, run variable-size expert groups directly and remove the capacity factor as a hyperparameter entirely. DeepSeek state plainly that V3 does not drop any tokens during training, and describe a deployment strategy that avoids dropping at inference too. If you are asked about capacity factors in 2026, the useful answer explains what they were for and why the frontier stopped needing them.
What does the kernel actually run?
Not $k$ forward passes. This is the misconception that a whiteboard question is designed to catch.
The sequence is: compute the router for the whole batch, take the top $k$, sort the token rows by expert index into one contiguous buffer, run a single grouped matrix multiply whose group boundaries are the expert offsets, then scatter the results back to token order and weight them by the gates.
The two permutations are pure data movement. For a batch of 8192 tokens with $k = 8$ and $d = 7168$ they move a few gigabytes per layer in BF16 and do zero arithmetic, which puts them at the far left of the roofline from Part 2 of the GPU series: bandwidth-bound, nothing else.
That is where a naive implementation loses most of its throughput, and it is why MegaBlocks, the Triton grouped-GEMM kernels and DeepSeek’s own DeepEP exist. It is also why the group sizes being unknown until the router has run is such a persistent problem: a kernel that wants static shapes has to wait for a number that arrives late, every layer, every step.
Rapid fire: can you do these from memory?
- Write the router, the top-k, the gate normalisation and the combine, in that order, with shapes.
- Why did large-bank models move from a softmax gate to a sigmoid one?
- Which quantity carries the router's gradient, and what receives no gradient at all?
- Why does top-1 with a renormalised gate give the router no gradient from the output, and what did Switch do instead?
- At fixed active parameters, what does splitting each expert into four change and what does it not?
- State the combinatorial argument against a shared expert.
- Define expert capacity, and say what happens to a token that overflows it.
- Describe what one MoE layer's kernel does, and why the permutations are the expensive part.
Part 3 is the failure mode: what happens to a router that nobody is supervising, and the three different mechanisms the field uses to stop it.