How does a token choose its experts?

A Mixture of Experts swaps the feed-forward network for a bank of experts, and something has to decide which experts each token visits. That something is the router, and it is the part an interviewer makes you write on a whiteboard: what shape it is, what its output multiplies, and where its gradient goes.

The whole apparatus is one matrix. No separate network, no lookup table, no talk between tokens. A row goes in, one score per expert comes out, the top $k$ win.

In DeepSeek-V3 that matrix is about 0.016% of an MoE layer’s weights, and it decides where all the rest go, one token at a time, blind to the rest of the batch. Parts 3 and 4 are the consequences of that last clause.

The film works it out on the toy from Part 1: four tokens, “the cat sat down”, two illustrative features each, four experts, top-2.

They will ask Walk me through a single token going through one MoE layer, from hidden state to output.
The 30-second version The hidden state, a d-vector, is multiplied by a d × N router matrix to give one score per expert. Mixtral takes a softmax over the top k scores; DeepSeek-V3 passes every score through a sigmoid, takes the top k, renormalises them to sum to one and multiplies by a fixed 2.5. Each chosen expert runs its full FFN on the token, and the outputs are added with the gates as weights; a shared expert, if the model has one, runs on every token and is added too. The gate is the router's only road to the loss: top-k is a hard choice with no gradient, so an expert that is not chosen for a token teaches the router nothing about that token. In a real kernel you do not run k separate passes: you route the whole batch, sort the tokens by expert, run one grouped matrix multiply, and put them back.

What does the router compute, in symbols?

Stack the tokens as rows of $X$, $n \times d$. The router $W_R$ is $d \times N$, one column per expert:

\[S = XW_R\]

Read it as: row $i$, column $j$ is how well token $i$’s features match what expert $j$ looks for. “sat” is $[1, 2]$ and scores $[1, 2, 3, -1]$. In DeepSeek-V3, $W_R$ is 7,168 × 256DeepSeek-V3’s router, worked out7,168 × 256 = 1,835,008 weights, 3.67 million FLOPs per tokenthe 8 chosen experts: 8 × 3 × 7,168 × 2,048 = 352 million weights, 705 million FLOPs3.67 ÷ 705 = 0.5%The decision costs half a percent of the work it commits the layer to..

Each row then keeps its $k$ largest scores. That is a hard yes or no: nudge a score and nothing changes until it overtakes a winner. In the toy, expert 3 is chosen by all four tokens and expert 4 by none, and nothing in this step notices.

The kept scores become gates:

\[g_i = \frac{e^{s_i}}{\sum_{j \in \mathrm{TopK}(s)} e^{s_j}}, \qquad y = \sum_{i \in \mathrm{TopK}(s)} g_i\,\mathrm{FFN}_i(x)\]

Read it as: a softmax over the chosen $k$ only. For “sat” that is 0.731 for expert 3 and 0.269 for expert 2. A softmax over all four experts, renormalised over the chosen two, gives the same gatesTwo recipes, worked oute to each score: 2.718, 7.389, 20.086, 0.368; total 30.561all four: 0.089, 0.242, 0.657, 0.012kept: 0.657 + 0.242 = 0.8990.657 ÷ 0.899 = 0.731, 0.242 ÷ 0.899 = 0.269The four-way total cancels when you renormalise., because the four-way total cancels. Skip the renormalising and the gates add to 0.899: the output shrinks with the router’s doubt. Switch Transformer routes to one expert and keeps that raw probability as its gate.

DeepSeek-V3’s config says scoring_func: sigmoid:

\[g_i = \lambda\,\frac{\sigma(s_i)}{\sum_{j \in \mathrm{TopK}(s)} \sigma(s_j)}, \qquad \lambda = 2.5\]

Read it as: score each expert on its own, keep the top $k$, normalise those, then scale. On “sat” the same two experts win with gentler gates, 0.520 and 0.480. A softmax shares one unit among all $N$ experts, so 256 equal scores get 0.0039 each and adding experts rescales every gate; a sigmoid does not care how many experts exist.

Why does the router get so little gradient?

Because the only differentiable thing in the selection is the $k$ gates. The output is each gate times its expert’s output, so each gate’s slope is that expert’s output, and for a two-way softmax $\partial g_3 / \partial s_3 = g_3 g_2$:

\[\frac{\partial y}{\partial s_3} = g_3 g_2\,\big(\mathrm{FFN}_3(x) - \mathrm{FFN}_2(x)\big), \qquad \frac{\partial y}{\partial s_1} = \frac{\partial y}{\partial s_4} = 0\]

Read it as: raising $s_3$ pulls the output toward expert 3’s answer and away from expert 2’s, by 0.197 · [−2, 2]The slope, worked outg₃g₂ = 0.731 × 0.269 = 0.197FFN₃(x) − FFN₂(x) = [0, 4] − [2, 2] = [−2, 2]0.197 × [−2, 2] = [−0.39, 0.39]The unchosen scores get exactly zero..

Each column of the router gets the token’s row times its own score’s gradient, so columns 1 and 4 learn nothing from “sat”. A router that stops choosing an expert stops hearing about it, which is why balancing has to come from outside the loss (Part 3).

The sharp corollary: with top-1 and a renormalised gate, the gate is $e^{s}/e^{s} = 1$, a constant, and the router gets no gradient from the output at all. Shazeer’s 2017 paper conjectured that routing needs $k \ge 2$; Switch Transformer kept the softmax probability over all experts as the gate instead, 0.657 here, whose slope survives with one expert.

What did DeepSeek change, and what does the kernel run?

Two things, both from the DeepSeekMoE paper. First, many narrow experts: Mixtral picks 2 of 8, which is 28 pairs, while DeepSeek-V3 picks 8 of 256, about $4.1 \times 10^{14}$ teams, at similar arithmetic per token. Kimi K2’s config lists 384 routed experts and Qwen3-235B’s 128, both with 8 per token.

Second, a shared expert that every token visits, outside the top-$k$, with no gate. DeepSeek-V3, Kimi K2 and GLM-4.5 each ship one. Qwen3’s report says its MoE design excludes them, and AI2’s matched test for OLMoE found one shared expert slightly worse: with 32 experts and 4 active, sharing one cuts the combinations from 35,960 to 4,495.

Older MoEs sized each expert’s buffer with a capacity factor: tokens per batch divided by the number of experts, times a factor. A token arriving at a full expert was dropped and passed on through the residual. DeepSeek-V3’s report says it drops no tokens during training.

And the kernel never runs $k$ separate passes. It routes the whole batch, sorts the token rows by expert, runs one grouped matrix multiply, and scatters the results back, weighted by the gates. The two sorts are pure data movement, and the group sizes are unknown until the router has run.

Rapid fire: can you do these from memory?

  1. Write the router, the top-k, the gates and the combine in order, with shapes.
  2. Why do a softmax over the chosen k and a renormalised softmax over all N give the same gates?
  3. Which quantity carries the router's gradient, and what receives none at all?
  4. Why does top-1 with a renormalised gate give the router no gradient, and what did Switch do instead?
  5. State the combinatorial argument against a shared expert.