How do you stop one expert from eating everything?
Go back to the toy from Part 1: “the cat sat down”, four experts, and a router that sends each token to its two best. Count where the tokens went. Expert 1 gets two, expert 2 gets two, expert 3 gets all four, and expert 4 gets nothing.
Nothing crashes. Every token still gets two experts, and the task loss never mentions the load. That is what makes this failure dangerous: it is quiet.
But expert 4 sits in memory and never runs, so it never learns. Expert 3 does twice its share, so the layer waits for it. And the router’s only gradient flows through the experts it already chose, so an early lead compounds. Left alone, a router drifts toward a few winners.
The film counts that load, says what it costs, and then works three fixes on the same four tokens. Watch it first; the notation comes after.
The 30-second version
It is routing collapse, and it is the default, not a bug: gradient reaches only the chosen experts, so a favoured expert improves and gets chosen more. Three fixes act in three places. The Switch balancing loss, α · N · Σᵢ fᵢPᵢ, adds a gradient; f is a count with no gradient, so all the push goes through P, the mean router probability. A capacity factor caps each expert's slots and drops the overflow. DeepSeek-V3 adds a per-expert bias to the scores used for choosing only, nudged after every batch, so balance never enters the loss. Separately, the router z-loss keeps the logits small for numerical stability. And log the load per layer, every step.What do the three fixes compute, in symbols?
Write $T$ for the tokens in a batch, $N$ for the experts and $k$ for the picks per token. The toy has $T = 4$, $N = 4$, $k = 2$: eight picks, so an even load is 2 each.
The balancing loss comes from GShard and Switch Transformer:
\[L_{\text{bal}} = \alpha \, N \sum_{i=1}^{N} f_i \, P_i\]$f_i$ is the share of all $Tk$ picks that went to expert $i$. $P_i$ is the router’s probability for expert $i$, a softmax over all $N$ experts, averaged over the batch. In words: for each expert, the share of tokens it got times the probability the router gives it, summed. Both vectors sum to one, so the sum is smallest when both are flat, and the $N$ in front makes that floor exactly 1.
On the toy, $f = [0.25, 0.25, 0.5, 0]$ and $P = [0.214, 0.209, 0.556, 0.021]$, so before $\alpha$ the loss is 1.5351.535, worked outf × P per expert: 0.25 × 0.214, 0.25 × 0.209, 0.5 × 0.556, 0 × 0.021= 0.0535 + 0.0522 + 0.2780 + 0 = 0.3837N × 0.3837 = 4 × 0.3837 = 1.535Even routing gives 4 × 4 × (¼ × ¼) = 1. The worst a top-2 router over four experts can do is 2.. Switch used $\alpha = 0.01$, after sweeping it from 0.1 down to 0.00001.
The detail interviewers probe: $f$ is a count, so it has no gradient. All the push goes through $P$. For one token’s score on expert $j$, with $\alpha = 1$, it is
\[\frac{\partial L_{\text{bal}}}{\partial s_j} = \frac{N}{T}\, p_j \,\big(f_j - \bar f\big), \qquad \bar f = \sum_i f_i\, p_i\]Read it as: push a score down where the tokens went, scaled by how sure the router already is. For “sat”, $\bar f = 0.411$, so descent lowers expert 3’s score and raises the other three. Expert 4 barely moves: the router gives it 0.012.
The batch you count $f$ over matters too. Qiu and colleagues showed in January 2025 that counting over the global batch instead of each micro-batch improves both perplexity and expert specialisation, at up to 42.8 billion parameters and 400 billion tokens.
The capacity factor fixes each expert’s slots before the router runs: $C = \mathrm{CF} \cdot Tk/N$. At CF = 1 the toy gets 2 slots each, filled in batch order. “sat” and “down” find expert 3 full and their picks are dropped, so sat’s output falls to [0.538, 0.538]sat without expert 3full: 0.269 × [2, 2] + 0.731 × [0, 4] = [0.538, 3.462]dropped: 0.269 × [2, 2] = [0.538, 0.538]A dropped pick is not computed; the token rides the residual connection on.. CF = 2 drops nothing but runs 8 of its 16 slots on padding. ST-MoE recommends top-2 with CF 1.25.
The bias is DeepSeek-V3’s auxiliary-loss-free balancing. Each expert gets a number $b_i$ that is added to its score only for choosing:
\[\text{choose: } \mathrm{Top}k\,(s_i + b_i), \qquad \text{weigh: } g_i \text{ from } s_i \text{ alone}\]After each batch, $b_i$ falls by $\gamma$ if expert $i$ was over the even load and rises if under. With $\gamma = 0.6$ on the toy, two updates move “cat” from expert 3 to expert 4 and the load becomes 2, 2, 3, 1. Two more overshoot to 2, 3, 0, 3. Four tokens are simply too few to split evenly with one number per expert.
On real batches the step is tiny. DeepSeek-V3 used $\gamma = 0.001$ for its first 14.3T tokens and 0 for the last 500B, plus a sequence-wise balancing loss at $\alpha = 0.0001$. The Loss-Free Balancing paper reports, at 1B parameters and 100B tokens, a perplexity of 9.50 against the balancing loss’s 9.56 and a worst overload of 0.04 against 0.72. DeepSeek-V4 kept the method.
One more term is often confused with these. ST-MoE’s router z-loss penalises the squared log-sum-exp of the router logits, with coefficient 0.001. It is about numerical stability, not balance: it keeps the logits small. Switch Transformer attacked the same problem by computing the router in float32.
What do you watch during a run?
The training loss will not show collapse until long after it starts. Log the load histogram per layer, every step, because collapse usually begins in one layer. From it, track the busiest expert’s load over the even load, minus one: the toy starts at 1.0 and the bias brings it to 0.5. Count the experts with zero tokens, and watch the router logit magnitude.
And keep balance and meaning apart. A perfectly even router can still assign tokens at random. Balance is about using the hardware you paid for; whether an expert specialises is a separate question.
Rapid fire: can you do these from memory?
- Describe the feedback loop behind routing collapse in two sentences.
- Write the balancing loss, define f and P, and say what its value is under even routing.
- Which factor of the balancing loss carries the gradient, and why?
- Given T, N, k and a capacity factor, how many slots does each expert get, and what happens to the overflow?
- Where does DeepSeek's routing bias enter, and where does it deliberately not?
Part 4 asks what all of this costs to run: memory for every expert, two all-to-alls a layer, and why a sparse model is harder to decode.