What is still unsolved about Mixture of Experts?

The first four parts describe a mechanism that looks settled. A router scores the experts, each token keeps its top few, a balancer keeps the load even, and expert parallelism spreads the bank over the machines.

Then, in April 2026, the lab whose recipe everyone had copied changed it. DeepSeek-V4 swapped its score function for a new one, routed its first three layers by a hash of the token id, and dropped the limit on how many machines a token’s experts may sit on.

None of those changes came from a theory. They came from runs that worked. That is the honest state of the subject, and it is where the hardest interview questions live: the ones where a confident answer is a bad sign.

So this part is six open questions, each told in three beats: the question, what has been tried, and why it is still open. Two films carry them, on the same four-word toy as the rest of the series. Every model fact in them comes from the model’s paper or config.

They will ask What is still unsolved in Mixture of Experts, and what would worry you if you had to build one?
The 30-second version The mechanism is settled; the reasons behind its parts are not. The router changes every year (softmax, then sigmoid, then DeepSeek-V4's square root of a softplus and hash routing) and nobody has derived any of it. Top-k is discontinuous, so under reinforcement learning about 10% of activated experts change after each update on a 30B model, and the fixes only pin the routing in place. Whether experts specialise depends on how the model was started and what the balancer averages over. The two published break-even points for upcycling sit 4.8 times apart. And a 2026 study of 1,800 runs found the best learning rate and batch size move with sparsity, so a learning rate tuned on a dense proxy comes out too high. What would worry me most is the last two, because they fail quietly.

Which questions about the router are still open?

The first film opens on one chart: from Mixtral to DeepSeek-V4-Pro, stored parameters grew 34 times34 and 3.8, worked outstored: 1,600B ÷ 47B = 34.0used per token: 49B ÷ 13B = 3.8Mixtral 8x7B (2023) against DeepSeek-V4-Pro (2026). while the parameters each token uses grew under 4 times. Then it takes three questions about the router itself: who should choose, whether the choice holds still while the model trains, and whether the experts it chooses between mean anything.

What is open around the router?

The second film takes the three questions around it. Should you start from a dense checkpoint? Do hyperparameters tuned on small models survive sparsity? And where does the bottleneck go once the network is hidden? It ends on a table of all six.

What do the open problems look like in symbols?

The toy is the one every part uses: four words, router scores $S = XW_R$, two experts per word.

Who chooses. Expert choice gives every expert a fixed bucket of words:

\[c = \frac{n\,k}{N} = \frac{4 \cdot 2}{4} = 2\]

Here $n$ counts words, $k$ experts per word and $N$ experts. In words: each expert takes exactly its fair share, so the load is even by construction. The price in the toy is that “the” gets one expert and “sat” three, and expert 3 picks “sat” and “down” over the earlier “cat”, which a decoder cannot know about yet.

The score function. Every learned router in the films normalises some rising function $\phi$ of the chosen scores:

\[g_i = \frac{\phi(s_i)}{\sum_{j \in \text{top-}k} \phi(s_j)}\]

For “sat”, with scores 3 and 2, expert 3’s gate is 0.731 with $\phi = \exp$ (Mixtral), 0.520 with a sigmoid (DeepSeek-V3) and 0.545 with the square root of a softplus (DeepSeek-V4). In words: the same two experts, and three different opinions on how much each one counts.

The flip. An illustrative update adds 0.6 to one router weight, and “sat”’s score for expert 1 becomes $1 \cdot 1 + 2 \cdot 0.6 = 2.2$, past expert 2’s 2. One choice in eight flips: 12.5%. The GSPO paper measured about 10% of activated experts changing per update on Qwen3-30B-A3B.

Upcycled copies. With two experts and a softmax over them,

\[\frac{\partial y}{\partial s_3} = g_3\, g_2\, (f_3 - f_2)\]

In words: the router learns from the gap between the experts it chose. Copies of one dense FFN have no gap, so the push is $[0, 0]$. The toy’s own experts give $0.197 \cdot [-2, 2] = [-0.39, 0.39]$.

Hyperparameters across sparsity. With $A$ the share of routed experts a token uses, the September 2026 fits are

\[\eta^* = 0.8343\, C^{-0.1385} A^{0.1361}, \qquad B^* = 6.4765\, D^{0.5181} A^{-0.0841}\]

In words: the best learning rate falls with compute and with sparsity, and the best batch grows with tokens and with sparsity. At one expert in 64, DeepSeek-V4-Pro’s ratio, that is 0.57 times0.57, worked outA^0.1361 with A = 1/64ln 64 = 4.159, and 0.1361 × 4.159 = 0.566e^(−0.566) = 0.57Same compute, compared with a dense model (A = 1). the dense learning rate and 1.42 times1.42, worked outA^(−0.0841) with A = 1/640.0841 × 4.159 = 0.350e^0.350 = 1.42Same tokens, compared with a dense model (A = 1). the batch.

Arithmetic per byte. Per token and expert, DeepSeek-V4 counts $6hd$ FLOPs (a SwiGLU’s three matrices) against $3h$ bytes on the wire (FP8 out, BF16 back), so

\[\frac{6hd}{3h} = 2d = 6{,}144 \ \text{FLOPs per byte}\]

for V4-Pro’s experts, $d = 3{,}072$ wide. In words: each GB/s of interconnect hides 6.1 TFLOP/s6.1 TFLOP/s, worked out6hd ÷ 3h = 2d = 2 × 3,072 = 6,144 FLOPs per byte6,144 × 10⁹ bytes per second = 6.1 × 10¹² FLOP/sV4-Flash’s 2,048-wide experts: 4,096 FLOPs per byte, two thirds as much. of expert arithmetic, and narrower experts hide less.

question what has been tried still open because
Which router? expert choice; hash routing (Hash Layers, V4’s first 3 layers); softmax, sigmoid, √softplus token choice still wins on quality, and no theory picks the gate
Does routing hold still? routing replay, GSPO’s sequence-level ratio, R3 every fix pins the jump instead of removing it
Do experts mean anything? routing studies of Mixtral, ST-MoE, OLMoE; global-batch balancing the answer depends on how the model started and what the balancer averages
Upcycle or start over? sparse upcycling (120%), OLMoE (25%), Drop-Upcycling, upcycling scaling laws the break-even point moves 4.8 times4.8, worked outOLMoE: 500B ÷ 2,000B tokens = 25%120% ÷ 25% = 4.8Google’s 120% is from T5 with expert choice. with the setting
Do the settings transfer? a 1,800-run law with $A$ as a factor fitted to 6B and checked at 12B; the frontier stores 1,600B
Where is the bottleneck? V4’s overlap of transfer and arithmetic; compressed attention it moves with the network, the expert width and the overlap

Rapid fire: can you do these from memory?

  1. What does expert choice buy, and why does it leak information in a decoder?
  2. Why can a softmax, a sigmoid and a square root of a softplus choose the same experts but weigh them differently?
  3. What is routing replay, and what does it cost?
  4. Why does the router of a freshly upcycled model get no gradient from the output?
  5. Which way do the best learning rate and batch size move as a model gets sparser, and by how much at one expert in 64?