Mixture of Experts · 5a. What is still unsolved: the router

Every toy number computed; every model fact from its paper or config

Stored parameters grew 34 times from Mixtral to DeepSeek-V4-Pro while the parameters used per token grew under 4 times. Then three open questions about the router, each on the toy: who should choose, whether routing holds still under training, and whether experts mean anything.

Open video ↗
Read the walkthrough

Grown in storage. Mixtral 8x7B (2023): 47B stored, 13B per token, 3.6×; DeepSeek-V3 (2024): 671B stored, 37B per token, 18.1×; Kimi K2 (2025): 1,000B stored, 32B per token, 31.2×; DeepSeek-V4-Pro (2026): 1,600B stored, 49B per token, 32.7×.

Who chooses. Token choice sends the toy’s words to experts [2, 2, 4, 0] times. Expert choice gives each expert c = n k ÷ N = 4 · 2 ÷ 4 = 2 words: loads [2, 2, 2, 2], but “the” gets 1 expert and “sat” 3, and expert 3 picks “sat” and “down” over the earlier “cat”. A hash of illustrative token ids also balances, with nothing to learn. On “sat” (scores 3 and 2) the gates are 0.731 with a softmax (Mixtral), 0.520 with a normalised sigmoid (DeepSeek-V3) and 0.545 with a normalised √softplus (DeepSeek-V4).

Does routing hold still? An illustrative update adds 0.6 to one router weight; “sat”’s score for expert 1 goes from 1 to 2.2 and it swaps expert 2 for expert 1: 1 of 8 choices, 12.5%. The GSPO paper measured about 10% of activated experts changing per update on Qwen3-30B-A3B; routing replay, GSPO and R3 pin the routing in place.

Do experts mean anything? Mixtral: routing looks the same across arXiv, PubMed and philosophy; consecutive tokens share a first-choice expert 23.6 to 28.4% of the time at layer 15 against 12.5% by chance. ST-MoE: encoder experts specialise, decoder experts do not, no language experts. OLMoE, trained from scratch, finds subject experts. Balancing over the global batch raised specialisation.