Mixture of Experts · 5b. What is still unsolved: around the router

Every toy number computed; every model fact from its paper or config

Upcycled copies give the router nothing to learn from, and the two published break-even points sit 4.8 times apart. A 2026 law puts sparsity into the best learning rate and batch. DeepSeek-V4's experts do 6,144 FLOPs per byte on the wire, so the network hides.

Open video ↗
Read the walkthrough

Upcycle or start over? With every expert a copy of FFN₂, y = 0.731 · [2, 2] + 0.269 · [2, 2] = [2, 2] for any pair, and ∂y/∂s₃ = g₃g₂(f₃ − f₂) = 0. From scratch caught up after about 120% of the dense budget in Google’s sparse upcycling and after 500B of 2,000B tokens (25%) in OLMoE.

Do the settings transfer? η* = 0.8343 C^-0.1385 A^0.1361 and B* = 6.4765 D^0.5181 A^-0.0841, from 1,800 runs (arXiv 2609.08690). At A = 1/64 the factors are 0.57 on the learning rate and 1.42 on the batch.

Where is the bottleneck? 6hd ÷ 3h = 2d; with h = 7,168 and d = 3,072, 132,120,576 ÷ 21,504 = 6,144 FLOPs per byte, 6.1 TFLOP/s per GB/s. At a million tokens V4-Pro needs 27% of V3.2’s FLOPs per token and 10% of its KV cache.