Mixture of Experts · 2. How a token picks its experts

A worked example, every number computed

S = X W_R scores four tokens against four experts. Each row keeps its top two; for “sat” a softmax over the chosen two gives 0.731 and 0.269, the same as a softmax over all four renormalised. The output is [0.538, 3.462], and only the chosen gates carry gradient back to the router.

Open video ↗
Read the walkthrough

Scores. S = X W_R: one row per token, one column per expert. “sat” = [1, 2] scores 1 · 1 + 2 · 1 = 3 for expert 3, and [1, 2, 3, −1] in all. Tokens per expert after top-2: [2, 2, 4, 0].

Gates. Softmax over the chosen two: e³ ÷ (e³ + e²) = 0.731. Softmax over all four gives [0.089, 0.242, 0.657, 0.012]; the kept two hold 0.899, and dividing by that gives 0.731 and 0.269 again. Without renormalising the gates add to 0.899 (Switch keeps the raw probability).

DeepSeek-V3. A sigmoid per expert, [0.731, 0.881, 0.953, 0.269], top-2, normalised to 0.520 and 0.480, then times 2.5. A softmax over 256 equal scores gives each 0.0039; a sigmoid of 0 is 0.5 however many experts there are.

The mix. y = 0.731 · [0, 4] + 0.269 · [2, 2] = [0.538, 3.462].

The gradient. ∂g₃/∂s₃ = g₃g₂ = 0.197, so ∂y/∂s₃ = 0.197 · [−2, 2]; ∂y/∂s₁ = ∂y/∂s₄ = 0, and columns 1 and 4 of W_R learn nothing from this token. Top-1 renormalised gives a gate of 1 and no slope; Switch's raw probability 0.657 keeps a slope of 0.225.

Narrow and shared. 8 choose 2 = 28 pairs against 256 choose 8 ≈ 4.1e+14 teams. A shared expert runs for every token outside the top-k: DeepSeek-V3, Kimi K2 and GLM-4.5 ship one, Qwen3 none; OLMoE's test found it slightly worse (35,960 → 4,495 combinations).