Mixture of Experts · 3. Keeping the experts busy

A worked example, every number computed

Four tokens pick two of four experts each, and the load comes out 2, 2, 4, 0. Expert 4 never runs and never learns, expert 3 sets the pace, and an early lead compounds. Then three fixes: the Switch balancing loss (1.535 on the toy, 1 when even), a capacity factor that drops overflow, and DeepSeek-V3's bias that changes only the choice.

Open video ↗
Read the walkthrough

The load. Router scores s = x W_R; each token keeps its two highest. Picks per expert: 2, 2, 4, 0. Eight picks over four experts would be 2 each.

Why it hurts. Expert 4 sits in memory and never runs; on four GPUs the layer waits for expert 3's 4 runs instead of 2; no output contains f₄, so ∂L/∂W_E4 = 0; more tokens make an expert better, which wins it more tokens.

A loss. L_bal = α · N · Σ f_i P_i. f = picks / 8 = [0.25, 0.25, 0.5, 0]; P averages a softmax over all four experts: [0.214, 0.209, 0.556, 0.021]. N Σ f_i P_i = 4 × 0.384 = 1.535; 1 is even, 2 the worst for top-2 of 4. f is a count, so the push goes through P: for sat, expert 3's score is pushed down and the others up. Switch Transformer used α = 0.01.

A cap. C = CF · T k / N = 2 · CF. At CF = 1, sat's and down's picks of expert 3 are dropped and expert 4's 2 slots run empty; sat's output falls from [0.538, 3.462] to [0.538, 0.538]. CF = 2 drops nothing but runs 8 of 16 slots on padding.

A bias. Choose with Top2(s_i + b_i), weigh with s_i alone; after each batch b_i falls by γ if the expert is over the even load and rises if under. With γ = 0.6, two updates move cat to expert 4 (load 2, 2, 3, 1; its gates 0.881 and 0.119 come from its raw scores); two more overshoot to 2, 3, 0, 3. DeepSeek-V3 used γ = 0.001.

Three levers. A loss changes the gradient, a cap changes what is computed, a bias changes who is picked.