Mixture of Experts · 4. The bill for sparsity
The toy on four GPUs, then DeepSeek-V3's numbersAn MoE stores every expert but runs only k per token. The toy stores 56 weights and runs 32; DeepSeek-V3 stores 671B and runs 37B. Spread over GPUs, tokens travel to their experts and back, the busiest GPU sets the pace, and decoding needs about 32 times the batch of a dense model.
Open video ↗Read the walkthrough
Two counts. Stored = N · P_e + P_R = 4 · 12 + 8 = 56. Used by one token = k · P_e + P_R = 2 · 12 + 8 = 32. Memory follows the first, arithmetic the second.
Memory. The router decides per token, mid-layer, so every expert stays resident. DeepSeek-V3: 671B parameters, 671 GB in FP8, more than one eight-GPU H100 node (640 GB); 37B used per token, 2 × 37B = 74 GFLOP.
Experts on GPUs. One expert and one token per GPU. Dispatch sends each token to its two experts: 5 of 8 copies cross GPUs. The send matrix's column sums are the load, 2, 2, 4, 0. Combine sends results home: sat gets 0.269 · [2, 2] from GPU 2 and 0.731 · [0, 4] locally, [0.538, 3.462].
The wire. Per token and expert: 6hd FLOPs against 3h bytes (FP8 out, BF16 back), so 2d = 4,096 FLOP per byte in DeepSeek-V3. An H100 at 989 TFLOP/s needs 241 GB/s; NVLink gives 450, a network port 50, where a pick takes 430 ns to move and 89 ns to compute.
Stragglers. On four GPUs the work is 2, 2, 4, 0 runs: the layer ends at 4 and half the GPU time idles. A copy of expert 3 on the idle GPU splits its runs 2 and 2 and halves the time.
Decoding. B tokens touch N(1 − (1 − k/N)^B) experts: 8, 163 and 252 of 256 at B = 1, 32, 128. In BF16 a dense layer reaches the H100 ridge of 295 FLOP per byte at 295 tokens; V3's routed experts need about 9,450, which is N/k = 32 times more.
The bill. Sparsity saves arithmetic, not memory, and pays in traffic.