What does an MoE cost to train and to serve?

The toy from Part 1 has four experts of 12 weights each and a router of 8. It stores 56 weights. Any one token runs 32 of them: the router and its two experts. Every sparse model has these two counts, and the whole systems story is which bill each one pays.

A dense layer’s weights are the same for every token, so you can shard them however you like and the tokens never move. An MoE layer’s weights are chosen per token, mid-layer, and the chosen ones usually live on another GPU. So the tokens move instead.

That one change has four consequences. Memory is set by the total. Every layer runs two all-to-alls whose sizes the router decides at run time. The busiest GPU sets the pace. And during generation, a small batch reads nearly the whole bank to do a little arithmetic.

The GPU series drew expert parallelism as one axis on the parallelism map, and the Parallelism series puts it beside the other four. The film walks the toy across four GPUs, then puts DeepSeek-V3’s numbers on every step.

They will ask Why is serving a Mixture of Experts harder than serving a dense model with the same active parameter count?
The 30-second version Memory: the router decides per token, mid-layer, so every expert must be resident; you size memory by total parameters and get FLOPs from active ones. Communication: the chosen experts sit on other GPUs, so each layer runs a dispatch and a combine all-to-all, with sizes known only at run time, and the slowest GPU sets the step. And arithmetic intensity: a batch of B decoding tokens touches about N(1 − (1 − k/N)^B) experts, which is nearly all of them by a few hundred tokens, while the FLOPs still scale with k. So the bytes scale with N and the work with k, and a sparse layer needs about N/k times the batch of a dense one to reach the roofline ridge: 32 times for DeepSeek-V3. That is why its decode runs expert parallelism 144 ways.

What does the bill look like, in symbols?

Write $N$ for the experts, $k$ for the picks per token, $h$ for the model width and $d$ for one expert’s hidden width. One SwiGLU expert holds $3hd$ weights.

Memory follows the total; FLOPs follow the active. The toy stores $N P_e + P_R = 4 \cdot 12 + 8 = 56$ weights and runs $k P_e + P_R = 32$ per token. DeepSeek-V3 stores 671B parameters and runs 37B per token: 256 routed experts and one shared expert in each of 58 MoE layers, of which a token runs 8 plus the shared one. At one byte per weight in FP8 that is 671 GB671 GB, worked outone expert: 3 × 7,168 × 2,048 = 44.0M weightsone MoE layer: 257 × 44.0M + router ≈ 11.3Bwhole model: 671B weights × 1 byte (FP8) = 671 GBAn eight-GPU H100 node has 8 × 80 = 640 GB., more than an eight-GPU H100 node holds, while each token costs about 2 × 37B = 74 GFLOP, like a 37B dense model. An MoE saves FLOPs, not bytes.

Expert parallelism gives whole experts to whole GPUs. Dispatch sends each token’s hidden state to its experts’ GPUs; combine sends each result home, weighted by its gate. On four GPUs the toy sends 5 of its 8 copies across, and the column sums of the send matrix are exactly the load from Part 3: 2, 2, 4, 0. DeepSeek dispatches in FP8 and combines in BF16, so per token per MoE layer that is 57 kB out and 115 kB backbytes per token, worked outout: 8 experts × 7,168 numbers × 1 byte = 57,344 Bback: 8 × 7,168 × 2 bytes = 114,688 BBefore node-limited routing removes duplicate trips..

The wire’s price fits in one ratio. A token-expert pair costs $6hd$ FLOPs and moves $3h$ bytes, $h$ out in FP8 and $2h$ back in BF16:

\[\frac{6hd}{3h} = 2d \ \text{FLOP per byte}\]

$h$ cancels. For DeepSeek-V3, $2d = 4{,}096$, so an H100 at 989 TFLOP/s needs 241 GB/s241 GB/s, worked out989 × 10¹² FLOP/s ÷ 4,096 FLOP per byte = 241 × 10⁹ B/sone pick over a 50 GB/s port: 21,504 B ÷ 50 GB/s = 430 nsits arithmetic: 88.1 MFLOP ÷ 989 TFLOP/s = 89 nsAbout 4.8 times longer to move than to compute, at peak and without overlap. of traffic to stay fed. NVLink gives 450 inside a node; a 400 Gb/s network port gives 50. So V3 let a token reach at most 4 nodes and overlapped every transfer with other work. DeepSeek’s April 2026 V4 report states the same rule with $d = 3{,}072$: 6,144 FLOP per byte, so each GB/s hides 6.1 TFLOP/s. V4 drops the node limit, and its wave-scheduled kernel, MegaMoE, reports 1.50 to 1.73 times speedups over non-fused baselines on inference, and up to 1.96 on RL rollouts.

Stragglers. The all-to-all is a meeting point. On four GPUs the toy’s work is 2, 2, 4 and 0 expert runs, so the layer takes 4 while half the GPU time idles. A copy of the hot expert on the idle GPU splits its runs 2 and 2 and halves the time. DeepSeek serves V3 with 32 redundant experts placed by a load balancer for exactly this reason.

Decoding generates one token per sequence per step. A batch of $B$ tokens, each picking $k$ of $N$ experts, reads about

\[\mathbb{E}[\text{experts read}] = N \left( 1 - \left(1 - \tfrac{k}{N}\right)^{B} \right)\]

For V3 that is 8, 163 and 252 of 256experts read, worked outeach expert is missed by one token with probability 1 − 8/256 = 0.96875B = 32: 256 × (1 − 0.96875³²) = 256 × 0.638 = 163B = 128: 256 × (1 − 0.96875¹²⁸) = 256 × 0.983 = 252B = 1 reads exactly k = 8. experts at $B$ = 1, 32 and 128. The bytes stop scaling with $k$ and start scaling with $N$, while the FLOPs still scale with $k$. In BF16 a dense layer does one FLOP per byte per token and reaches the H100 ridge of 295 at 295 tokens; V3’s routed experts do $Bk$ over experts read and need about 9,450, which is $N/k = 32$ times more.

So the serving layout spreads the bank wide. DeepSeek decodes V3 with routed experts over 144 GPUs, 2 per GPU, and prefills over 32 GPUs, 9 per GPU: 288 slots either way, the 256 experts plus 32 redundant copies. Prefill already brings thousands of prompt tokens per step, so it needs less width. Decode needs the width: few experts per GPU leaves memory free for a batch big enough to share each read.

The same formula is why the memory floor stays the total. Offloading cold experts to the host works at batch 1, where a token reads 8 of 256; batch up and you read nearly all of them every step.

Why is temperature zero not deterministic?

Because your token’s path can depend on the other tokens in the batch. If capacity is enforced, an expert that fills up drops whatever arrives next, and what arrives next depends on whose requests were batched with yours.

Dropless kernels remove that, but a subtler version remains. Routing is a top-$k$ over floating-point scores, so a different kernel, batch shape or summation order can flip which expert wins. Discrete selection turns a rounding difference into a different computation.

Rapid fire: can you do these from memory?

  1. Which parameter count sets an MoE's memory, and which its FLOPs per token? Give DeepSeek-V3's two numbers.
  2. Why does expert parallelism need an all-to-all where tensor parallelism needs an all-reduce?
  3. Derive the FLOP per byte of one token-expert pair, and say what bandwidth an H100 needs to hide it.
  4. Derive the expected number of experts a batch of B decoding tokens reads.
  5. Why do prefill and decode want different expert-parallel widths?

Part 5 leaves the mechanism behind and asks what is still unsettled.