Why is a diffusion model four times faster for one user and slower for sixty-four?
A diffusion language model starts its answer as a row of masked slots and fills several of them in each forward pass, where an autoregressive (AR) model writes one token per pass. Ask it to name a US city and its most famous bridge, and San Francisco: the Golden Gate can come out in 3 passes instead of 6. Every part of this series leans on that speed.
This part asks what the speed is made of on real hardware, what it costs to keep, what the models cost to train, and how anyone can tell whether one is better than another. It starts with the number every launch post quotes, tokens per second, and with the question that number hides.
The 30-second version
Not the model: the GPU's bottleneck. For one user every pass is waiting on memory: it reads all 16 GB of weights in 4.8 ms whether it processes 1 position or 295, so diffusion's 32 positions per pass cost nothing extra, and it keeps 4 of them. With 64 users, AR processes 64 tokens per pass, still free, while diffusion processes 2,048 positions and keeps 256. That is far past the GPU's ridge point, so every position now costs arithmetic: 33 ms per pass. AR writes 13,400 tokens a second, diffusion 7,727. It pays for 32 positions to keep 4.Start with what one forward pass costs. An 8B-parameter model in bf16 is 16 GB16 GB, worked outbf16 stores each parameter in 2 bytes8 × 10⁹ parameters × 2 bytes = 16 × 10⁹ bytes = 16 GB of weights, and a pass has to stream every byte from the GPU’s memory into its compute units. On an H100 at 3.35 TB/s that takes 4.8 ms4.8 ms, worked out16 GB ÷ 3,350 GB/s= 0.00478 s ≈ 4.8 ms. One token per pass caps one user at about 209 tokens a second209 tokens/s, worked out1 token ÷ 4.78 ms≈ 209 tokens per second.
The arithmetic is cheap by comparison. Each position the pass processes costs about 2 FLOPs per parameter, and the H100 does 989 TFLOP/s of dense bf16, so one position takes 16.2 µs16.2 µs, worked out2 FLOPs × 8 × 10⁹ parameters = 16 × 10⁹ FLOPs per position16 × 10⁹ ÷ 989 × 10¹² FLOP/s ≈ 16.2 × 10⁻⁶ s. For one user the GPU spends 4.8 ms reading and 16 µs computing.
That gap is a budget. A pass can process more positions at no extra time until the arithmetic catches up with the reading, at about 295 positions295, worked out989 × 10¹² FLOP/s ÷ 3.35 × 10¹² bytes/s ≈ 295 FLOPs per bytea pass does 2 FLOPs per parameter per position and reads 2 bytes per parameterso its FLOPs per byte equals its positions: break-even at 295The GPU series calls this the ridge point of the roofline.. That is the H100’s ridge pointRidge pointPeak arithmetic divided by memory bandwidth: how many FLOPs a kernel must do per byte it reads before the arithmetic, not the memory, sets its speed. Below it a kernel is memory-bound and extra arithmetic is free; above it, compute-bound.. Left of it a pass is memory-bound and extra positions ride along free; right of it every position adds 16.2 µs.
Diffusion spends that budget. Most models that shipped this year decode in blocks: the answer is written block by block, left to right, and inside a block the masked slots are filled in parallel. Take blocks of 32 positions that commit 4 tokens per pass, 4 tokens per forwardTokens per forward (TPF)Committed tokens per forward pass, the series’ unit of parallelism. An AR model’s TPF is 1 by definition. Positions that a pass computes and does not commit do not count.. For one user it processes 32 positions, keeps 4, and the pass still takes 4.8 ms: 838 tokens a second838 tokens/s, worked out32 positions is far below 295, so the pass still takes 4.78 ms4 tokens ÷ 4.78 ms ≈ 838 tokens per second4 times AR’s 209, for free., four times AR.
Now share the GPU. A serving system batches users, so one pass carries every user’s work. With B users, AR processes B tokens and keeps B; block diffusion processes 32B positions and keeps 4B. Up to about 9 users9 users, worked outdiffusion processes 32 positions per user295 ÷ 32 ≈ 9.2 users before its passes reach the ridge both stay left of the ridge and both scale for free: at 8 users, 1,675 against 6,700 tokens a second8 users, worked outAR: 8 tokens ÷ 4.78 ms ≈ 1,675 per seconddiffusion: 8 × 32 = 256 positions, still left of 295, so 4.78 ms8 × 4 = 32 tokens ÷ 4.78 ms ≈ 6,700 per second.
Past that, diffusion’s passes cost arithmetic, and its throughput hits a ceiling. A pass can never keep more than 4/32 of the positions the arithmetic allows, which caps it at 7,727 tokens a second7,727, worked outat full arithmetic: 989 × 10¹² ÷ 16 × 10⁹ FLOPs ≈ 61,812 positions per seconddiffusion keeps 4 of every 32: 61,812 × 4 ÷ 32 ≈ 7,727. AR keeps climbing at 209 per user and reaches that ceiling at about 37 users37 users, worked outAR adds 209.4 tokens/s per user while its passes stay free7,727 ÷ 209.4 ≈ 36.9 users. At 64 users the ranking has flipped: 13,400 against 7,72764 users, worked outAR: 64 tokens, still free: 64 ÷ 4.78 ms ≈ 13,400 per seconddiffusion: 64 × 32 = 2,048 positions × 16.2 µs ≈ 33.1 ms per pass256 kept ÷ 33.1 ms ≈ 7,727 per second7,727 ÷ 13,400 ≈ 0.58: diffusion at 58% of AR..
One line holds the whole toy. A pass takes whichever is slower, reading the weights or doing the arithmetic, and throughput is what it keeps divided by that time:
\[\text{tokens per second} = \frac{\text{tokens kept per pass}}{\max\left(\dfrac{\text{weight bytes}}{\text{memory bandwidth}},\ \dfrac{\text{positions per pass} \times 2 \times \text{parameters}}{\text{peak FLOP/s}}\right)}\]In words: while the left term in the bracket wins, extra positions are free and diffusion’s 4-for-1 is pure profit. Once the right term wins, every position is paid for, kept or not.
| users | AR tokens/s | block-32 diffusion tokens/s | diffusion ÷ AR |
|---|---|---|---|
| 1 | 209 | 838 | 4.0× |
| 8 | 1,675 | 6,700 | 4.0× |
| 16 | 3,350 | 7,727 | 2.3× |
| 37 | 7,747 | 7,727 | 1.0× |
| 64 | 13,400 | 7,727 | 0.58× |
| 295 | 61,766 | 7,727 | 0.13× |
It is a toy. It ignores attention and the cache traffic that grows with context, and those favor neither side simply. The rule it teaches survives every real measurement below: diffusion’s speed for one user is bought with arithmetic the GPU had spare. Once the GPU is busy, positions computed and not committed are a cost, and the fix is to commit more of what you compute.
What do real servers measure?
The toy is mine; the papers measured. The first thing to check in any speed claim is how many requests were running at once.
DiffusionGemma’s technical report is unusually plain about it. On one H100 in FP8, serving one request, a pass refines a canvas of 256 tokens in 13.56 ms; at its average of 19.74 tokens per forward that is 1,456 tokens a second1,456 tokens/s, worked out19.74 tokens per pass ÷ 13.56 ms per pass ≈ 1,456 per secondagainst Gemma 4 AR’s 204: 1,456 ÷ 204 ≈ 7.1 timesThis is the report’s own worked figure; its benchmark table measures 1,479 over seven benchmarks., 7.1 times its AR parent, Gemma 4. Then it adds users, and AR (with multi-token prediction) takes the throughput lead at around 32 concurrent requests. The report notes that batch sizes above 1 were not tuned.
The toy put the crossover at 37 users; Google measured about 32 for a 26B mixture-of-experts model with a 256-token canvas. Different model, same shape.
NVIDIA’s tri-mode Nemotron-Labs-Diffusion builds the lesson into the model. In self-speculation mode, where diffusion drafts a block and the same model’s AR mode verifies it, its 8B commits about 6 times the tokens per forward of Qwen3-8B (2.57 times in pure diffusion mode).
The paper turns that into 4 times the throughput on SPEED-Bench with SGLang on one GB200, measured at a concurrency of one. It keeps an AR mode because, in its own words, AR “remains well suited for compute-bound high-concurrency scenarios”.
Its throughput plot (on an RTX Pro 6000) shows the same lesson from another side. In self-speculation mode the best block shrinks as load rises: 64 positions at one or two streams, 32 at four, 16 at eight and sixteen, 8 at 32 and 64. And the AR mode at 128 streams sits above every self-speculation point. Fewer speculative positions when the GPU is full: the toy’s lesson, measured.
Uno, from Part 2, keeps an AR model’s weights, adds light diffusion weights that draft several tokens in one pass, and verifies them so the output is exactly the AR model’s distribution. Its authors measured at every concurrency on one H200 (1,024 tokens in, 8,192 out).
On a Qwen3-8B base, the speed-up over plain AR falls from 2.5 times at one request to 1.6 times at 642.5× to 1.6×, worked outone request: 445 ÷ 176 tokens/s ≈ 2.564 requests: 5,733 ÷ 3,662 tokens/s ≈ 1.6, and the best setting moves from a 16-token draft tree to a 4-token draft.
The same paper measured the open diffusion models at the largest batch that fits. DiffusionGemma, the fastest per request at 836 tokens a second, delivers 1,136 tokens a second in aggregateaggregate throughput, comparedUno 8B: 5,255 tokens/s; its own AR base: 3,577Nemotron-Labs-Diffusion 14B: 2,794; DiffusionGemma: 1,1365,255 ÷ 1,136 ≈ 4.6Uno’s authors’ measurement, one H200, bf16., against 5,255 for the 8B Uno. Uno’s authors add that both diffusion models are slower than their own AR bases at large batch.
I-DLM, also from Part 2, stays a diffusion model but makes each pass commit most of what it computes. Its decoder proposes 4 new tokens per pass while checking the previous ones, and keeps 2.96 per pass in its tests. Converted from Qwen3, on H100s, it reports about 3 times the throughput of earlier diffusion models: 2.2 to 3.8 times LLaDA-2.1-mini at 16 to 32 concurrent requests.
The catch, from Uno’s authors: I-DLM’s speed-ups change the parent model’s outputs, and they could not reproduce the lossless speed-ups claimed for its gated-LoRA variant.
Against AR with EAGLE-3 speculative decoding (a small drafter guesses, the big model checks), I-DLM’s per-request throughput goes from 341 against 238 tokens a second at one request to 199 against 176 at 32, on MATH-500. Still ahead, by less.
It also names a cost the toy hides: in block diffusion every request in a batch waits for the slowest block. I-DLM advances every request at least one token per step, so ordinary continuous batching works, and its stationary-batch scheduler reuses the batch across steps instead of rebuilding it.
One user is the regime diffusion was built for, and a phone always serves one user. llada.cpp runs LLaDA-8B on Snapdragon phones by moving the dense work to the phone’s NPU, a matrix engine. One denoising pass over 32 positions takes 4,239.6 ms on the CPU and 478.1 ms on the NPU, 8.87 times faster8.87×, worked out4,239.6 ms ÷ 478.1 ms ≈ 8.87, and the full engine is 17 to 42 times faster than the CPU baseline.
For 128-token answers it still trails Llama-3-8B on the oldest of its three phones and beats it on the two newer ones. The authors’ reason is the toy’s: AR’s one-token passes give a matrix engine almost nothing to chew on.
| claim | requests at once | hardware | measured by |
|---|---|---|---|
| DiffusionGemma 1,456 tokens/s, 7.1× Gemma 4 | 1 | 1 H100, FP8 | Google DeepMind |
| AR overtakes DiffusionGemma | about 32 | 1 H100, FP8 | Google DeepMind |
| Nemotron-Labs-Diffusion 4× on SPEED-Bench, self-speculation | 1 | 1 GB200, SGLang | NVIDIA |
| Uno 2.5× falling to 1.6× over AR | 1 to 64 | 1 H200 | Uno's authors |
| DiffusionGemma 1,136 tokens/s in aggregate | largest that fits | 1 H200 | Uno's authors |
| I-DLM 2.2× to 3.8× over LLaDA-2.1-mini | 16 to 32 | H100s | I-DLM's authors |
| llada.cpp 17× to 42× over the CPU | 1 | Snapdragon phones | llada.cpp's authors |
Why can’t a bidirectional model simply keep a KV cache?
Everything above assumed each pass costs only the weight read. At long context, attention costs too, and AR’s answer to that is a cache.
Attention lets every position look at the others. Each position computes a query (what it is looking for), a key (what it offers to be found by) and a value (what it hands over). Its output mixes the values of the positions it looks at, weighted by how well its query matches their keys.
In an AR model a position only looks back, so once a token’s key and value are computed they never change. The model stores them, the KV cacheKV cacheThe stored keys and values of every earlier token, in every layer. With it an AR model runs only the newest token through the layers: the new token computes its own query, key and value and reads the rest from the cache. Its size grows with every token of context., and each new token is the only one that runs through the layers.
A diffusion model like LLaDA attends both ways. Every slot looks at every other, masks included. When a pass unmasks Golden in slot 5, every other slot now sees a different context, so its key in every layer above the first changes too, though its own token did not. No key is final until the whole answer is, and the exact implementation recomputes all of them on every pass.
The 2025 answer was to cache anyway and accept an approximation. dKV-Cache noticed that a token’s key and value barely move once it is decoded but jump on the step it is decoded, so it caches them one step late and refreshes the whole cache every few steps (every 8 on LLaDA). It reports 2 to 10 times faster decoding.
Fast-dLLM decodes in blocks and reuses the keys and values of the prompt and the earlier blocks for every step inside the current block, refreshing them when the block is done. On LLaDA with GSM8K on one A100, the cache alone lifted 6.7 tokens a second to 21.2, and with confidence-based parallel decoding to 54.4Fast-dLLM’s speed-ups, worked outcache alone: 21.2 ÷ 6.7 ≈ 3.2 timescache and parallel decoding: 54.4 ÷ 6.7 ≈ 8.1 timesGSM8K accuracy 79.3% before, 78.5% after., at 78.5% accuracy against 79.3%.
Both work because neighboring steps’ keys are nearly the same. Neither is exact: the model was trained to see every slot’s latest state, and the cache shows it a stale one.
Block diffusion makes the cache exact by training for it. A slot attends to its own block and the blocks before it, never to a later block. A finished block’s keys then depend only on finished text, so they are final, and caching them changes nothing. Block size 1 is plain AR; one block the length of the answer is full diffusion. DiffusionGemma uses blocks of 256, LLaDA2.0-mini blocks of 32.
The exact cache brings back AR’s old problem with it: an attention cache grows with every token.
What does a cache that never grows buy at 256k tokens?
This is the question of our Fixed State paper. An attention layer keeps a key and a value for every token. A state-space layer such as Mamba reads the sequence as a recurrence instead: it carries a fixed-size state and folds each new input into it.
Block diffusion suits that well. When a block is finished we fold the whole block into the state once, and the next block reads only the state.
Two things make it exact. The sequence mixer has to summarize finished blocks into a state the next block can reuse. And the model has to be trained the way it decodes: earlier blocks clean, one frontier block partly masked, later blocks fully masked, with the Mamba layers’ backward scan kept inside the frontier block so nothing leaks from the future.
We pretrained three 3B block-diffusion denoisers on 300B tokens, with blocks of 32 and one objective: attention in all 28 layers, Mamba-2 in all 28, and a hybrid with attention in 5. Then we decoded all three through one cached interface on a single H100.
Attention’s cache costs about 280 KiB per token280 KiB, worked out28 layers × 2 (key and value) × 2,560 numbers × 2 bytes = 286,720 bytes286,720 ÷ 1,024 = 280 KiB per tokenThe measured table grows by about as much: (81.91 − 44.26) GB ÷ 131,072 tokens ≈ 287 KB.. At 256k tokens the model’s peak memory reaches 81.91 GB and a decode step takes 29.31 ms. The Mamba model’s peak memory stays between 7.41 and 7.65 GB at every length we measured, and its step between 6.72 and 6.77 ms.
At 256k that is 4.3 times lower latency4.3×, worked out29.31 ms ÷ 6.76 ms ≈ 4.3, 11 times less memory11×, worked out81.91 GB ÷ 7.65 GB ≈ 10.7, which rounds to 11Peak memory, weights included: the 3B weights alone are about 6 to 7 GB. and 2.6 times the throughput2.6×, worked out297 ÷ 114 tokens/s ≈ 2.6, one stream each of a single stream.
The constant footprint also buys batch. At 256k, attention cannot run even two streams on the card. The Mamba model runs eight in about 8 GB, 1,593 tokens a second: 14 times14×, worked outMamba, eight streams: 1,593 tokens/sattention, the one stream that fits: 114 tokens/s1,593 ÷ 114 ≈ 14.0 attention’s aggregate. The hybrid sits between: 21.52 GB and 10.74 ms per step at 256k.
It also reaches further. We trained at 1,024 tokens. On a passkey test (find a number hidden in long filler), attention falls from 100% at 1,024 tokens to 12.3% at 2,048 and to 0 at 8k, while Mamba keeps 23.6% at 8k and 22.3% at 16k. Validation losses of the three sit within about 0.03 nats of each other.
The limits are real. These are 3B models trained at 1,024 tokens, so everything past that is extrapolation, and the absolute accuracies far out are modest.
Mamba trails attention by about one point on our downstream average (0.421 against 0.434), its recall favors recent text, and we have not yet combined the cache with confidence-based parallel decoding. What it shows is that the cache does not have to be attention’s: a recurrent state plus a training objective that matches decoding gives an exact cache whose size does not depend on the context.
Once the cache works, where does the time go?
Into moving memory, it turns out. Flash-dLLM profiled cached diffusion decoding and found the time going not to attention but to traffic between kernels: the query, key and value projection, the rotary position embedding and the cache write each ran separately, writing intermediates to GPU memory and reading them back.
It fuses the three into one kernel that writes keys and values straight into the cache, 1.37 times faster per layer, and lets the model draft and verify its own tokens. On LLaDA-1.5 on one A100, GSM8K went from 41.7 tokens a second under Elastic-Cache, the strongest previous cache, to 210.6, and HumanEval from 16.8 to 185.6: 5.1 and 11.0 times5.1× and 11.0×, worked outGSM8K: 210.6 ÷ 41.7 ≈ 5.1HumanEval: 185.6 ÷ 16.8 ≈ 11.0Accuracy 82.79% → 83.02% and 37.80% → 40.24%., at equal or better accuracy. The catch: one model family, and open-ended generation left for later.
A mixture-of-expertsMixture-of-experts (MoE)A layer that holds many small feed-forward networks, the experts, and a router that sends each token to a few of them. The model has many parameters but each token uses only a slice, which is cheap as long as only a few experts’ weights have to be read. model makes the weight read itself grow. Serving one user, an AR pass reads only the experts its one token chose. A diffusion pass routes every position of the block, and each picks its own.
DiffusionGemma’s report counts it: Gemma 4 activates 8 experts per token per MoE layer, DiffusionGemma about 84 per 256-token canvas, and its MoE kernel runs 4.3 times slower. The extra positions of the first section rode along free because the weight read was fixed. With experts, it is not.
dMoE routes per block instead: it adds up the router’s scores across the block into one block-level choice, while each token still uses as many experts as before. On LLaDA2.0-mini with blocks of 32, the unique experts per layer on MATH500 fell from 70.0 to 16.3, under a quartera quarter, worked out16.3 ÷ 70.0 ≈ 0.23Across four benchmarks the paper reports 69.5 falling to about 16.5., with accuracy going from 72.0% to 71.2%. The end-to-end speed-up is modest, up to about 2 times.
Expert-choice routing (COLM 2026) turns routing around: each expert picks a fixed number of tokens instead of each token picking experts. That balances the load by construction, and only a diffusion pass can do it, because it sees all its tokens at once; an AR model cannot let experts choose among tokens it has not written. Training reached the same loss in 10.6 hours instead of 20.7, 2.0 times faster2.0×, worked out20.7 h ÷ 10.6 h ≈ 2.0.
Because capacity becomes a dial, the same paper spends more of it on the denoising steps where few slots are still masked, whose tokens learn about ten times more efficiently. Existing token-choice models convert by replacing only the router.
How do you get valid JSON out of a sampler that writes many tokens at once?
Structured output is the other serving feature that assumed left to right. An AR server enforces a JSON schema or a function-call format by masking, at each step, every next token that would break it.
A diffusion pass samples many slots at once, each from its own prediction. Each slot can be valid on its own and the combination still broken: Part 3 calls that “New Francisco”, two slots each sampled sensibly that do not belong together. With braces and quotes, it is a parse error.
Dang and Ermon write the constraint as a finite automaton, a state machine that accepts exactly the valid strings, and treat it as a graphical model over the slots. Then they sample exactly from the model’s own predictions restricted to valid sequences, in a number of steps that grows with the logarithm of the length rather than the length.
Every output is valid by construction. On BFCL-Live function calling with Dream-7B, greedy accuracy goes from 63.9% to 71.5%, and sampled accuracy, which collapses without the constraint, from 22.3% to 69.0%, for under 5% extra wall-clock time.
Is a diffusion model really 16 times more expensive to train?
Serving is half the bill. In 2025 a scaling study (Nie et al., ICLR 2025) found that masked diffusion needs about 16 times the training compute of an AR model to reach the same likelihood, and the number became the standard objection to training diffusion from scratch.
In October, Wang et al. trained about 1,700 models across ten variants and read every multiple at one fixed budget, 10²⁰ FLOPs, instead of extrapolating. Then they changed one thing at a time.
Re-measured in their setup, the old kind of model (full-sequence, decoder-only, scored with the ELBO, a bound on its likelihood) needs 17.8 times AR’s compute, close to the published 16. Scoring the same model with its exact likelihood, in the order it actually decodes rather than the random orders the bound averages over, gives 12.8. Nothing about the model changed: nearly a third of the gapa third, worked outextra compute over AR: 17.8 − 1 = 16.8 times, then 12.8 − 1 = 11.8(16.8 − 11.8) ÷ 16.8 ≈ 0.30 was the measurement.
Then the model. Train it as block diffusion with blocks of 32: 9.6 times. Split the network into an encoder that reads the clean text so far and a decoder that denoises the current block: 7.8. Smaller blocks of 8: 4.6. Blocks of 4: 3.0.
The catch is in the last step. A block of 4 can commit at most 4 tokens per pass, so the cheapest diffusion model to train is also the least parallel one to serve; a block of 1 would be AR itself, at 1. The fits stop at 10²⁰ FLOPs, with one held-out check at three times that, one seed and one decoding rule. The gap is now a dial rather than a wall.
Training long contexts has its own plumbing. A block-diffusion training sequence holds the clean text and a corrupted copy of each block, and the usual context parallelism, sharding the sequence across GPUs by position, ships both kinds of keys and values around.
Block parallelism notices that the objective separates by target block, so it puts each corrupted block’s computation on one GPU and shards only the shared clean text. On 16 H200s at 256k context it trains 1.18 to 1.45 times faster than the best baseline, 1.61 times at 512k, and it trains a DFlash2 drafter 7.59 times faster at a million tokens on 8 H100s.
For anyone starting smaller, dLLM is an open library that puts training, inference and evaluation of diffusion language models in one place. It reproduces LLaDA and Dream and has small recipes that turn a BERT-style encoder or an AR model into a diffusion model.
How do you measure a model whose likelihood you cannot compute?
AR models are compared by perplexity: the exponential of the average negative log-probability per token on held-out text. A diffusion model gives no exact log-probability for free. The usual substitute is the ELBO, a lower bound on the log-likelihood, so a diffusion model’s reported perplexity is usually a bound, not the thing itself.
Bounds compare badly across families. In Sahoo et al.’s scaling study, uniform-state diffusion has worse validation perplexity than masked diffusion and still beats both AR and masked diffusion on GSM8K at 1.7B parameters (65.8% against 62.9% and 58.8%, after fine-tuning, with every model decoding left to right one token at a time). Their reading: perplexity is informative within a family and misleading across them.
Even inside one family the estimator matters. LangFlow reported a perplexity of 24.6 for its continuous diffusion model; RePlaid’s authors re-scored the public checkpoint at 32.2. With self-conditioning on, LangFlow’s estimator drops a term, and dropping it flatters the checkpoint by 8.35 points8.35, worked outRePlaid’s estimator on LangFlow’s checkpoint, term included: 33.29the same estimator with the term dropped, as LangFlow does: 24.9433.29 − 24.94 = 8.35Lower perplexity looks better, so the missing term reads as a gain.. Part 5 follows that thread into the choice of noise.
What does the bound measure, exactly? Casado Noguerales et al. prove that for any noising process, the negative ELBO equals the entropy of the data plus a KL divergence between two processes over whole denoising trajectories:
\[-\mathrm{ELBO} = H(\text{data}) + \mathrm{KL}\left(\text{exact reversal of the noise} \,\Vert\, \text{the model's reversal}\right)\]In words: the bound is the floor every model shares, the data’s own unpredictability, plus how far the model’s denoising paths are from the true ones. Since the floor is the same for every noising process, a gap between masked and uniform diffusion’s bounds says nothing about the noise itself; it comes from capacity and trainability. And the bound is loose because it charges the model for taking a different path even when the path ends at the same text.
The 16× study is the practical version of that looseness: the same model looked 17.8 times worse than AR scored with the ELBO and 12.8 times worse scored exactly.
For unconditional samples the field reports generative perplexity: generate text, score it with a frozen AR model (usually GPT-2 large), report that model’s perplexity. Franca and Tong show it can be gamed. A sampler that just cycles through the 400 most common tokens of OpenWebText scores 19.6, better than a 1.3B Plaid diffusion model’s 20.7, against 17.1 for real text.
It even has higher unigram entropy, the usual guard against repetitive text (5.93 against 5.32), while its MAUVE, a score of how close the generated distribution is to real text, is 0.004 against Plaid’s 0.896. Generative perplexity rewards being predictable to the scorer, and collapsing to one sequence is optimal for it. Their prescription is a set of metrics that expose different properties: distributional distances, diversity across samples, and judgments by LLMs.
What is still open?
Serving at high concurrency is the first. Every measurement above agrees with the toy: diffusion’s lead is largest for one user and shrinks as the GPU fills. DiffusionGemma’s crossover at about 32 requests, Uno’s 2.5 times falling to 1.6, Nemotron’s shrinking blocks and its AR mode for busy servers.
The models that hold up best commit most of what they compute. Whether a pure diffusion model can keep a lead over a tuned AR server on a fully loaded GPU is still open: the evidence so far stops at 32 to 64 requests.
Exact caches at scale are second. Ours works at 3B with a 1,024-token training context; what happens at frontier scale, with long-context training, with recall that does not favor recent text, and combined with aggressive parallel decoding, is open. So is the MoE question: a block wakes many experts, and per-block routing and expert choice are first answers.
Training cost is third. 3 times is not 1, and the cheapest setting is the least parallel one. And measurement is fourth: exact likelihoods depend on the decoding order, ELBOs are not comparable across noising processes, and generative perplexity can be gamed. Until the field agrees on a set of metrics, “better” in a diffusion paper means better on the metric that paper picked. Part 9 puts these next to the rest of the year’s open problems.
Rapid fire: can you do these from memory?
- Why does one forward pass of an 8B bf16 model on an H100 take about 4.8 ms whether it processes 1 position or 200?
- What is the H100's ridge point, and what does it mean for how many positions a pass can process for free?
- A block-32 diffusion model keeps 4 tokens per pass. What caps its throughput on a busy H100, and at about how many users does AR catch up?
- Why does unmasking one slot change the keys of the other slots in a bidirectional model, and how does block diffusion avoid it?
- What two conditions make the Fixed State cache exact, and why does its memory not grow with the context?
- Why does a diffusion pass in an MoE model read so many more experts than an AR pass, and what do dMoE and expert-choice routing change?
- Walk the 16× training-compute gap down to 3×. Which step changed only the measurement?
- How can a sampler that writes nonsense beat a trained model on generative perplexity?
Where can you go deeper?
- Vaibhav Singh, Pierre-André Noël, Torsten Scholak and Eugene Belilovsky, Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale (preprint, 2026), arXiv:2609.11998. Our paper: attention, Mamba and hybrid block-diffusion denoisers at 3B, the exact constant-size cache, and the 256k measurements.
- Xinyin Ma et al., dKV-Cache: The Cache for Diffusion Language Models (preprint, 2025), arXiv:2505.15781. Delayed caching of decoded tokens’ keys and values, with periodic refresh.
- Chengyue Wu et al., Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (preprint, 2025), arXiv:2505.22618. The block-wise approximate cache and confidence-aware parallel decoding.
- Quan Nguyen-Tri, Mukul Ranjan and Zhiqiang Shen, Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs (preprint, 2026), arXiv:2609.26796. The fused cache kernel and self-drafting; project page, code.
- DiffusionGemma Team, DiffusionGemma Technical Report (2026), arXiv:2608.00146. Section 6 has the single-request breakdown, the 84 experts per canvas, and the multi-user crossover.
- Yonggan Fu et al., Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding (2026), arXiv:2607.05722. Figure 9 has throughput against concurrency.
- Subham Sekhar Sahoo et al., Unlocking Lossless Speedups in LLMs via Discrete Diffusion (preprint, 2026), arXiv:2609.04010. Uno and its measurements at every batch size; project page, code.
- Yifan Yu et al., Introspective Diffusion Language Models (preprint, 2026), arXiv:2604.11035. I-DLM, introspective strided decoding and the stationary-batch scheduler; project page, code.
- Tuowei Wang, Yanfan Sun and Ju Ren, Efficient On-Device Diffusion LLM Inference with Mobile NPU (preprint, 2026), arXiv:2606.13740. llada.cpp on Snapdragon NPUs.
- Sicheng Feng et al., dMoE: dLLMs with Learnable Block Experts (preprint, 2026), arXiv:2605.30876. Block-level expert routing; project page, code.
- Shuibai Zhang et al., Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models (COLM 2026), arXiv:2604.01622. Expert choice, timestep-dependent capacity and router retrofits; code.
- Meihua Dang and Stefano Ermon, Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata (preprint, 2026), arXiv:2607.07026. Exact constrained sampling for parallel samplers.
- Guanghan Wang et al., Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models (preprint, 2026), arXiv:2610.17590. The 16× to 3× walk, about 1,700 models; project page.
- Shen Nie et al., Scaling up Masked Diffusion Models on Text (ICLR 2025), arXiv:2410.18514. The study behind the original 16×.
- Tarun Suresh et al., Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training (preprint, 2026), arXiv:2609.19242. Block parallelism and its context-sharded version; code.
- Zhanhui Zhou et al., dLLM: Simple Diffusion Language Modeling (preprint, 2026), arXiv:2602.22661. The open training, inference and evaluation library; code.
- Subham Sekhar Sahoo et al., Scaling Beyond Masked Diffusion Language Models (preprint, 2026), arXiv:2602.15014. Perplexity within and across diffusion families.
- Rodrigo Casado Noguerales et al., What Does a Discrete Diffusion Model Learn? One Reverse Process in Three Coordinates (preprint, 2026), arXiv:2607.05381. The negative ELBO as data entropy plus a path KL.
- Yuxin Chen et al., LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling (preprint, 2026), arXiv:2604.11748. The reported 24.6 perplexity.
- Zhihan Yang et al., Continuous Diffusion Scales Competitively with Discrete Diffusion for Language (preprint, 2026), arXiv:2605.18530. RePlaid; Appendix E re-scores LangFlow’s checkpoint and isolates the dropped term.
- Antonio Franca and Alexander Tong, Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics (preprint, 2026), arXiv:2606.08417. Four training-free samplers that game gen-PPL, and the metrics to use instead.