Why is almost every big diffusion model borrowed from an autoregressive one?
Make a list of the diffusion language models that mattered in 2026 and ask where each one came from. DiffusionGemma is Google DeepMind’s Gemma 4 mixture-of-experts model, fine-tuned into a diffusion model with fewer than 10% of the tokens Gemma 4 was trained on. NVIDIA’s TwoTower is two copies of Nemotron 3 Nano. FLARE and dQwen3.5 start from Qwen3.5; I-DLM, OPDLM and several drafters start from Qwen3.
Only a few were trained from scratch: ByteDance Seed’s iLLaDA, an 8B model on 12 trillion tokens, and Tohoku’s Sumi, a 7B model on 1.5 trillion. Almost everyone else borrowed an autoregressive brain.
An autoregressive (AR) model writes one token per forward pass, left to right, each token predicted from the ones before it. It is the model behind almost every chatbot, and the field has spent enormous compute making good ones. A diffusion model writes many tokens per pass, filling a canvas of masked slots. The bet of 2026 was that what the first kind knows can be moved into the second for a small part of the price.
The price gap is large. Here is what it cost to get a diffusion model, counting only the tokens of diffusion training:
| model | starting point | diffusion training tokens |
|---|---|---|
| iLLaDA-8B | from scratch | 12T |
| LLaDA-8B (2025) | from scratch | 2.3T |
| Sumi-7B | from scratch | 1.5T |
| TwoTower | Nemotron 3 Nano 30B-A3B, frozen | about 2.1T, for the new half only |
| Dream-7B (2025) | Qwen2.5-7B | 580B |
| Nemotron-Labs-Diffusion | Ministral3, plus 1T of AR training | 300B |
| dQwen3.5-9B | Qwen3.5-9B | 50B |
| FLARE | Qwen3.5 (2B, 4B, 9B) | about 10B |
| I-DLM-8B | Qwen3-8B | 4.5B |
| OPDLM-8B | Qwen3-8B | 0.066B |
The LLaDA-8B and Dream-7B counts are the ones dQwen3.5’s paper lists; the rest come from each model’s own paper. TwoTower’s 2.1T is about 8%8%, worked outthe denoiser trained on about 2.1T tokens, as the abstract reports (its method section says about 1.4T)the backbone it borrows was pretrained on 25T2.1 ÷ 25 = 0.084, about 8% of what its backbone was pretrained on, and OPDLM’s 0.066B is 66 million tokens66 million, worked out0.066B = 0.066 × 10⁹= 66 × 10⁶ tokensPrompts from about 60,000 examples, with the model writing its own answers.. From scratch costs trillions; 2025’s conversions cost hundreds of billions; this year’s cost billions or less.
This part is about that borrowing: what converting means and breaks, the cheaper recipes, the designs that keep the AR model instead of replacing it, and a twist: diffusion models working for AR models as drafters. The running example is the series’ two-bridge canvas.
Ask a model: Name a US city and its most famous bridge. It has learned two answers equally often, six tokens each: San Francisco: the Golden Gate and New York: the Brooklyn Bridge. Each slot’s prediction on its own, its marginal, is San or New at 0.5 each, Francisco or York at 0.5, the colon and the certain, then Golden or Brooklyn and Gate or Bridge at 0.5.
Sample every slot from its own marginal at once and you can get New Francisco: the Golden Bridge. Only 12.5%12.5%, worked outfour uncertain slots, two words each: 2 × 2 × 2 × 2 = 16 combinationsreal answers among them: 22 ÷ 16 = 12.5% of such draws are real answers. The series calls that mix “New Francisco”, and it comes back at the end of this part in a new role.
The 30-second version
“New” survives: the AR model starts with New half the time, and the drafter proposed it exactly that often, so the check keeps it. “Francisco” is rejected, because after New the AR model gives it probability 0; at that slot the AR model writes its own token, York, and everything drafted after the rejection is thrown away. The rule keeps each drafted token with probability min(1, p/q), the AR model's probability over the drafter's, and draws any replacement from the probability the draft left uncovered. Together those two give every token exactly the AR model's probability, so a bad draft costs speed, never correctness.What does converting an autoregressive model actually change?
Two things, and everything else is reused: the weights, the tokenizer, the shape of every layer.
The first is the attention mask. An AR model is trained with a causal mask: each position may look only at the positions before it, which is what makes it a next-token predictor. A diffusion model has to fill a slot using the slots on both sides, so the mask is flipped to bidirectional, at least inside the block being written.
The second is the loss. Instead of predicting the next token, the model is shown an answer with a random fraction of its tokens replaced by masks and trained to recover them. Most conversions also keep the AR model’s logit shiftLogit shiftAn AR model’s output at position i predicts the token at position i + 1. Masked diffusion models usually predict the token at the same position. Keeping the shift means a converted model goes on using each position to predict the next one, the habit its weights were trained for., so the weights keep doing the job they learned.
That sounds harmless. It is not. The OPDLM paper names two things that go wrong. Changing the objective and the attention pattern can discard knowledge the AR model had. And training sees answers masked at random, while generation sees the states the model’s own confidence rule produces, an order nobody trained it on.
The damage is measurable. FLARE ran a controlled study on Qwen3-1.7B: the same data, trained once as AR fine-tuning and once as plain block diffusion. The diffusion version scored 21.8 points lower, averaged over its benchmark groups. I-DLM saw the same on Qwen3-8B: trained as a standard block diffusion model, HumanEval fell from 92.7 to 60.3.
Why does borrowing work at all, then? Because diffusion models end up writing mostly left to right anyway. The dQwen3.5 paper measures how often a model decodes earlier positions before later ones: 0.5 is random order, 1.0 is strict left to right. Its converted models score 0.88 to 0.97 over the whole answer, and so does LLaDA-8B, trained from scratch. The AR brain’s left-to-right habit is not wasted; text is mostly written that way.
Where on the dial between one token and the whole answer do converted models sit?
Almost every shipped diffusion model is a block diffusion model. It splits the answer into blocks of B slots, fills one block at a time by diffusion, and moves left to right. A finished block never changes, so its keys and values go into a KV cacheKV cacheThe stored keys and values of tokens already processed, kept so that later passes can attend to them without recomputing them. AR models keep one; a diffusion model can keep one only for text that will no longer change. exactly as an AR model’s would.
The block size is a dial. At B = 1 every block is one slot, and the model is an AR model. At B equal to the whole answer it is full diffusion. Turn the dial on the canvas:
The schedule is the series’ simple one: commit every slot whose top word reaches 0.9, and if none does, commit the most confident slot with a fair coin picking the word. At B = 1 the answer takes 6 passes6 passes, worked outeach block is one slot, and each pass finishes one block6 slots ÷ 1 per pass = 6 passes, 1 token per forward. At B = 3 it takes 44 passes, worked outblock 1: the colon is sure (pass 1), a coin writes San (pass 2), Francisco follows (pass 3)block 2: the, Golden and Gate are all sure given San Francisco (pass 4)6 tokens ÷ 4 passes = 1.5 tokens per forward. At B = 6 it takes 33 passes, worked outpass 1: the colon and the are surepass 2: a coin writes Sanpass 3: Francisco, Golden and Gate are now sure6 tokens ÷ 3 passes = 2 tokens per forward.
Big blocks give more tokens per pass. Converted models still choose small ones: FLARE and OPDLM train with blocks of 4, TwoTower with 16. DiffusionGemma’s 256 is the exception, bought with an extended fine-tuning run and a distilled sampler.
TwoTower’s ablations show why. Shrinking the training block from 128 to 32 raised its general, code and math scores from 69.59, 63.56 and 75.73 to 72.94, 68.64 and 80.57. And sampling with a bigger block than it was trained on is a cliff: trained at 16, sampled at 64, its GSM8K score fell from 89.84 to 2.20.
The traces say the same. Inside a block, TwoTower commits earlier positions first, an upper-left triangle when the paper plots position against step. Its authors suspect the causal backbone: 23 of its 29 sequence-mixing layers are Mamba layers, which read only left to right. A borrowed brain thinks left to right, and a small block keeps each prediction close to the next-token prediction it was trained for.
Can the conversion keep more of the old model, for less?
Three 2026 recipes keep the AR model in the loop while the diffusion model trains, instead of throwing it away after initialization: as a frozen anchor, as a teacher, and as a training objective that never switches off.
REPR-ALIGN, from Peng and colleagues, starts from a hypothesis: the hard part of language is the representations, not the order of writing. So it keeps a frozen copy of the AR model running beside the diffusion student. The AR copy reads the clean text; the student reads the masked text. At every layer, the student’s hidden state at each position is pulled toward the frozen one’s:
\[\mathcal{L} = \mathcal{L}_{\text{denoise}} + \lambda \cdot \underset{\text{layers, positions}}{\operatorname{mean}} \Big(1 - \cos\big(h_{\text{student}},\, h_{\text{AR}}\big)\Big), \qquad \lambda = 10\]In words: denoise as usual, and pay a penalty whenever a hidden state points in a different direction from the AR model’s at the same layer and position. The student learns a new way to decode inside the old feature space.
It works where they tested it. On Qwen3-1.7B, HumanEval pass@10 after conversion went from 31.1 to 40.5; a 0.8B-token subset beat the full 50B-token stream at the same number of steps; the abstract reports up to 4 times faster training. The catch: models up to 4B, and code benchmarks only.
OPDLM, from Su and colleagues at Texas A&M, attacks both of the mismatches from earlier at once with on-policy distillation. The student, the AR model with its mask flipped, writes its own answers with its own confidence rule. Training then takes a half-finished state from one of those trajectories, and at each masked slot asks the frozen AR model what it would predict there, given the student’s own finished answer to the left.
So the student trains on the states it actually visits, and the targets come from the model whose knowledge it is meant to keep. OPDLM-8B needed 66 million tokens; the paper reports 15 to 7,000 times fewer than earlier conversions. The catch is in the fine print: the headline results decode one token per step. At a confidence threshold of 0.8 it writes just over 2 tokens per step, at what the paper calls a moderate accuracy cost.
The third recipe has to solve an architectural problem first. The newest AR models are hybrids that interleave attention with recurrent layersRecurrent layersLayers such as Mamba or Gated DeltaNet that carry a fixed-size state from token to token instead of attending to every earlier token. They are cheaper on long text, and they read strictly left to right: there is no attention mask to flip.. Qwen3.5 uses Gated DeltaNet; Nemotron 3 Nano uses Mamba. A recurrent layer has no mask to flip.
dQwen3.5 simply leaves them causal and flips only the attention layers. It works: against a full-attention control of matched size, the hybrid reached a given training loss in about half the tokens, and dQwen3.5-9B after 50B tokens topped 4 of 7 benchmarks against Dream-7B (580B tokens), Dream-Coder-7B (322B) and LLaDA-8B (2.3T).
FLARE goes further on the same Qwen3.5 backbones. It realizes the diffusion mask inside the recurrent state itself: each masked block starts from the clean state at its left boundary, and the block reads its own end state. It also keeps the AR objective alive in a clean causal stream beside the diffusion stream. About 10B tokens were enough, and with custom kernels FLARE-2B trains at 24.81% of the GPU’s peak arithmetic against 24.04% for plain AR training.
FLARE’s most useful finding is unglamorous. Once the clean stream is in place, the quality of the transfer data explains most of the remaining gap, more than the loss or the mask. Its catches: the two streams roughly double the compute per training step, and FLARE still trails its source model on several benchmarks.
What if you keep the autoregressive model and add a diffusion part beside it?
Every recipe so far changes the AR model’s weights and then fights to keep what they knew. TwoTower, from NVIDIA, refuses to change them at all. Its announcement drew 4,755 likes on X, the most of any paper in this part.
It takes two copies of Nemotron 3 Nano 30B-A3B, a 52-layer hybrid of Mamba, attention and mixture-of-experts layers. The context tower is frozen and only ever reads clean text, causally, the way it was trained, keeping every layer’s cache. The denoiser tower is trained to fill the next block, attending both ways inside the block.
The towers connect layer by layer: denoiser layer i attends to the keys and values of context layer i, so the denoiser reads the prompt through features the frozen model already computed, at every depth. Here is the toy version with blocks of three:
The real model uses blocks of 16 and commits slots above a confidence of 0.8. As the abstract reports, it keeps 98.7% of the AR baseline’s benchmark quality at 2.42 times its wall-clock throughput on two H100s, with the denoiser trained on about 2.1T tokens. Lower thresholds pass 3 times the throughput, with more quality lost.
The ablations make the case for keeping the towers apart. Training the backbone further as an AR model on the new data lost 8 to 18% relative accuracy. Tying the two towers into one network under a joint AR and diffusion loss lost 21 to 28%. The costs are honest too: both towers’ weights sit in memory, and every finished block costs one extra pass of the context tower.
DiffusionGemma makes the same split with a single set of weights. Its report describes an encoder-decoder with shared weights: the network reads the history causally into a cache, then denoises a canvas of 256 tokens with bidirectional attention, cross-attending to that cache. Its canvas starts as uniformly random tokens, not masks. The finished weights can still be run as an AR model, scoring between the diffusion mode and the original Gemma 4.
Nemotron-Labs-Diffusion keeps both behaviors in one network by training them together: the AR loss plus 0.3 times a block diffusion loss, after a stage of AR training alone. The paper finds the two losses complementary: both modes peak at the same weight, 0.3, and the 8B model’s AR mode scores 0.86 points above Qwen3-8B on average.
The result is a tri-mode model: AR decoding, diffusion decoding, and a third mode where the diffusion mode drafts and the AR mode checks. The paper recommends the AR mode for busy, high-concurrency servers. That third mode is the subject of the rest of this part.
One more design keeps the causal machinery itself. CARD, from Ruan and colleagues, puts diffusion inside a strictly causal mask: each position predicts its own token from a noised version of the text before it.
Every token gets a training signal in one pass and the KV cache works as in an AR model. Trained from scratch at 1B on 300B tokens, it matched AR training speed, where block diffusion took 3 times as long. Its catch is scale: 1B parameters.
How can a fast guesser plus a slow checker give exactly the slow model’s answer?
Speculative decoding (Leviathan and colleagues, 2023) rests on one asymmetry. Writing with an AR model takes one pass per token, but checking a whole proposed block takes one pass, because the model scores every position in parallel.
For one user, that pass costs the same whether it scores one position or six. An 8B model in bf16 is 16 GB of weights, and every pass has to read all of them: on an H100 that takes about 4.8 ms4.8 ms, worked out8 × 10⁹ parameters × 2 bytes = 16 GB16 GB ÷ 3,350 GB/s ≈ 4.8 ms per passOne token per pass caps one user at about 209 tokens a second.. The arithmetic for six positions hides inside that read.
So a cheap drafter guesses a block, and the AR model checks it. For each drafted token, let $q$ be the probability the drafter gave it and $p$ the probability the AR model gives it, reading the draft to its left.
The check keeps the token with probability $\min(1, p/q)$. At the first rejection, the AR model writes its own token instead, drawn from $\max(0, p - q)$ renormalized, and everything after it is dropped. If nothing is rejected, the AR model adds one more token of its own for free.
Why is that exact? Take any token $t$ at the first open slot. It can be written in two ways: drafted and kept, or rejected and replaced:
\[P(\text{write } t) = \underbrace{\min\big(q(t),\, p(t)\big)}_{\text{drafted, then kept}} + \underbrace{\max\big(0,\, p(t) - q(t)\big)}_{\text{rejected, then replaced by } t} = p(t)\]The first term is $q(t)$ times $\min(1, p/q)$. The second is the chance of a rejection, which is the total of $\max(0, p - q)$ over all tokens, times the share of that leftover that $t$ gets. Whichever of $p$ and $q$ is larger, the two terms add up to $p(t)$. With greedy decoding the rule shrinks to: keep drafted tokens while they match the AR model’s top choice.
Now run it on the canvas with a diffusion drafter that guesses all six slots in one pass, each from its own marginal:
The drafter proposes New Francisco: the Golden Bridge. New is kept with probability $\min(1, 0.5/0.5)$, which is 1. Francisco is rejected, since the AR model gives it 0 after New, and the AR model writes York. The drafter goes again from New York, every remaining slot is now certain, and the check keeps all four: six tokens in two checks, 3 per check3 per check, worked outcheck 1: New kept, York written by the AR model: 2 tokenscheck 2: the colon, the, Brooklyn and Bridge kept: 4 tokens6 tokens ÷ 2 checks = 3 per checkThe AR model alone needs 6 passes..
That is the puzzle’s answer. The drafter’s New Francisco cost a token of speed, and nothing else: the answer is a sample from the AR model, whatever the drafter believed. DFlash writes the speed-up as one line:
\[\text{time per token} = \frac{T_{\text{draft}} + T_{\text{verify}}}{\tau}\]where τ is the average number of tokens a cycle produces, including the AR model’s own. Faster drafting and longer accepted runs are the only two levers.
Why make the drafter a diffusion model, and what goes wrong with its guesses?
The usual drafter is a small AR model. The strongest of 2025, EAGLE-3EAGLE-3A 2025 speculative decoder: a one-layer AR drafter that reads features from inside the target model and drafts a tree of candidates, one draft pass per depth of the tree., is a single transformer layer. It has to be tiny, because an AR drafter needs a pass per drafted token, and the drafting time grows with every token it guesses.
A diffusion drafter guesses a whole block in one pass, so it can afford to be deeper. That is DFlash, from Z Lab (ICML 2026). Its drafter has five layers and fills a block of 16 positions in one pass. Its key idea, in the authors’ words, is that the target knows best.
The AR model’s hidden states already carry information about the next several tokens. DFlash takes them from five of the target’s layers, fuses them, and injects them as keys and values into every layer of the drafter, so the drafter is less a small language model than a diffusion adapter on the big one.
It shares the target’s embeddings and output head, frozen. Its training blocks start at a clean token, mirroring the token the AR model adds after each check.
On Qwen3-8B with greedy decoding, DFlash produces 7.87 tokens per draft-and-check cycle on MATH-500 and runs 6.08 times faster than plain decoding; averaged over its seven benchmarks, 4.86 times against 1.76 for EAGLE-3 with a tree of 16. It is integrated into the SGLang serving engine. On a B200, its speed-up on Qwen3-8B shrinks from 5.1 times to 2.85.1 and 2.8, worked out1 request at a time: 230 → 1,175 tokens/s, 1,175 ÷ 230 ≈ 5.132 at a time: 5,694 → 16,076 tokens/s, 16,076 ÷ 5,694 ≈ 2.8MATH-500, SGLang, one B200. as concurrency grows from 1 to 32. Part 8 explains why spare compute runs out on a busy server.
The trouble starts when drafters offer a tree instead of one guess: several candidate branches, all checked in one pass, the longest accepted branch winning. A diffusion drafter predicts each slot on its own, so its tree is built from marginals:
With a budget of four branches over the city words, the slot-by-slot drafter scores San Francisco, San York, New Francisco and New York at 0.25 each. The AR model gives two of them 0. Half the budget went to answers that cannot be accepted, whatever the AR model picks. A drafter whose second slot reads the first builds only the two real branches, but needs a draft pass per slot.
JetSpec, from Hu and colleagues, names this the causality-efficiency dilemma and breaks it with a causal mask inside a one-pass draft head: each draft position’s hidden state reads the positions to its left. Its example is a real New Francisco. On one MATH-500 prompt the diffusion head’s top branch was given told that, two openings spliced together, scored −3.76 by the drafter and −63.32 natsNatsUnits of log-probability with the natural logarithm. A branch at −63 nats has probability around e⁻⁶³: the target model considers it impossible. by the target.
The causal head ranked are told that first, close to the target’s own score, and the check accepted 6 tokens instead of 4.
With a 256-token tree on Qwen3-8B, JetSpec reaches 9.64 times on MATH-500, against 8.78 for DDTree, the tree version of DFlash: about 10% faster10%, worked out9.64 ÷ 8.78 ≈ 1.10. At a budget of 16 the two are nearly the same. On a vLLM server the right budget depends on load: at one request, raising it from 16 to 128 took the speed-up from 1.75 to 4.33 times; at sixteen requests, 32 did best.
Notice the role reversal. In a standalone diffusion model, New Francisco is a wrong answer, and Part 3 is about that wall. In a drafter it is a wasted guess, and the AR model’s check guarantees the answer anyway. That is why so much 2026 diffusion work went into drafting.
Can one model draft and check itself, with no second model at all?
A separate drafter is a second model to train, serve and keep in memory. Three papers fold the drafter into the model.
Nemotron-Labs-Diffusion’s third mode, self-speculation, drafts with the diffusion mode and checks with the AR mode of the same weights. The paper’s headline is that the 8B decodes 6 times as many tokens per forward as Qwen3-8B: 5.99 in this mode, against 2.57 for its pure diffusion mode.
Read what it counts: on SPEED-Bench the paper’s own analysis has 6.82 tokens accepted per draft-and-check cycle, which is 3.41 per pass3.41, worked outeach cycle is two passes: one draft, one check6.82 ÷ 2 = 3.41 tokens per pass once both passes are counted. In wall-clock terms, up to 3.3 times plain AR decoding for one user on a GB200, and 3.97 times with an optimized kernel.
Uno, from Sahoo and colleagues, makes the separation exact. Every layer has its AR weights plus small LoRALoRALow-rank adaptation: a small trainable correction added to a frozen weight matrix, here of rank 128. It adds few parameters and can be switched on or off per position. diffusion weights. The draft pass uses both; the check uses the AR weights alone, and they are frozen while the diffusion weights train, so the checker is exactly the original model. Its sampler, Ψ-Spec, drafts with the diffusion weights and applies the acceptance rule from above, so the output is lossless. One model, one KV cache.
Uno is careful about counting. A cycle is two passes, so with a block of B its tokens per forward can never exceed (B + 1) ÷ 2(B + 1) ÷ 2, worked outone cycle = one draft pass + one check passat most B drafted tokens kept + 1 from the AR model = B + 1 per cycle(B + 1) ÷ 2 per pass: B = 4 gives 2.5, B = 16 gives 8.5. On Qwen3-8B, trained on 14.7B tokens with the AR weights untouched, it runs 2.5 times faster than plain decoding for one request.
At the largest batch that fits, Uno runs 1.6 times faster than plain decoding: 5,733 tokens per second against 5,351 for DFlash, about 7% more7%, worked out5,733 ÷ 5,351 ≈ 1.07With 0.35B added parameters against DFlash’s 1.05B, and lower peak memory., and 4,944 for EAGLE-3. Its own 8B, trained from scratch, holds 1.5 times at batch 64. The abstract claims up to 3 times over the base model.
Uno’s paper also reports its 8B beating DiffusionGemma and Mercury 2 on every agentic, coding and long-context benchmark it ran. That is Uno’s evaluation of its rivals, and its own table shows DiffusionGemma about twice as fastabout twice, worked outper request at batch 1: DiffusionGemma 836 tokens/s, Uno 405836 ÷ 405 ≈ 2.1 for a single request. Uno wins on throughput across a full batch, where DiffusionGemma’s speed-up fades.
I-DLM, from Yu and colleagues, asks why converted diffusion models fall behind their AR parents, and names the answer introspective consistency. Let a model generate a token from a distribution q, then re-read the finished text and compute the probability p it gives that token. Average the acceptance rule over the answer:
\[\alpha = \frac{1}{L}\sum_{k=1}^{L} \min\Big(1,\, \frac{p_k(x_k)}{q_k(x_k)}\Big)\]For an AR model p and q are the same distribution, so α is 1: it always agrees with what it wrote. On IFEval, SDAR scores 0.699 and LLaDA2.0-flash 0.568: they often would not endorse their own tokens.
I-DLM converts Qwen3 with a causal mask, the logit shift and fully masked training inputs, on 4.5B tokens, 12 times fewer12 times, worked outSDAR, converted from the same Qwen3-8B: 54B tokens54 ÷ 4.5 = 12 than SDAR. Its decoder proposes a few tokens at masked positions and, in the next pass, re-checks them with the acceptance rule while proposing more. I-DLM-8B scores 69.6 on AIME-24 against Qwen3-8B’s 73.1 and SDAR’s 10.0, writing 2.96 tokens per forward.
The paper also offers a LoRA-gated variant it calls bit-for-bit lossless. Uno’s authors report they could not reproduce lossless speed-ups with the released adapters. A good reminder that “lossless” is a property of a sampler and its implementation, and worth checking.
Is the endgame a diffusion model, or an autoregressive model with a diffusion head?
The dial in the first figure turned out to be continuous. Set Diffusion, from Arriola and Kuleshov (ICML 2026), gives every position a random reveal time and a window width w. At w = 1 any order is possible, masked diffusion; at w = 1/L positions come strictly one at a time, AR, and the training bound becomes exact.
Its set-causal architecture updates the KV cache after every step, not after every block. On small models it beats block diffusion on speed and quality, and infills better at up to about 25% faster.
So the question is not which family wins. It is where on that spectrum the best systems settle, and 2026 gave evidence both ways.
For an AR model with a diffusion head: drafting is lossless by construction, so quality is the AR model’s. DFlash and Uno keep a speed-up at every batch size they tested, while standalone diffusion models lose theirs on busy servers. I-DLM matched its AR parent most closely by making diffusion more causal. Nemotron’s own tri-mode paper sends heavy traffic to the AR mode.
For a diffusion model: checking costs a second pass and accepts only a prefix, and that caps it. Nemotron-Labs-Diffusion measured the ceiling of its own diffusion mode with an ideal sampler: 6.02 tokens per pass, against 3.41 for self-speculation, 76.5% more76.5%, worked outideal diffusion sampler: 6.02 tokens per passself-speculation: 3.41 per pass6.02 ÷ 3.41 ≈ 1.765: 76.5% more. DiffusionGemma averages about 20 tokens per pass for one user. And only a diffusion model can edit, infill and revise, which Part 4 takes up.
What would settle it? Three things to watch. A sampler that gets near the diffusion ceiling without a checker, which is Part 3’s problem. Conversions whose quality matches the parent at many tokens per pass, not one. And serving: whether a diffusion model can keep its speed-up at high concurrency, Part 8’s problem.
The line between a diffusion LLM and speculative decoding dissolved this year. The honest answer to the question in the title is that nobody knows yet, and the strongest systems of 2026 hedge: TwoTower, Nemotron-Labs-Diffusion, DiffusionGemma and Uno all keep a working AR model inside.
Rapid fire: can you do these from memory?
- What two changes turn an AR checkpoint into a masked diffusion model, and what does keeping the logit shift preserve?
- Name the two mismatches OPDLM says conversion creates, and how on-policy distillation answers both.
- On the two-bridge canvas, how many passes at block size 1, 3 and 6? Why do converted models still pick small blocks?
- In TwoTower, which tower is frozen, what does each denoiser layer attend to, and what does the design cost at inference?
- State the speculative-decoding acceptance rule and show it gives every token exactly the target's probability.
- Why does a diffusion drafter's tree waste budget, and what does JetSpec change to fix it?
- Why can Uno's tokens per forward never exceed (B + 1) ÷ 2, and what makes it lossless?
- What is introspective acceptance, and why is it exactly 1 for an AR model?
Where can you go deeper?
- Fred Zhangzhi Peng and others, Don’t Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment (preprint, 2026), arXiv:2605.06885; code at github.com/pengzhangzhi/Open-dLLM. REPR-ALIGN: layer-wise cosine alignment to the frozen AR model.
- Xingyu Su and others, Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation (preprint, 2026), arXiv:2606.06712; project page, code. OPDLM: the two mismatches and on-policy distillation from the frozen AR model.
- Anton Xue and others, dQwen3.5: Hybrid-Attention Diffusion Language Models (preprint, 2026), arXiv:2609.20751. Converting hybrid backbones with causal recurrent layers; the left-to-right measurements and the table of conversion budgets.
- Yuchen Zhu and others, FLARE: Diffusion for Hybrid Language Model (preprint, 2026), arXiv:2606.01774. The controlled transfer study, the recurrent-state mask and one checkpoint with two decoding modes.
- Fitsum Reda and others, Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context (preprint, 2026), arXiv:2606.26493; weights. The frozen context tower, the layer-aligned denoiser and the block-size ablations.
- DiffusionGemma Team, DiffusionGemma Technical Report (preprint, 2026), arXiv:2608.00146; weights. The shared-weight causal encoder and bidirectional decoder, and the retained AR mode.
- Yonggan Fu and others, Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding (preprint, 2026), arXiv:2607.05722. The joint loss, the three modes and the speed-of-light analysis.
- Junhao Ruan and others, Causal Autoregressive Diffusion Language Model (preprint, 2026), arXiv:2601.22031. CARD: diffusion inside a causal mask.
- Yaniv Leviathan, Matan Kalman and Yossi Matias, Fast Inference from Transformers via Speculative Decoding (ICML 2023), arXiv:2211.17192. The acceptance rule and its proof.
- Jian Chen, Yesheng Liang and Zhijian Liu, DFlash: Block Diffusion for Flash Speculative Decoding (ICML 2026), arXiv:2602.06036; project page, code. The block-diffusion drafter conditioned on the target’s features.
- Lanxiang Hu and others, JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting (preprint, 2026), arXiv:2606.18394; project page, code. The causality-efficiency dilemma and the causal draft head.
- Subham Sekhar Sahoo and others, Unlocking Lossless Speedups in LLMs via Discrete Diffusion (preprint, 2026), arXiv:2609.04010; project page, code. Uno: AR weights plus diffusion LoRA weights, and the Ψ-Spec samplers.
- Yifan Yu and others, Introspective Diffusion Language Models (preprint, 2026), arXiv:2604.11035; project page, code. I-DLM: introspective acceptance and strided decoding.
- Marianne Arriola and Volodymyr Kuleshov, Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding (ICML 2026), arXiv:2607.01775; project page, code. The ordering spectrum from AR to masked diffusion.
- Shen Nie and others, Improved Large Language Diffusion Models (preprint, 2026), arXiv:2606.25331; code. iLLaDA: 8B from scratch on 12T tokens.
- Mengyu Ye and others, Sumi: Open Uniform Diffusion Language Model from Scratch (preprint, 2026), arXiv:2606.19005; project page, code. A 7B uniform diffusion model from scratch on 1.5T tokens.