What does noise do to a word?

A diffusion model is trained backwards. You take clean data, damage it on purpose, and teach a network to undo the damage a little at a time. To generate, you start from pure damage and run the network again and again until something clean comes out.

For images the damage is obvious. A pixel is a number, so you add Gaussian noise to every pixel until only static is left, and “slightly noisier” means something precise.

A word is not a number. What is San, slightly noisier? There is no single answer, and the choice is not a detail. It decides what the model can see halfway through writing, what it can take back, and whether it can keep a question open.

This series follows one example. Asked to name a US city and its most famous bridge, a model has learned two answers equally often, six tokens each: San Francisco: the Golden Gate and New York: the Brooklyn Bridge. A diffusion language model fills all six slots of its answer in a few forward passes (one pass is one run of the whole network over the answer) instead of one slot per pass.

Most diffusion language models of the last two years corrupt text the same way: they hide words behind a mask. That choice won on one number, perplexity. In 2026 it stopped looking settled. Google’s DiffusionGemma swaps words for random words instead, continuous models came back at 3B and 8B parameters, and a process built on sets of candidate words beat both kinds of discrete noise on logic puzzles.

The puzzle Halfway through writing the bridge answer, the model believes slot 1 is San or New, 50/50. Masked diffusion, uniform diffusion, continuous diffusion and candidate-set diffusion each store a half-finished slot differently. Which of them can carry both options, still alive, into the next step?
The 30-second version Only continuous and candidate-set diffusion. A masked slot holds either [M], which records nothing about San or New, or a committed word, which is final. A uniform slot holds exactly one word: it can change later, but it is one guess at a time. A continuous slot is a vector, which can sit halfway between the embeddings of San and New, and a candidate-set slot is literally the set {San, New}. Masked and uniform models can still pass the 50/50 on, but only beside the state, through self-conditioning, which is what DiffusionGemma does.

Here are the four answers to “what does noise do to San?”, in plain words.

Masked. At a random moment the word is replaced by a special token, [M], which means “unknown”. Before that moment the slot shows San; after it, nothing. The mask is absorbing: once a slot is masked, more noise never brings a word back.

Uniform. At random moments the word is swapped for a word drawn at random from the whole vocabulary, every word equally likely. San might become Gate, then York.

Continuous. The word is replaced by its embeddingEmbeddingThe vector of numbers a model learns for each token of its vocabulary, and the form in which a token enters the network. Words used alike end up with nearby vectors. Continuous diffusion adds its noise to these vectors instead of to the tokens., and Gaussian noise is added to that vector, as to the pixels of an image. The point drifts away from San and blurs.

Candidate set. The slot holds the set of words that are still possible. Noise adds words to the set until every word is possible.

All four end in the same place, with nothing of San left. They take different routes. The figure raises the noise on one slot under all four at once.

What can a half-noisy slot still say?

Stop the figure at noise level 0.5 and read each row.

The masked slot is either San or [M], nothing in between. That has a strength people tend to overlook: under masking, a word you can see is always the true word. The model never has to wonder whether San itself is noise.

The uniform slot has no such guarantee. At noise level 0.5, with the figure’s ten-word vocabulary, it shows San with probability 0.550.55, worked outleft alone: probability 1 − 0.5 = 0.5swapped, but the random draw lands on San again: 0.5 × 1/10 = 0.050.5 + 0.05 = 0.55With a real vocabulary of 100,000 tokens the second term all but vanishes.. A visible word is either the original or a random draw, and a wrong word looks exactly as real as a right one.

Written for any noise level $t$ and a vocabulary of $V$ words:

\[P(\text{the slot shows San}) = (1 - t) + \frac{t}{V}\]

In words: the slot was left alone, or it was swapped and the draw happened to land on San.

The continuous point is still nearest San, but its blur reaches New and Francisco: it holds a lean, not a word. The candidate set says “one of these five words”, and San is among them.

Now run time the other way, which is what generation does. Halfway through writing the bridge answer, the model believes slot 1 is San 0.5, New 0.5. That belief is the model’s output. What reaches the next pass is the state written into the slot, and the four kinds of state differ in what they can write.

A masked slot can stay [M], which records nothing about San or New, or commit one of them, which is final. A uniform slot must hold one word, so a coin turns 50/50 into San. Only the continuous point, halfway between the two embeddings, and the candidate set {San, New} carry both options into the next step.

That matters because of the series’ central failure. If two slots are settled by separate coins in the same pass, San or New in slot 1 and Francisco or York in slot 2, then 2 of the 42 of 4, worked outtwo coins, each 50/50: 2 × 2 = 4 equally likely outcomesreal answers: San Francisco, New Yorkmixed: New Francisco, San York, so 2 of 4 are wrong equally likely outcomes are mixed answers such as New Francisco (Part 3 is about this). A state that can say “San or New” lets the model put the choice off until the other slots have settled.

Why did masking win?

Around 2024 the race between kinds of noise was scored with perplexityPerplexityThe exponential of the average negative log-probability a model gives each true token of held-out text. A perplexity of 20 means the model is, on average, as unsure as a fair choice among 20 tokens; lower is better. A diffusion model cannot compute its log-probability exactly, so it reports an upper bound from its ELBO., the standard measure of how well a language model predicts held-out text. A diffusion model cannot compute that number exactly. It reports an upper bound derived from its ELBOELBOThe evidence lower bound: a quantity a diffusion model can compute that is guaranteed to be at most the log-probability of the text. Training pushes it up. Turned into a perplexity it gives an upper bound, and how loose that bound is depends on the noise process., the training objective.

The scores, as one 2026 paper tabulates them for models of about 130M parameters on OpenWebText, after the same training:

model noise perplexity (lower is better)
autoregressive transformer none, left to right 17.5
MDLM masked 23.2
SEDD absorb masked 24.1
Duo uniform 25.2
UDLM uniform 27.4
SEDD uniform uniform 29.7

Upper bounds for the diffusion models, from Table 2 of the LangFlow paper.

Masked models sat closest to the autoregressive (AR) baseline. MDLM, in 2024, also made masking simple: its training loss is a weighted mixture of ordinary masked-language-model losses, the fill-in-the-blank task BERT is trained on, at every noise level.

The mask also removes a whole question from the model’s job. Every visible token is true, so the model only has to fill blanks. A uniform model must fill blanks and decide which visible words to trust, with nothing to tell the two apart.

Continuous diffusion was further behind still. Plaid, the strongest likelihood-trained continuous model of 2023, needed 64 times an AR model’s training compute to reach the same likelihood, a figure quoted for years as proof that continuous diffusion does not scale.

So the field went with masks. LLaDA, Dream, LLaDA2.x and Nemotron-Labs-Diffusion all hide words behind [M]. A 2026 scaling-law study puts the scoreboard in one line: to match an AR model’s validation loss, masked diffusion needs about 16 times the training compute (14 times with a lower-variance loss) and uniform diffusion about 23 times.

Masking has a price, and it is the absorbing property. Run backwards, it means that once a slot is unmasked, its word is final. A model that commits New in slot 1 and Francisco in slot 2 has no way back. Part 4 covers the editing machinery 2026 added to fix that.

Why is perplexity a bad referee between kinds of noise?

The scaling-law study is Sahoo and colleagues’ Scaling Beyond Masked Diffusion Language Models, the first to fit scaling laws for uniform diffusion and for an interpolating family. Its thesis is in the abstract: perplexity “is informative within a diffusion family but can be misleading across families”.

The strongest evidence is at 1.7B parameters. The authors pretrained an AR model, MDLM (masked), Duo (uniform) and Eso-LM (an AR and masked hybrid) on 2.1 trillion tokens each, then fine-tuned all four on 385,000 grade-school math problems.

On GSM8K, Duo solved 65.8%, AR 62.9%, MDLM 58.8% and Eso-LM 33.4%. The uniform model, whose perplexity scales worse than the masked model’s, came out 2.9 points2.9 and 7.0, worked outover AR: 65.8 − 62.9 = 2.9 pointsover masked: 65.8 − 58.8 = 7.0 points ahead of AR and 7.0 ahead of masked.

Read the catch before quoting that. All four models were decoded left to right, one token per pass, because the authors note that confidence-based decoding “effectively collapses to left-to-right generation” anyway. So the result is about what a uniform model learns, not about parallel speed, and it is one benchmark at one scale.

Why would perplexity mislead? Each family’s number is an upper bound on its own process, and different noise processes give bounds of different tightness, so a gap in the bounds need not be a gap in the models.

And perplexity says nothing about sampling. A uniform model can rewrite earlier words, so it can produce decent text in few steps. On the paper’s speed and quality frontier (sample quality scored by Llama 2 7B), AR wins below 200 tokens per second, Duo wins from 200 to 400 and above 600, and Eso-LM, which can cache, wins in between.

The largest open release of 2026 sided with uniform noise. DiffusionGemma’s report says its denoiser “uses multinomial (or uniform) diffusion rather than masked diffusion”, because “all tokens can transition between one another” and so “the model can continuously correct its own errors”. Its canvas starts as 256 uniformly random tokens, not masks. Words already accepted in the current canvas can still change; only finished canvases are frozen.

Sumi, from Tohoku University, is the first fully open uniform model trained from scratch at scale: 7B parameters on 1.5 trillion tokens, competitive with AR models trained on similar token budgets on knowledge, reasoning and code. It is a clean reference point, and it also shows the catch.

Given extra passes to revise finished answers, Sumi overwrote committed tokens in 58% to 100% of revision steps, yet at most 1% of final tokens changed and at most 1 answer in 30 flipped. The edits were mostly round trips, A to B and back to A. Being able to change a word is not the same as knowing which word is wrong.

Diffusion papers have a second number, and it misleads too. Since the true likelihood is out of reach, they often report generative perplexityGenerative perplexityGenerate samples from the model, then score them with a separate, frozen autoregressive model, usually GPT-2 Large. A low score means the scorer found the samples predictable. It is usually reported with the samples’ unigram entropy, the spread of words inside each sample, as a check against repetition.: generate text, then ask a fixed AR model how predictable it finds it.

Franca and Tong built four samplers with no training at all, from word frequencies, that write incoherent text by construction. One, Periodic-400, cycles through the 400 most common tokens and emits the same sequence on every draw.

On OpenWebText it scores a generative perplexity of 19.6. The AR baseline scores 35.3 and real text 17.1. The usual guard, unigram entropy, passes as well: 5.93 against real text’s 5.48, because the entropy is measured inside one sample, and one sample of 400 common tokens is varied.

Metrics that compare whole sets of samples with real text catch it at once. Its MAUVE score, a measure of how close the distribution of samples is to real text, is 0.004 against the AR baseline’s 0.925. The authors argue for a panel of such metrics, plus diversity checks and LLM judges, instead of one number. Part 8 returns to measurement.

What is the sampling wall, and how do you get over it?

Go back to the moment the puzzle describes and look at it as a masked or uniform model sees it. Pass 2 believes slot 1 is San 0.5, New 0.5. A discrete step must write a token, so a coin writes San.

Pass 3 is given the canvas and nothing else. A belief of 0.5 / 0.5 and a belief of 0.9 / 0.1 that both drew San look identical to it. Jo and colleagues named this the sampling wall (ICLR 2026): once a token is sampled, the distribution behind it collapses into one hard choice that cannot reach the next step.

They tie two known waste patterns to it. Many steps reproduce the previous step’s tokens and make no progress, and others oscillate, because each pass rebuilds its picture of the canvas from hard tokens alone.

Their fix, loopholing, opens a second road. Besides the sampled tokens, each step hands the next one the model’s last hidden state, a continuous vector that still holds the distribution. Training it naively would mean running the whole chain of steps, so they use self-conditioningSelf-conditioningFeeding a denoiser its own previous prediction as an extra input. In training, a first pass makes a prediction with nothing fed back, and a second pass gets that prediction (with no gradient through it) and is trained on the loss. In sampling, every step gets the previous step’s prediction. Chen, Zhang and Hinton introduced it in 2022 for continuous diffusion.: a first pass produces a stand-in state, and a second pass learns to use it.

The gains are large on their benchmarks. Generative perplexity fell 55% for a masked base model and 61% for a uniform one, and the masked model’s gap to AR on that metric shrank from 3.17 times to 1.43 times. Added to a masked reasoning model, it lifted accuracy on the four-number Countdown arithmetic puzzle from 45% to 56.3%.

DiffusionGemma uses the same idea, under the name self-conditioning, and cites loopholing among its sources. Each denoising step returns the predicted probabilities and a new self-conditioning signal for the next step:

\[z_{\text{next}} = \mathrm{FFW}(\hat{p}\,E)\]

In words: multiply the predicted probabilities $\hat{p}$ by the embedding table $E$, which gives each slot the average of the word embeddings weighted by its belief, and pass that through a small feedforward network, a 7.8M-parameter block added to the model for this alone. A 50/50 slot gets a vector halfway between San and New. The open-source training kit applies it to half the examples in each batch.

The sampler around it is built for the same purpose. It accepts tokens in order of lowest entropy until an error budget is spent, and puts every other position back to a uniformly random word, so the next pass explores again. A 256-token canvas is allowed at most 48 passes and in practice takes around 12, about 21 tokens per pass21 tokens per pass, worked out256 tokens ÷ about 12 passes ≈ 21The report’s suite-wide average is about 20 tokens per forward pass..

The belief is not free. The report measures a text-diffusion sampling step at 3.06 ms against 0.56 ms for an AR step’s sampling, about 5.5 times as long5.5 times, worked out3.06 ms ÷ 0.56 ms ≈ 5.5. It points to the self-conditioning multiply and a softmax over all 256 positions, done in full precision over a 262,000-token vocabulary.

Take the idea to its end and you get LAW, Loopholing All the Way (Deschenaux, Ilieva and colleagues, October 2026). It drops the tokens and carries only the belief: starting from uniformly random tokens, each step passes on the expected embedding under the model’s prediction, and the tokens are decoded once, at the end.

\[\bar{e} = \sum_{w} p(w)\, e_w \;=\; 0.5\, e_{\text{San}} + 0.5\, e_{\text{New}}\]

In words: the belief-weighted average of the word embeddings, which for our slot is the midpoint between San and New.

That choice of state matters. The paper reports 35.8% on GSM8K with the expected embedding against under 1.5% when it propagates the final hidden state instead, as loopholing does.

Trained on synthetic grade-school math (TinyGSM), LAW solves 42.2% of GSM8K problems against 34.5% for uniform loopholing, at 1,024 steps. Fine-tuning the pretrained uniform model Duo with LAW lifts its accuracy from 17.2% to 48.1%, almost three timesalmost three times, worked out48.1% ÷ 17.2% ≈ 2.8. On hard Sudoku it solves 96.5%, against 91.0% for uniform loopholing and 58.4% for Duo.

The catch is scale and steps: small models, synthetic tasks, and 1,024 steps for the math numbers. Notice also what LAW has become. A model whose state is a weighted average of embeddings is no longer discrete at all; it is a continuous diffusion model that happens to start from tokens.

Why did continuous diffusion fall behind, and what brought it back?

Continuous diffusion for text is old: Diffusion-LM in 2022, Plaid in 2023. Three problems kept it behind masking, and 2026’s papers name them.

The first is that the noise means nothing. When tokens are one-hot vectors, a 1 in the token’s position and 0 everywhere else, all tokens are the same distance apart. Deschenaux and Gulcehre point out that adding Gaussian noise to them then “does not have a clear semantic interpretation”, unlike an image, where noise degrades structure gradually.

The second is identifiability. In a one-hot space, Davis and colleagues observe, Gaussian noise has an expected length of √V while the token’s vector has length 1. With GPT-2’s 50,257-token vocabulary, the noise is about 224 times longer224 times, worked outa Gaussian vector in V dimensions has length about √V√50,257 ≈ 224, against 1 for a one-hot token than the signal, so the model struggles to recognize the very tokens it is generating.

The third is a mismatch between training and sampling. In training, the model sees noisy versions of clean tokens. In sampling, it follows its own predictions, which are never perfectly sure, so it travels through states it rarely saw in training. Davis and colleagues show that self-conditioning helps by exposing the model to exactly such inference-like states.

What continuous diffusion offers in return is in the figure. The slot is a point until the last step, so it can wait between two words, and every pass sees every other slot’s current point. The sampling is not factorized: slot 1 and slot 2 are not settled by separate coins, so they can lean the same way together.

LangFlow (Chen and colleagues, April 2026) calls itself the first continuous model to rival discrete diffusion. It ties embedding-space diffusion to flow matching, derives an ODE-based likelihood bound, learns its noise schedule, and adds self-conditioning. It reports a perplexity of 24.6 on OpenWebText and 30.0 on LM1B, at about 130M parameters.

Self-conditioning carries much of that. In LangFlow’s ablation it moves LM1B perplexity from 49.0 to 30.0, a gain of 19 points19 points, worked out49.0 − 30.0 = 19.0, while for the masked MDLM it makes perplexity slightly worse, 31.0 to 32.7. The same trick does opposite things to the two families.

RePlaid (Yang and colleagues, May 2026) asked whether Plaid’s 64-times compute gap was real. Rebuilt with a modern transformer and trained under the exact protocol used for the discrete scaling laws, it needs about 20 times AR’s compute to match its loss: a gap 3.2 times smaller3.2 times, worked out64 ÷ 20 = 3.2Masked needs about 14 times and uniform about 22 times under the same protocol., between masked at 14 times and uniform at 22 times.

At about 0.1B parameters RePlaid reaches a perplexity bound of 22.1 on OpenWebText, below MDLM’s 23.1. Its ablation names the biggest lever: learning the token embeddings. Freeze them at random and perplexity collapses to 39.4.

How far did continuous diffusion get in 2026?

Up to 8B parameters. Sigma (Yang and colleagues, October 2026) is the first continuous diffusion language model at 3B and 8B. It denoises Gaussian-corrupted token embeddings of only 16 dimensions, in blocks of 32 tokens, and warm-starts from an AR model before about 300 billion tokens of training.

At 3B, in diffusion mode, Sigma scores 84.5% on GSM8K, against 83.4% for Nemotron-Labs-Diffusion-3B, a masked model trained under an aligned pipeline, 77.2% for Dream-7B and 71.0% for LLaDA-8B. At inference the authors found two knobs essential: classifier-free guidanceClassifier-free guidanceA trick from image diffusion: run the denoiser with and without the prompt, and push the prediction further in the direction the prompt moves it. It trades diversity for fidelity, and it needs a continuous state to push. and a score temperature.

The catch is the step count. That number uses 256 denoising steps for each 32-token block, 8 forward passes per token8 passes per token, worked out256 steps ÷ 32 tokens = 8 passes per tokenAn AR model needs 1 pass per token., far slower than AR. The paper says so: continuous decoding “remains slower per generated token at high step counts”.

Quality holds down to 16 steps per block, which is 2 tokens per pass2 tokens per pass, worked out32 tokens ÷ 16 steps = 2 tokens per passat 4 steps: 32 ÷ 4 = 8 tokens per pass. At 4 steps it drops sharply: at most 32% on GSM8K without distillation, 68% after the authors distill the many-step model into a few-step one.

Davis and colleagues’ design-space study makes the self-conditioning story precise. DoubleCat adds a second corruption, on the model’s own output distribution, so that training sees inference-like states without the extra forward pass. On TinyGSM-trained math it moves accuracy from 4.42% to 16.00%, 3.6 times3.6 times, worked out16.00% ÷ 4.42% ≈ 3.6.

Their SemiCat then fixes identifiability directly, by revealing tokens as clean one-hot inputs at random times. With standard hyperparameters and a tuned sampler it reaches 49.94% on the same task, discrete-diffusion level. At 1.7B parameters SemiCat beats discrete diffusion on four of six likelihood-based evaluations.

Three more 2026 moves change the geometry or the speed. Hyperspherical flows (Deschenaux and Gulcehre) rotate unit vectors on a sphere instead of adding noise to one-hot vectors. GSM8K accuracy goes from under 1% for earlier flow models to 12% to 18%, matching masked and uniform models at temperature 1, though at temperature 0.1 they reach 33% and 36% against about 18%.

ELF (Hu and colleagues) runs flow matching on contextual embeddings from a frozen T5 encoder and turns them into tokens only at the final step, so image tools such as classifier-free guidance carry over. It reports better generative perplexity than discrete and continuous baselines in 8 to 32 steps, with 10 times fewer training tokens, at about 100M parameters.

H-CDLM (Ollu and Komodakis) diffuses each token together with a coarser cluster of similar tokens, the coarse guiding the fine, and reaches 27.4% on GSM8K, ahead of earlier continuous models. Distribution matching distillation (Le Van Kiem and colleagues) gets a generative perplexity of 45.6 for 1,024-token samples in 4 steps, 49% better than the strongest baseline at matched entropy.

A last group moves the noise one level up, into a learned latent space. Cola DLM (ByteDance Seed) trains a text autoencoder, models the latents with a block-causal diffusion transformer, then decodes, with scaling curves up to about 2,000 EFLOPs against matched 2B AR and LLaDA baselines. Its authors warn that its perplexity looks poor while its generations are good, because the two measure different things.

AURORA-LM (Nanjing University) argues against compressing the latent to make diffusion easy: keep a wide latent that decodes faithfully and make the diffusion model work harder. It reports the strongest results among the continuous and diffusion models it evaluated on OpenWebText generation and XSum summarization, and further gains at 1B parameters and about 1,500 EFLOPs.

What if the noise were a set of candidates?

Think of a Sudoku solver with a pencil. A cell is not blank or filled: it holds small marks, {2, 7, 9}, narrowed to {7, 9} as neighbors fill in, sometimes reopened to include 2 when a guess elsewhere turns out wrong, and finally settled on 7.

Casado Noguerales and colleagues (October 2026) turned that into a noise process. Their abstract states the problem with the mainstream: masked and uniform diffusion “erase a token’s identity in a single corruption event”, so a noisy state cannot say that several alternatives remain plausible.

In Binary Discrete Diffusion a slot’s state is a membership vector, one bit per vocabulary word: 1 if the word is still a candidate. A clean token is a set of one. In the symmetric version, noise flips bits both ways, so candidates can be removed and later restored; at full noise the set is a random half of the vocabulary. In the simpler absorbing version, which the figure’s first step drew, noise only adds candidates.

The network does not need a table of every possible set. It embeds a set by applying its ordinary embedding table to the membership vector scaled to length 1, so {San, New} enters as 0.71 · San + 0.71 · New0.71, worked outmembership vector: 1 for San, 1 for New, 0 elsewhereits length is √(1² + 1²) = √2 ≈ 1.41scaled to length 1: 1 ÷ 1.41 ≈ 0.71 for each word, the two embeddings added with equal weight. Every reverse move changes one bit, so the rates cost no more to compute than an ordinary prediction over the vocabulary.

With matched 43M-parameter transformers, five seeds each and the same tuning budget, the symmetric version wins on all three logic tasks. On the hardest Sudoku puzzles it solves 92.66%, against 89.81% for uniform and 76.12% for masked, 16.5 points16.5 points, worked out92.66 − 76.12 = 16.54 points over masked92.66 − 89.81 = 2.85 points over uniform ahead of masking.

The gap is widest far from the training data. Coloring graphs ten times larger than any seen in training, it succeeds 64.13% of the time; masked diffusion manages 4.85%. Note the order, too: uniform beats masked on every task, the same direction as the GSM8K result.

On language it is competitive but not a winner. No kernel led on all three corpora, and masked diffusion had the best MAUVE on OpenWebText. These are small models on puzzles. Whether pencil marks help a 7B model write code is untested.

What is still open?

Almost everything about the noise. Here is where each kind of state stands at the end of 2026.

noisea half-finished slot holdscan it hold "San or New"?strongest 2026 evidencethe catch
masked[M] or a final wordnobest perplexity; LLaDA2.x and Nemotron-Labs-Diffusion at scalecommits are final without extra editing
uniformone word, which can changeno65.8% on GSM8K at 1.7B, above AR and maskedworse perplexity; edits rarely land (Sumi)
uniform plus self-conditioningone word, plus a belief vector beside itbeside the stateDiffusionGemma; LAW lifts Duo from 17.2% to 48.1%an extra cost per step; the belief is hidden
continuousa point in embedding spaceyesSigma at 3B: 84.5% on GSM8K256 steps per 32-token block for that score
latenta point in a learned latent spaceyesCola DLM and AURORA-LM scaling curvesperplexity hard to compare at all
candidate seta set of live wordsyes92.66% on hard Sudoku against 76.12% masked43M models, logic tasks only

There is no consensus, and the reason is not only that the evidence is new. A fair comparison is itself hard to make, for at least four reasons.

The scores are bounds of different tightness, so perplexity misleads across families, and generative perplexity can be gamed outright. The same checkpoint can even score differently: RePlaid’s authors evaluated LangFlow’s public model and got 32.2 on OpenWebText where LangFlow reports 24.6, 7.6 points apart7.6 points, worked out32.2 − 24.6 = 7.6RePlaid scored it with its own bound, and separately argues that LangFlow’s estimator drops a term when self-conditioning is on, worth about 8 points..

The scales do not match. Most continuous and candidate-set results are at 0.1B parameters or below; only Sigma reaches 8B, and only masked and uniform models have shipped at 7B and beyond.

The step counts do not match. A continuous score at 256 steps per block and a masked score at 20 tokens per pass are different products, and reporting accuracy without passes per token hides the trade.

The tasks do not match. Candidate sets shine on Sudoku, uniform on fine-tuned math, masked on likelihood. Nobody has yet run all of them, at one scale, on one budget, under one set of metrics.

What would settle it? A uniform, continuous or candidate-set model trained at frontier scale under the same budget as a masked one, decoded at the same tokens per pass, and scored with metrics no sampler can game. Until then, the safe reading of 2026 is the one in the puzzle: the field is moving away from states that force an early coin flip.

There is a cost to watch. Self-conditioning and continuous states keep the belief, but they keep it in vectors nobody reads. Part 6 asks how much of that hidden computation is good, and whether it stays readable.

Rapid fire: can you do these from memory?

  1. Name four things noise can do to one word in a diffusion language model, and say what a half-noisy slot holds under each.
  2. At noise level 0.5 with a 10-word vocabulary, how likely is a uniform slot to show its original word? Why does a masked model never face this question?
  3. What does "absorbing" mean for a mask, and what does it imply when the process runs backwards?
  4. Why can perplexity mislead when comparing masked and uniform diffusion, and what result at 1.7B parameters makes the point?
  5. How did a sampler that emits the same sentence every time score well on generative perplexity, and which metrics catch it?
  6. What is the sampling wall? Write the self-conditioning signal DiffusionGemma passes from one step to the next and say it in words.
  7. Give three reasons continuous diffusion lagged behind masking, and one 2026 fix for each.
  8. How does Binary Discrete Diffusion represent "San or New", and on which tasks did it beat masked and uniform diffusion?

Where can you go deeper?

  • Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux and others, Scaling Beyond Masked Diffusion Language Models (preprint, 2026), arXiv:2602.15014, project page. The first scaling laws for uniform and interpolating diffusion, the 16, 14, 23 and 32 times compute multipliers, and the 1.7B GSM8K comparison.
  • DiffusionGemma Team, DiffusionGemma Technical Report (preprint, 2026), arXiv:2608.00146, model page. Section 3.2 for multinomial diffusion and the self-conditioning signal, Section 3.3 for the entropy-bounded sampler.
  • Mengyu Ye, Keito Kudo and others, Sumi: Open Uniform Diffusion Language Model from Scratch (preprint, 2026), arXiv:2606.19005, project page, code. A 7B uniform model on 1.5T tokens, and the revision-budget probe.
  • Antonio Franca and Alexander Tong, Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics (ICML 2026 Workshop on Structured Probabilistic Inference and Generative Modeling), arXiv:2606.08417. Four training-free samplers that game generative perplexity, and a rebenchmark of diffusion models with distributional metrics.
  • Mingyu Jo, Jaesik Yoon, Justin Deschenaux, Caglar Gulcehre and Sungjin Ahn, Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall (ICLR 2026), arXiv:2510.19304. The sampling wall, the deterministic latent pathway, and training it with self-conditioning.
  • Adrien Deschenaux, Nadezhda Ilieva and others, Loopholing All the Way: Language Modeling with pure Self-Conditioning (preprint, 2026), arXiv:2610.30068, code. LAW: propagate only the expected embedding, decode once at the end.
  • Yuxin Chen, Chumeng Liang and others, LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling (preprint, 2026), arXiv:2604.11748, blog, code. Flow matching through a Bregman divergence, an ODE likelihood bound, a learned Gumbel noise schedule, and the self-conditioning ablation.
  • Zhihan Yang, Wei Guo and others, Continuous Diffusion Scales Competitively with Discrete Diffusion for Language (preprint, 2026), arXiv:2605.18530. RePlaid, the 20 times compute gap, and the appendix on estimating likelihood with self-conditioning.
  • Oscar Davis, Daan Roos and others, On the Design Space of Continuous Diffusion Language Models (preprint, 2026), arXiv:2610.13834. Why self-conditioning helps, token identifiability, DoubleCat and SemiCat.
  • Justin Deschenaux and Caglar Gulcehre, Language Modeling with Hyperspherical Flows (preprint, 2026), arXiv:2605.11125. A flow on the unit sphere instead of one-hot vectors.
  • Keya Hu, Linlu Qiu and others, ELF: Embedded Language Flows (preprint, 2026), arXiv:2605.10938, code. Flow matching on contextual embeddings, discretized only at the final step.
  • Zhihan Yang, Wei Guo and others, Large Language Continuous Diffusion Models (preprint, 2026), arXiv:2610.02665. Sigma at 3B and 8B: blockwise continuous diffusion, classifier-free guidance and score temperature, few-step distillation.
  • Mathias Ollu and Nikos Komodakis, Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling (preprint, 2026), arXiv:2610.08738, code. Tokens and coarse clusters diffused together.
  • Paul Le Van Kiem, Dario Shariatian, Umut Simsekli and Alain Durmus, Distribution Matching Distillation for Continuous Diffusion Language Models (preprint, 2026), arXiv:2609.40235. Simplex-DMD and Reinforce-DMD for few-step generation.
  • Hongcan Guo, Qinyu Zhao and others, Continuous Latent Diffusion Language Model (preprint, 2026), arXiv:2605.06548, project page, code. Cola DLM: a text autoencoder plus a block-causal diffusion transformer over its latents.
  • Jiajun Liang, Yucheng Liao and others, AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling (preprint, 2026), arXiv:2608.02602, project page, code. A wide, faithfully decodable latent and a diffusion model built to learn it.
  • Rodrigo Casado Noguerales, Dimitri von Rütte and others, Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks (preprint, 2026), arXiv:2610.28543. Candidate sets as membership vectors, symmetric and absorbing kernels, Sudoku-Extreme, Boxoban and graph coloring.
  • Background: Subham Sekhar Sahoo and others, Simple and Effective Masked Diffusion Language Models (NeurIPS 2024), arXiv:2406.07524; Subham Sekhar Sahoo and others, The Diffusion Duality (ICML 2025), arXiv:2506.10892, the uniform model Duo; Yair Schiff and others, Simple Guidance Mechanisms for Discrete Diffusion Models (ICLR 2025), arXiv:2412.10193, UDLM; Aaron Lou, Chenlin Meng and Stefano Ermon, Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (ICML 2024), arXiv:2310.16834, SEDD; Ishaan Gulrajani and Tatsunori Hashimoto, Likelihood-Based Diffusion Language Models (NeurIPS 2023), arXiv:2305.18619, Plaid; Ting Chen, Ruixiang Zhang and Geoffrey Hinton, Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (ICLR 2023), arXiv:2208.04202.